Disconnected by Henti Smith, on Flickr (CC BY-NC-ND 2.0).

From Mobile First to Offline First

Provide a better, faster user experience—both offline and online—with an Offline First approach.

What follows is a narrative version of my Abstractions presentation From Mobile First to Offline First. This is not a new presentation topic for me. I recently helped to organize Offline Camp. I’ve been advocating for an Offline First approach to building web and mobile apps for over a year. I’ve given talks on Offline First apps and related technologies at numerous conferences including at Apache: Big Data, O’Reilly Fluent, ConFoo, CodeMash, Node.js Interactive, All Things Open, NoSQL Now!, That Conference, OSCON, Cloud Expo, and php[tek].

Somehow, Abstractions feels like a different type of event, and I wanted to take the opportunity to rethink my Offline First presentation. First, the speaker lineup is amazing. Second, the message and community around Offline First has started to come together in a meaningful way. This is the right audience, at the right time, to spread the message about Offline first. So, here it is…


Screenshot of my corporate VPN login.

Let me walk you through how I start my day most mornings. I work remotely for a large enterprise, so of course the first step is to login to my corporate VPN. Every. Single. Morning.

Screenshot of my web-based email client login.

Next, I login to my web-based email client. Every. Single. Morning.

Screenshot of my web-based calendar application in need of a browser reload.

Finally, I need to reload my browser tab containing my calendar application. Every. Single. Morning. For security, of course. The whole process of getting to a place of productivity can take a minute or two on a good day, and even longer if there’s a problem with cookies or some other technical glitch. This is especially fun if I’ve got a conference call first thing in the morning.

Screenshot of a session expiration notice asking me to copy-and-paste my data before logging in again.

There are a lot of, well, interesting ways in which web apps can treat a session expiration. I find this unnamed web app particular fun. What am I supposed to do here? Why can’t my data just be saved locally and then re-synced with the server once I login again?

Screenshot of Trello with a “could not connect to Trello” error message.

It’s not just enterprise software. If my laptop has been closed for any period of time then I often get a “could not connect to Trello” message on my Trello boards, even if my session hasn’t expired. I then need to reload the page to ensure that I have the latest updates and that new changes that I make are not lost. As you can see, my day begins with lots of browser tab reloading.

Thin Client — OpenVMS by Leonardo Rizzi, on Flickr (CC BY-SA 2.0).

This is not atypical for web-based apps. Historically, we’ve treated web browsers as thin clients for our server-side apps. In this model, the browser’s job is simply to render our server-side generated HTML. We’ve slowly moved more-and-more functionality to the client side, but we still don’t completely trust web browsers as app platforms.

Screenshot of my computer’s login screen.

But why not? Before logging in to my VPN, logging in to my email client, and reloading my calendar tab I had already logged in to my computer. My computer is owned and managed by my employer. Once I’m logged in to my computer, why should I need to login to anything else after that?

Screenshot of my iPhone’s login screen.

In addition to my laptop, I also have a corporate-managed iPhone. At any moment I can check my email or look at my calendar almost instantly by logging in to my phone using either Touch ID or by entering my passcode. It takes me a total of about three seconds before I can view my email or calendar.

My smartphone is not treated like a thin client, but rather like a, well, smart client. Logins are persisted in most apps, as the phone’s built-in security features are trusted by these apps. Data is often stored locally, allowing me to use my apps instantly, regardless of connectivity.


Mobile platforms such as iOS are heavily controlled by vendors. As a result, these platforms provide nice, cozy environments for developers to work within. The web, on the other hand, is a radically open platform with no centralized control. Web development is messy business. Screen sizes, browser compatibility, antiquated browsers, rapidly evolving web standards. These are just a few of the complications that web developers contend with on a daily basis.

Screen size visualisation by Orde Saunders, on Flickr (CC BY 2.0).

When Luke Wroblewski articulated the Mobile First approach to web development he outlined an approach that was all about embracing, rather than fighting, the lack of control experienced by web developers. Have no idea what size device your end user will be using to browser your website? That’s OK! Don’t stress about it. Techniques such as responsive web design and fluid layouts allow you to embrace the chaos that is web design and development, and to create better apps out of this chaos.

fluffy chaos by xdxd_vs_xdxd, on Flickr (CC BY-SA 2.0).

Letting go of control is hard. Most of the time, though, we’re not actually letting go of control. What we’re really letting go of is the illusion of control. Even when it’s just an illusion, though, a sense of control can help us feel safe in a chaotic world.


As developers, we need to make assumptions. Even with a Mobile First approach, we decide on the smallest device size that we will support. This is the art of progressive enhancement: start with an assumption of base-level capabilities and take advantage of more capabilities as they become available.

Device landscape by Jeremy Keith, on Flickr (CC BY 2.0).

There can be some debate as to the starting point for progressive enhancement. What is the baseline? To some, progressive enhancement means starting with the assumption of no JavaScript capabilities. Others say that progressive enhancement means ensuring that your web app provides a base-level experience without JavaScript and without CSS.

I like to think of progressive enhancement in the abstract. To me, it doesn’t matter as much what baseline you pick. It can still be progressive enhancement if you pick a baseline and then progressively enhance from that baseline. Just be sure to put some thought into what baseline makes sense for your context and your app.


Cell phone tower by Gary Lerude, on Flickr (CC BY-ND 2.0).

Speaking of assumptions and baselines, it’s easy to think of network connectivity as being ubiquitous. App developers often take having a network connection for granted, an assumed baseline from which to progressively enhance. If not ubiquitous, at least mostly reliable, right? OK, maybe not mostly reliable but at least you can know whether or not your app has connectivity, right? Along similar lines, it’s really easy to make assumptions about latency and bandwidth.

The problem is that these are all false assumptions. If you haven’t read it before, the The Eight Fallacies of Distributed Computing is a great resource on this topic. Let’s take a look at some scenarios…

Cloud Formation Over the Adirondacks by Bradley Holt, on Twitter.

Users of your app are obviously offline when in airplane mode. That’s an easy one. Sure, many flights now offer Wi-Fi, but not typically below 10,000 feet. Also, what happens when your plane’s Wi-Fi doesn’t work? Or, what about those of us who don’t want to pay for the onboard Wi-Fi?

Developing countries don’t have the same level of ubiquitous connectivity as we have here in the United States. If you have any interest in these markets then you can’t take network connectivity for granted. Joel Worrall works on HospitalRun, which is an open source software project for developing world hospitals. He recently had this to say:

However, this issue becomes a business imperative when bringing products and services to a market with less reliable, more costly, or even less available connectivity. In markets such as Kenya, with tremendous opportunity for growth due to smartphone sales, the demand to address the Offline First problem in mobile applications is quickly becoming critical.
Money by thethreesisters, on Flickr (CC BY 2.0).

Many wireless plans include data caps. It’s important to limit the bandwidth usage of your app if at all possible so as not to eat up all of your users’ data for the month or cost them money on data overage charges.

Even in large cities with great connectivity it’s not uncommon to temporarily lose connectivity in subways or to find other areas where wireless signals are blocked by buildings or other obstructions. If your app requires a network connection in every moment, then the user experience will suffer when there is the inevitable blip in connectivity.

workstation by Stev Ngo, on Flickr (CC BY 2.0).

You’ve developed your app on your local computer, in a controlled environment, connecting to a development version of you mobile backend that’s also running on your local computer. It’s so fast! Super-low latency! Once you ship your app to real users, though, in the real world of slow, unreliable networks it’s a completely different story. Your users are frustrated and unhappy with your app.

Illustrations of signal indicator bars: no signal, two bars, three bars, four bars, and five bars.

Then there’s “Lie-Fi.” You’re using an app and the device says that you’ve got a connection, but your message just won’t go through. Your device thinks it’s connected, but it can’t actually send or receive any data. You may have a great connection to a Wi-Fi access point, but its connection to the internet is broken. This is the distributed nature of networks. Each hop in the network needs to work or your message won’t go through—and you won’t know if the network works or not until you try to send the message. You can’t just indiscriminately trust an “online” device status.


I hope that I’ve convinced you that you shouldn’t build your apps to rely on a network connection. But what’s the alternative? You have two basic options: graceful degradation or progressive enhancement. Graceful degradation involves building your app as usual, assuming that it will have a good network connection, and then gracefully handling the resultant errors when the network connection inevitably does not work. The progressive enhancement approach involves assuming that, as a baseline, your app will not have a working network connection. Then, your app can be progressively enhanced to take advantage of network connectivity when available.

Waterfall by Paulo Valdivieso, on Flickr (CC BY-SA 2.0).

The progressive enhancement approach is what we call Offline First. Don’t assume that lack of connectivity is an error condition. Build your app to work without a connection. Get updated content, sync your data, or enable features that aren’t practical to make work offline when the app is connected. Let users still interact with your app when there’s no reliable connection available.

The term Offline First was first coined by Alex Feyerke in a 2013 A List Apart article. The open source Hoodie project (which Alex and others work on) was probably the first framework to embrace the Offline First moniker and approach. Around the same time that Hoodie was created PouchDB was also created. PouchDB is a JavaScript database that syncs. More recently, Service Workers were introduced into Google Chrome, enabling new Offline First use cases in web browsers. Firefox and Opera have also implemented Service Workers, and Microsoft Edge plans to support Service Workers as well.


Screenshot of managing alerts settings in The Weather Company’s Progressive Web App.

If you’ve heard of Progressive Web Apps then some of what I’ve talked about probably sounds familiar. The basic idea of Progressive Web Apps is to combine the discoverability of web apps with the power of native mobile apps. As an end user, you browse to a Progressive Web App just like you would browse to any other website. As you use the app more-and-more, it gains additional native-app-like capabilities. For example, installation to your home screen or the ability to send you alerts and notifications.

Google is currently making a big push for Progressive Web Apps and many companies have invested in building Progressive Web Apps. For example, The Weather Company, which IBM recently acquired, has built a Progressive Web App which allows them to send alerts and notifications to users.

Progressive Web Apps are an opportunity for web developers to rethink how we approach building web apps. The market has clearly demonstrated that it likes the interaction model of native apps. There’s a lot that we can learn from how native apps are built.

Don’t get me wrong, I’m a huge believer in the power of the open web. I don’t think we should just turn web apps into native apps and call it done. I do think that we can incorporate some of the best parts of native apps into how we build web apps while not losing the soul of the open web.


Service Workers fetch diagram by Mozilla Contributors (CC-BY-SA 2.5).

There are many Offline First technologies available that you can start using right now. The newest and biggest development in Offline First for web developers is the Service Worker specification. You may remember the HTML5 Application Cache (AppCache), which was the first attempt at providing a browser mechanism for caching content and assets for offline usage. There were numerous issues with the AppCache and it has since been deprecated by most browsers. Service Workers is a new approach that aims to solve the same basic problem, while avoiding the pitfalls of AppCache. Service Workers give you fine-grained control over caching content and assets for offline usage by intercepting client requests and returning previously-stored results when offline.

Service Workers are awesome for enabling offline usage of content and assets. But, sometimes having an actual database can be helpful. This is where IndexedDB, WebSQL, or localStorage can be helpful. Or, at least they would be helpful if browsers implemented these standards consistently. The browser support and implementation quirks can make it very difficult to actually use these technologies. Fortunately there’s localForage, a wrapper for IndexedDB, WebSQL, or localStorage which provides a unified API.

I encourage you to also check out PouchDB. It’s not as lightweight as localForage, but it includes a bunch of awesome features that are probably worth the extra overhead. The most exciting feature of PouchDB is its ability to sync with anything that implements the CouchDB Replication Protocol. This, of course, includes Apache CouchDB. It also includes IBM Cloudant, a fully-managed cloud database that is based on CouchDB. PouchDB allows you to read and write data directly on the device, and then sync this data to the cloud when you have an internet connection. This makes create, read, update, and delete operations extremely fast as all of your data access happens locally.

Not only can PouchDB run in a web browser, but it can also run in any JavaScript environment including in Node.js, in Apache Cordova or Ionic for hybrid mobile apps, in Electron for desktop apps, and even on a Tessel for Internet of Things (IoT) apps. Nolan Lawson, one of the core contributors to PouchDB, has written up some excellent demos of running PouchDB in these environments and others.

Our team at IBM Cloudant also created open source Cloudant Sync libraries for iOS and Android, in case you prefer to build native mobile apps and would like to take an Offline First approach. While they have different APIs than PouchDB, these libraries, like PouchDB, store data locally on the device and can also sync with anything that implements the CouchDB Replication Protocol.

PouchDB and Cloudant Sync can provide you with an Offline First database for your app. If you want something higher-level then check out Hoodie, a JavaScript framework that uses an Offline First, no backend architecture.


Illustration of the one-database-per user pattern. database by Tim Morgan, on Flickr (CC BY 2.0).

There are still many challenges to be addressed around Offline First. A common pattern for developers using PouchDB or Cloudant Sync is to create one database per user. Each user is given their own local database on their device, as well as their own database in the cloud with which only they can replicate. This works great if you have an app where each user’s data is fairly well segmented. However, as soon as you want to allow a user to access another user’s data, or you want to create aggregate queries across multiple users’ data, the one-database-per-user pattern starts to break down.

Illustration of a whiteboard and the collaborators on the Cloudant Envoy open source project.

Cloudant Envoy is an open source project that our team created to give developers the best parts of one-database-per-user without the pain of one-database-per-user. Cloudant Envoy is effectively a proxy which provides the illusion of one-database-per-user to the PouchDB or Cloudant Sync database on the user’s device, but it actually stores all of the data in one CouchDB or Cloudant database. An access control list is maintained by Cloudant Envoy which dictates who can access what data. This is still an experimental project, but please take a look and let us know how it works for you.


Screenshot of Cloudant FoodTracker.

Our Developer Advocacy team has built several Offline First sample apps to help you learn how to build Offline First apps. Cloudant FoodTracker is an app that we created to demonstrate how to build an Offline First iOS app in Swift using Cloudant Sync. Apple provides a great tutorial on starting to develop iOS apps in Swift. The tutorial walks readers through creating a simple meal tracking app. The app allows users to view and edit a list of meals. Each meal includes a meal name, rating, and a photo.

Apple’s FoodTracker is offline-only, meaning that all of the meal data is stored only on the device and the app provides no means of syncing that data to the cloud. Cloudant FoodTracker replaces the data layer with Cloudant Sync and then syncs with Cloudant or CouchDB.

Screenshot of Field Work.

Field Work is a web app that allows offline editing and mapping of geospatial data. The idea of Field Work is to demonstrate how a company in a field-based industry, such as the utility industry, could benefit from an app which their field personnel could use even when disconnected.

Architectural diagram of Field Work.

Field Work stores GeoJSON locally on the device in PouchDB. It uses Leaflet and Mapbox.js to render the geospatial data on a map. Read-only base map layers are synced from a Cloudant database to a PouchDB database on the device. An editable map layer is then synced between a separate Cloudant database and a separate PouchDB database on the device. There are other aspects to the app, including the ability create work orders.

Screenshot of Location Tracker.

Location Tracker is another iOS sample app, written in Swift. The app tracks a user’s location as they move and stores this data in a local Cloudant Sync database, which is synced with Cloudant when an internet connection is available. As a user moves, and new locations are recorded, the app queries the server for points of interests near the user’s location. Blue pins mark each location recorded by the app. A blue line is drawn over the path the user has travelled. Each time the Location Tracker app records a new location a radius-based geo query is performed in Cloudant to find nearby points of interest.


Tessel 2 by SparkFun Electronics, on Flickr (CC BY 2.0).

I’ve talked about Offline First primarily in the context of Progressive Web Apps. However, Offline First is a concept that is applicable to any type of app that would typically depend on an internet connection. I’ve touched on how an Offline First approach can be used for native mobile apps, for hybrid mobile apps using tools such as Apache Cordova and Ionic, for desktop apps using Electron, and even for IoT apps using Tessel. I’d like to spend a bit more time talking about IoT as this a space that could benefit quite a bit from some Offline First thinking.

What happens when your consumer IoT device can’t connect to the internet? Do you at least provide an option to connect over a local Wi-Fi or Bluetooth network? Do you provide physical controls as a backup option for users? I think that there are many things that developers can do to create better IoT user experiences.

传感器 Sensors by Lei Gao, on Flickr (CC BY-SA 2.0).

Offline First can also be very helpful in industrial IoT settings. Take sensor data, for example. Many IoT devices transmit sensor data via the MQTT protocol. This protocol offers a “retry” feature, but what happens if the sensor is disconnected for a longer period of time? With an Offline First approach, you could store the sensor data in a queue locally on the IoT device and push this data from the queue to the cloud when an internet connection is available.


The venue for Offline Camp in the Catskill Mountains.

Offline First is still a relatively new concept. While there are a growing number of technologies to help, much of this is still a work-in-progress. So, this past June we took the Offline First community for a three day retreat in the Catskill Mountains. It was a small event with only 30 people. The goal was to help foster the growing community around the Offline First movement.

Participants were enthusiastic about the future of the Offline First movement, and came away inspired to keep building this movement. As one of my colleagues said, “It was far and away the most impactful tech event I’ve ever attended or organized”. I want to share just a few of the reflections from campers…


A partial Offline Camp schedule, handwritten in permanent marker.

Offline Camp was an “unconference” style event where the participants drove the agenda. We ended up having more than 15 sessions and 15 “passion talks” over the course of the weekend. Many of these topics have been turned into posts for the Offline Camp Medium publication. Each post was written by a participant in Offline Camp. I want to share with you a few of these stories, as they provide a good cross section of the unique perspectives as well as the challenges and opportunities in the Offline First space.

Offline Maps

Many campers were interested in offline maps. Which parts of the world do you load for offline usage? How do you ensure you’ve loaded the parts that will be needed by the user? How do you geocode, meaning search for places, when offline? The post highlights several ways that you can get involved in helping to address these challenges and others.

Building the Offline First Community

There were a couple of sessions on the topic of building the Offline First community. In the first session, we talked about the opportunity to enable others as Offline First advocates by providing tools to help them “sell” Offline first to stakeholders and to their teams. This could include sample apps, case studies, a wiki, Meetups, better utilizing the offlinefirst.org website, and helping to build the business case for Offline First.

Building the Offline First Community Through Face-to-Face Events

In the second community session, we talked about the importance of face-to-face events in building the Offline First community. This could be conferences, Meetups, or any other excuse to get people together.

Offline First UX Design Patterns

It became clear that the community needs more examples of Offline First UX design patterns. The main focus of this session was on discovering, developing, and curating UX design patterns.

The Decentralized Web

Several campers were excited about the idea of the Decentralized Web. This is a concept that goes well with Offline First. Several technologies were discussed including the Interplanetary File System (IPFS), Conflict-free Replicated Data Types (CRDTs), Distributed Hash Tables (DHT), the remoteStorage API, and the Socially Linked Data (Solid) project.

Tooling

Yet another session covered the myriad of tools that are available to help with Offline First development. This included Dash, PFSense, Wireshark, the W3C Battery Status API, OpenSignal, and Chrome Dev Tools.

Notifications and Alerts

Notifications and alerts are another big challenge for Offline First apps. For example, how do you prioritize alerts when there may be limited connectivity? How do you handle vital alerts in disaster scenarios?

Security

Last, but certainly not least, there was a great session on the security of Offline First apps. How do you validate data when replicating to the cloud? How do you handle shared computers if data is stored locally? How do you ensure that a logout truly logs out the user, deleting or at least removing access to their data? What are the best practices for encrypting data?


The Offline Camp logo.

You too can write a post for the Offline Camp Medium publication! We accept guest contributions from anyone working in the Offline First space. We’re planning another Offline Camp on the West Coast this fall, and we’ll probably host one in Europe in the spring. Look for a larger Offline First conference in the fall of 2017.


I hope that you will join me in further exploring the potential for Offline First in web, mobile, desktop, and IoT apps. Here’s how you can get involved:


Update: I’ve embedded the slide deck below.

Campfire at Offline Camp in the Catskill Mountains. Photo courtesy of Jason Pelletier.