From Mobile First to Offline First
Provide a better, faster user experience—both offline and online—with an Offline First approach.
What follows is a narrative version of my Abstractions presentation From Mobile First to Offline First. This is not a new presentation topic for me. I recently helped to organize Offline Camp. I’ve been advocating for an Offline First approach to building web and mobile apps for over a year. I’ve given talks on Offline First apps and related technologies at numerous conferences including at Apache: Big Data, O’Reilly Fluent, ConFoo, CodeMash, Node.js Interactive, All Things Open, NoSQL Now!, That Conference, OSCON, Cloud Expo, and php[tek].
Somehow, Abstractions feels like a different type of event, and I wanted to take the opportunity to rethink my Offline First presentation. First, the speaker lineup is amazing. Second, the message and community around Offline First has started to come together in a meaningful way. This is the right audience, at the right time, to spread the message about Offline first. So, here it is…
Let me walk you through how I start my day most mornings. I work remotely for a large enterprise, so of course the first step is to login to my corporate VPN. Every. Single. Morning.
Next, I login to my web-based email client. Every. Single. Morning.
Finally, I need to reload my browser tab containing my calendar application. Every. Single. Morning. For security, of course. The whole process of getting to a place of productivity can take a minute or two on a good day, and even longer if there’s a problem with cookies or some other technical glitch. This is especially fun if I’ve got a conference call first thing in the morning.
There are a lot of, well, interesting ways in which web apps can treat a session expiration. I find this unnamed web app particular fun. What am I supposed to do here? Why can’t my data just be saved locally and then re-synced with the server once I login again?
It’s not just enterprise software. If my laptop has been closed for any period of time then I often get a “could not connect to Trello” message on my Trello boards, even if my session hasn’t expired. I then need to reload the page to ensure that I have the latest updates and that new changes that I make are not lost. As you can see, my day begins with lots of browser tab reloading.
This is not atypical for web-based apps. Historically, we’ve treated web browsers as thin clients for our server-side apps. In this model, the browser’s job is simply to render our server-side generated HTML. We’ve slowly moved more-and-more functionality to the client side, but we still don’t completely trust web browsers as app platforms.
But why not? Before logging in to my VPN, logging in to my email client, and reloading my calendar tab I had already logged in to my computer. My computer is owned and managed by my employer. Once I’m logged in to my computer, why should I need to login to anything else after that?
In addition to my laptop, I also have a corporate-managed iPhone. At any moment I can check my email or look at my calendar almost instantly by logging in to my phone using either Touch ID or by entering my passcode. It takes me a total of about three seconds before I can view my email or calendar.
My smartphone is not treated like a thin client, but rather like a, well, smart client. Logins are persisted in most apps, as the phone’s built-in security features are trusted by these apps. Data is often stored locally, allowing me to use my apps instantly, regardless of connectivity.
Mobile platforms such as iOS are heavily controlled by vendors. As a result, these platforms provide nice, cozy environments for developers to work within. The web, on the other hand, is a radically open platform with no centralized control. Web development is messy business. Screen sizes, browser compatibility, antiquated browsers, rapidly evolving web standards. These are just a few of the complications that web developers contend with on a daily basis.
When Luke Wroblewski articulated the Mobile First approach to web development he outlined an approach that was all about embracing, rather than fighting, the lack of control experienced by web developers. Have no idea what size device your end user will be using to browser your website? That’s OK! Don’t stress about it. Techniques such as responsive web design and fluid layouts allow you to embrace the chaos that is web design and development, and to create better apps out of this chaos.
Letting go of control is hard. Most of the time, though, we’re not actually letting go of control. What we’re really letting go of is the illusion of control. Even when it’s just an illusion, though, a sense of control can help us feel safe in a chaotic world.
As developers, we need to make assumptions. Even with a Mobile First approach, we decide on the smallest device size that we will support. This is the art of progressive enhancement: start with an assumption of base-level capabilities and take advantage of more capabilities as they become available.
I like to think of progressive enhancement in the abstract. To me, it doesn’t matter as much what baseline you pick. It can still be progressive enhancement if you pick a baseline and then progressively enhance from that baseline. Just be sure to put some thought into what baseline makes sense for your context and your app.
Speaking of assumptions and baselines, it’s easy to think of network connectivity as being ubiquitous. App developers often take having a network connection for granted, an assumed baseline from which to progressively enhance. If not ubiquitous, at least mostly reliable, right? OK, maybe not mostly reliable but at least you can know whether or not your app has connectivity, right? Along similar lines, it’s really easy to make assumptions about latency and bandwidth.
The problem is that these are all false assumptions. If you haven’t read it before, the The Eight Fallacies of Distributed Computing is a great resource on this topic. Let’s take a look at some scenarios…
Users of your app are obviously offline when in airplane mode. That’s an easy one. Sure, many flights now offer Wi-Fi, but not typically below 10,000 feet. Also, what happens when your plane’s Wi-Fi doesn’t work? Or, what about those of us who don’t want to pay for the onboard Wi-Fi?
A few weeks ago, I had the privilege of attending and participating in an important moment in the technology world and…medium.com
Developing countries don’t have the same level of ubiquitous connectivity as we have here in the United States. If you have any interest in these markets then you can’t take network connectivity for granted. Joel Worrall works on HospitalRun, which is an open source software project for developing world hospitals. He recently had this to say:
However, this issue becomes a business imperative when bringing products and services to a market with less reliable, more costly, or even less available connectivity. In markets such as Kenya, with tremendous opportunity for growth due to smartphone sales, the demand to address the Offline First problem in mobile applications is quickly becoming critical.
Many wireless plans include data caps. It’s important to limit the bandwidth usage of your app if at all possible so as not to eat up all of your users’ data for the month or cost them money on data overage charges.
Even in large cities with great connectivity it’s not uncommon to temporarily lose connectivity in subways or to find other areas where wireless signals are blocked by buildings or other obstructions. If your app requires a network connection in every moment, then the user experience will suffer when there is the inevitable blip in connectivity.
You’ve developed your app on your local computer, in a controlled environment, connecting to a development version of you mobile backend that’s also running on your local computer. It’s so fast! Super-low latency! Once you ship your app to real users, though, in the real world of slow, unreliable networks it’s a completely different story. Your users are frustrated and unhappy with your app.
Then there’s “Lie-Fi.” You’re using an app and the device says that you’ve got a connection, but your message just won’t go through. Your device thinks it’s connected, but it can’t actually send or receive any data. You may have a great connection to a Wi-Fi access point, but its connection to the internet is broken. This is the distributed nature of networks. Each hop in the network needs to work or your message won’t go through—and you won’t know if the network works or not until you try to send the message. You can’t just indiscriminately trust an “online” device status.
I hope that I’ve convinced you that you shouldn’t build your apps to rely on a network connection. But what’s the alternative? You have two basic options: graceful degradation or progressive enhancement. Graceful degradation involves building your app as usual, assuming that it will have a good network connection, and then gracefully handling the resultant errors when the network connection inevitably does not work. The progressive enhancement approach involves assuming that, as a baseline, your app will not have a working network connection. Then, your app can be progressively enhanced to take advantage of network connectivity when available.
The progressive enhancement approach is what we call Offline First. Don’t assume that lack of connectivity is an error condition. Build your app to work without a connection. Get updated content, sync your data, or enable features that aren’t practical to make work offline when the app is connected. Let users still interact with your app when there’s no reliable connection available.
We assume our users are like us-with the latest devices, the most recent software, and the fastest connections. And…alistapart.com
If you’ve heard of Progressive Web Apps then some of what I’ve talked about probably sounds familiar. The basic idea of Progressive Web Apps is to combine the discoverability of web apps with the power of native mobile apps. As an end user, you browse to a Progressive Web App just like you would browse to any other website. As you use the app more-and-more, it gains additional native-app-like capabilities. For example, installation to your home screen or the ability to send you alerts and notifications.
Google is currently making a big push for Progressive Web Apps and many companies have invested in building Progressive Web Apps. For example, The Weather Company, which IBM recently acquired, has built a Progressive Web App which allows them to send alerts and notifications to users.
Progressive Web Apps are an opportunity for web developers to rethink how we approach building web apps. The market has clearly demonstrated that it likes the interaction model of native apps. There’s a lot that we can learn from how native apps are built.
Don’t get me wrong, I’m a huge believer in the power of the open web. I don’t think we should just turn web apps into native apps and call it done. I do think that we can incorporate some of the best parts of native apps into how we build web apps while not losing the soul of the open web.
There are many Offline First technologies available that you can start using right now. The newest and biggest development in Offline First for web developers is the Service Worker specification. You may remember the HTML5 Application Cache (AppCache), which was the first attempt at providing a browser mechanism for caching content and assets for offline usage. There were numerous issues with the AppCache and it has since been deprecated by most browsers. Service Workers is a new approach that aims to solve the same basic problem, while avoiding the pitfalls of AppCache. Service Workers give you fine-grained control over caching content and assets for offline usage by intercepting client requests and returning previously-stored results when offline.
localForage - Offline storage, improved. Wraps IndexedDB, WebSQL, or localStorage using a simple but powerful API.github.com
Service Workers are awesome for enabling offline usage of content and assets. But, sometimes having an actual database can be helpful. This is where IndexedDB, WebSQL, or localStorage can be helpful. Or, at least they would be helpful if browsers implemented these standards consistently. The browser support and implementation quirks can make it very difficult to actually use these technologies. Fortunately there’s localForage, a wrapper for IndexedDB, WebSQL, or localStorage which provides a unified API.
It enables applications to store data locally while offline, then synchronize it with CouchDB and compatible servers…pouchdb.com
I encourage you to also check out PouchDB. It’s not as lightweight as localForage, but it includes a bunch of awesome features that are probably worth the extra overhead. The most exciting feature of PouchDB is its ability to sync with anything that implements the CouchDB Replication Protocol. This, of course, includes Apache CouchDB. It also includes IBM Cloudant, a fully-managed cloud database that is based on CouchDB. PouchDB allows you to read and write data directly on the device, and then sync this data to the cloud when you have an internet connection. This makes create, read, update, and delete operations extremely fast as all of your data access happens locally.
tessel-pouchdb-demo - Quick demo of PouchDB running on a Tessel.github.com
sync-android - A JSON-based document datastore for Android applicationsgithub.com
Our team at IBM Cloudant also created open source Cloudant Sync libraries for iOS and Android, in case you prefer to build native mobile apps and would like to take an Offline First approach. While they have different APIs than PouchDB, these libraries, like PouchDB, store data locally on the device and can also sync with anything that implements the CouchDB Replication Protocol.
It's a complete backend for your apps: develop your frontend code, plug it into our API and your app is ready.hood.ie
There are still many challenges to be addressed around Offline First. A common pattern for developers using PouchDB or Cloudant Sync is to create one database per user. Each user is given their own local database on their device, as well as their own database in the cloud with which only they can replicate. This works great if you have an app where each user’s data is fairly well segmented. However, as soon as you want to allow a user to access another user’s data, or you want to create aggregate queries across multiple users’ data, the one-database-per-user pattern starts to break down.
Cloudant Envoy is an open source project that our team created to give developers the best parts of one-database-per-user without the pain of one-database-per-user. Cloudant Envoy is effectively a proxy which provides the illusion of one-database-per-user to the PouchDB or Cloudant Sync database on the user’s device, but it actually stores all of the data in one CouchDB or Cloudant database. An access control list is maintained by Cloudant Envoy which dictates who can access what data. This is still an experimental project, but please take a look and let us know how it works for you.
Our Developer Advocacy team has built several Offline First sample apps to help you learn how to build Offline First apps. Cloudant FoodTracker is an app that we created to demonstrate how to build an Offline First iOS app in Swift using Cloudant Sync. Apple provides a great tutorial on starting to develop iOS apps in Swift. The tutorial walks readers through creating a simple meal tracking app. The app allows users to view and edit a list of meals. Each meal includes a meal name, rating, and a photo.
Apple’s FoodTracker is offline-only, meaning that all of the meal data is stored only on the device and the app provides no means of syncing that data to the cloud. Cloudant FoodTracker replaces the data layer with Cloudant Sync and then syncs with Cloudant or CouchDB.
Field Work is a web app that allows offline editing and mapping of geospatial data. The idea of Field Work is to demonstrate how a company in a field-based industry, such as the utility industry, could benefit from an app which their field personnel could use even when disconnected.
Field Work stores GeoJSON locally on the device in PouchDB. It uses Leaflet and Mapbox.js to render the geospatial data on a map. Read-only base map layers are synced from a Cloudant database to a PouchDB database on the device. An editable map layer is then synced between a separate Cloudant database and a separate PouchDB database on the device. There are other aspects to the app, including the ability create work orders.
Location Tracker is another iOS sample app, written in Swift. The app tracks a user’s location as they move and stores this data in a local Cloudant Sync database, which is synced with Cloudant when an internet connection is available. As a user moves, and new locations are recorded, the app queries the server for points of interests near the user’s location. Blue pins mark each location recorded by the app. A blue line is drawn over the path the user has travelled. Each time the Location Tracker app records a new location a radius-based geo query is performed in Cloudant to find nearby points of interest.
I’ve talked about Offline First primarily in the context of Progressive Web Apps. However, Offline First is a concept that is applicable to any type of app that would typically depend on an internet connection. I’ve touched on how an Offline First approach can be used for native mobile apps, for hybrid mobile apps using tools such as Apache Cordova and Ionic, for desktop apps using Electron, and even for IoT apps using Tessel. I’d like to spend a bit more time talking about IoT as this a space that could benefit quite a bit from some Offline First thinking.
What happens when your consumer IoT device can’t connect to the internet? Do you at least provide an option to connect over a local Wi-Fi or Bluetooth network? Do you provide physical controls as a backup option for users? I think that there are many things that developers can do to create better IoT user experiences.
Offline First can also be very helpful in industrial IoT settings. Take sensor data, for example. Many IoT devices transmit sensor data via the MQTT protocol. This protocol offers a “retry” feature, but what happens if the sensor is disconnected for a longer period of time? With an Offline First approach, you could store the sensor data in a queue locally on the IoT device and push this data from the queue to the cloud when an internet connection is available.
Offline First is still a relatively new concept. While there are a growing number of technologies to help, much of this is still a work-in-progress. So, this past June we took the Offline First community for a three day retreat in the Catskill Mountains. It was a small event with only 30 people. The goal was to help foster the growing community around the Offline First movement.
What’s the secret to creating a meaningful live experience for a tech community?medium.com
Participants were enthusiastic about the future of the Offline First movement, and came away inspired to keep building this movement. As one of my colleagues said, “It was far and away the most impactful tech event I’ve ever attended or organized”. I want to share just a few of the reflections from campers…
Offline Camp was an “unconference” style event where the participants drove the agenda. We ended up having more than 15 sessions and 15 “passion talks” over the course of the weekend. Many of these topics have been turned into posts for the Offline Camp Medium publication. Each post was written by a participant in Offline Camp. I want to share with you a few of these stories, as they provide a good cross section of the unique perspectives as well as the challenges and opportunities in the Offline First space.
What’s the first app you ever used on a smartphone?medium.com
Many campers were interested in offline maps. Which parts of the world do you load for offline usage? How do you ensure you’ve loaded the parts that will be needed by the user? How do you geocode, meaning search for places, when offline? The post highlights several ways that you can get involved in helping to address these challenges and others.
Building the Offline First Community
The idea for Offline Camp was born out of the desire to connect the community and bring folks together from different…medium.com
There were a couple of sessions on the topic of building the Offline First community. In the first session, we talked about the opportunity to enable others as Offline First advocates by providing tools to help them “sell” Offline first to stakeholders and to their teams. This could include sample apps, case studies, a wiki, Meetups, better utilizing the offlinefirst.org website, and helping to build the business case for Offline First.
Building the Offline First Community Through Face-to-Face Events
Next steps for expanding our reach through live experiencesmedium.com
In the second community session, we talked about the importance of face-to-face events in building the Offline First community. This could be conferences, Meetups, or any other excuse to get people together.
Offline First UX Design Patterns
Design Patterns make expectations, intentions, and contexts understandable. Things you can’t ignore, but especially…medium.com
It became clear that the community needs more examples of Offline First UX design patterns. The main focus of this session was on discovering, developing, and curating UX design patterns.
The Decentralized Web
I recently attended the first Offline Camp, a gathering of a very diverse group of people that were interested and…medium.com
Several campers were excited about the idea of the Decentralized Web. This is a concept that goes well with Offline First. Several technologies were discussed including the Interplanetary File System (IPFS), Conflict-free Replicated Data Types (CRDTs), Distributed Hash Tables (DHT), the remoteStorage API, and the Socially Linked Data (Solid) project.
There are tons of tools out there, both proprietary and open source, whose sole purpose is to make app developers more…medium.com
Yet another session covered the myriad of tools that are available to help with Offline First development. This included Dash, PFSense, Wireshark, the W3C Battery Status API, OpenSignal, and Chrome Dev Tools.
Notifications and Alerts
Ideas for building notifications and alerts that enable quality user experiences in a semi-connected worldmedium.com
Notifications and alerts are another big challenge for Offline First apps. For example, how do you prioritize alerts when there may be limited connectivity? How do you handle vital alerts in disaster scenarios?
At Offline Camp I participated in a session on security of Offline First web apps. We were able to pick out a couple…medium.com
Last, but certainly not least, there was a great session on the security of Offline First apps. How do you validate data when replicating to the cloud? How do you handle shared computers if data is stored locally? How do you ensure that a logout truly logs out the user, deleting or at least removing access to their data? What are the best practices for encrypting data?
You too can write a post for the Offline Camp Medium publication! We accept guest contributions from anyone working in the Offline First space. We’re planning another Offline Camp on the West Coast this fall, and we’ll probably host one in Europe in the spring. Look for a larger Offline First conference in the fall of 2017.
I hope that you will join me in further exploring the potential for Offline First in web, mobile, desktop, and IoT apps. Here’s how you can get involved:
- Join the Offline First Slack team.
- Follow @OfflineCamp on Twitter.
- Read the Offline Camp Medium publication.
- Vote for and comment on our Offline First SXSW panel, featuring Maureen McElaney, Gregor Martynus, Dan Zajdband, and Nolan Lawson.
- Join us at Offline Camp on the West Coast, in Europe, or at the planned Offline First conference!
Update: I’ve embedded the slide deck below.