Offline Storage for Progressive Web Apps

The Pokedex.org Progressive Web App uses IndexedDB for application state and the Pokemon data set while the Cache API is used for URL addressable resources.

2016 will hopefully be the year we build for network resilience.

Internet connections can be flakey or non-existent on the go, which is why offline support and reliable performance are common features in Progressive Web Apps. In this post, I’ll summarize some ideas around offline data storage for PWAs — think the JSON payloads, images and general static data required to provide a meaningful experience offline.

A recommendation for storing data offline:

For URL addressable resources, use the Cache API (part of Service Worker). For all other data, use IndexedDB (with a Promises wrapper).

Some quick answers to common questions on why:

• Both APIs are asynchronous (IDB is event based and the Cache API is Promise based). They also work in Web Workers, Window and Service Workers.
• IDB is available everywhere. Service Workers (and the Cache API) are available in Chrome, Firefox, Opera and are in development for Edge.
• While IDB doesn’t support Promises, several strong libraries giving us Promise wrappers exist. See below for recommendations. The API has mandatory complexity (transactions, schema versioning) that these libraries also try to smooth over where possible.
• Native support for IDB Promises has been proposed as have observers.
• How much can you store? In Chrome and Opera: Your storage is per origin (rather than per API). Both storage mechanisms will store data until the browser quota is reached. Apps can check how much quota they’re using with the Quota Management API. Firefox: no limits, but will prompt after 50MB data stored. Mobile Safari: 50MB max, Desktop Safari: unlimited (prompts after 5MB), IE10+ maxes at 250MB and prompts at 10MB. PouchDB track IDB storage behavior. Future facing: For apps requiring more persistent storage, see the on-going work on Durable Storage.
• Safari 10 has fixed many long-standing IDB bugs in their latest Tech Previews. That said, some folks have run into stability issues with Safari 10’s IDB and PouchDB have found it to be a little slow. Until more research has been done here, YMMV. Please do test and file browser bugs so the folks @webkit and related OSS library authors can take a look. LocalForage, PouchDB, YDN and Lovefield use WebSQL in Safari by default due to UA sniffing (there wasn’t an efficient way to feature-test for broken IDB at the time). This means these libraries will work in Safari 10 without extra effort (just not using IDB directly).
• URL addressable resources are typically static resources that surprisingly..live at a URL. For PWAs, you could cache the static files composing your application shell (JS/CSS/HTML files) in the Cache API and fill in the offline page data from IndexedDB. There are no hard and fast rules around this however. Some applications might be sufficiently simple that they can just use the Cache API alone while others may find it valuable to partially cache their JSON payloads in IDB so that in browsers without Cache API support you still get the benefit of some local caching during the session.
• Debugging support for IndexedDB is available in Chrome (Application tab), Opera, Firefox (Storage Inspector) and Safari (see the Storage tab). Debugging IDB is not currently supported in Edge (however, it is possible to debug the underlying JetDB) — vote here for built in support.

IndexedDB Libraries worth checking out

• localForage(~8KB, Promises, good legacy browser support)
• idb-keyval (<500 bytes, Promises, use if only need key-value support)
• idb (~1.7KB, Promises, also does iteration, indexing)
• Dexie (~16KB, Promises, complex queries, secondary indices)
• PouchDB (~45KB (supports custom builds), synchronization)
• Lovefield(relational)
• LokiJS (in-memory)
• ydn-db(dexie-like, works with WebSQL)

Service Worker Libraries worth checking out

• sw-toolbox (offline caching for dynamic/runtime requests)
• sw-precache (offline precaching for static assets/application shells)
• Webpack users can directly use the above or offline-plugin

What about other storage mechanisms?

• Web Storage (e.g LocalStorage) is synchronous, has no Web Worker support and is size-limited (only strings). Although ideas for async LS have been kicked around in the past, current focus is on getting IndexedDB 2.0 in a good state. I would personally use IDB + a library.
• Cookies have their uses but are synchronous, lack Web Worker support and are size-limited. In previous projects I’ve used js-cookie for handling cookies via JS (~800bytes gzipped). Support for an Async Cookies API is being sketched out right now with a polyfill in the works.
• WebSQL is asynchronous (callback-based), however it also has no Web Worker support and was rejected by Firefox and Edge but is in Chrome and Safari.
• File System API is asynchronous too (callback-based) and does work in Web Workers and Windows (albeit with a synchronous API). Unfortunately it doesn’t have much interest outside of Chrome and is sandboxed (meaning you don’t get native file access).
• File API is being improved over in the File and Directory Entries API and File API specs. A File API library exists and for file-saving, I’ve been using FileSaver.js as a stop-gap. The writable-files proposal may eventually give us a better standards-track solution for seamless local file interaction.

Current and future offline storage work

If offline storage interests you, the below efforts are worth keeping an eye on. I’m particularly excited about Promises support in IndexedDB being possible without the need for a library.

IDB with some sweet await goodness.

• Durable Storage: protect storage from the user agent’s clearing policies
• Indexed Database API 2.0: advanced key-value data management
• Promisified IndexedDB: native support for a Promise-friendly version of IDB
• IndexedDB Observers: native IDB observation without needing wrapper around the database
• Async Cookies API: async JavaScript cookies API for documents and workers
• Quota Management API: check how much quota an app/origin is using
• writable-files: allow sites to interact with local files more seamlessly
• Directory downloads: allow sites to download directories without .zip files
• File and Directory Entries API: support for file and directory upload by drag-and-drop

Resources

• State of offline storage APIs by Joshua Bell
• Browser Database Comparison by Nolan Lawson
• IndexedDB, WebSQL, LocalStorage — what blocks the DOM?
• How to think about databases (Pokedex research)
• Which APIs are supported in Web Workers and Service Workers?

Offline storage isn’t quite magical and an understanding of the underlying APIs will go far in helping you make the most out of what we now have available. Whether you prefer to directly use these APIs or work with an abstraction library, take some time to get familiar with your options.

Hopefully this will help craft an offline experience that makes your PWA ✨

With thanks to Nolan Lawson, Joshua Bell (whose work on Open Web Storage and BlinkOn talk heavily inspired this article), Jake Archibald, Dru Knox and others for their previous work in the web storage space.

Update September 2nd 2016 with some more FAQ:

What eviction gotchas exist for IndexedDB and the Cache API? Is it possible to currently guarantee that a response will always be available offline?

An origin is given an amount of space to do with as it pleases. This free space is shared across all forms of origin storage (IndexedDB, Cache API, localStorage etc). This amount given isn’t specified and will vary depending on device and storage conditions (e.g if the device is already pretty constrained).

When web storage is low (under pressure), a UA will clear storage to make space available. This can suck for offline apps and so the recently updated Storage spec defines a “persistent” and “best effort” strategy here with “best effort” being the default. “best effort” means the storage can be cleared without interrupting the user, but means it is less durable for long-term or super critical data. IndexedDB and the Cache API fall into the “best effort” bucket today.

“persistent” storage is not automatically cleared when storage is low and the user will need to manually clear out this storage (via browser settings) themselves. Chrome has been experimenting with support for Persistent Storage under an origin trial, and the latest news suggests it will be shipping in Chrome 55.

How can I tell how much storage space my app is using?

The Quota Management API lets you query for the size of storage space currently used and how much is available to the application. It also enables requesting for more storage if needed. A newer Storage Quota Estimate API tries to make it even easier to discover how much quota an origin is using with support for Promises.