Libraries of Note: Sharing Open Source Code From the Guardian Mobile Innovation Lab’s Storytelling experiments
A guide to nearly all of the code written during two years of experimentation with mobile-first storytelling formats.
We wrote a lot of code in the two years of the mobile lab and, wherever possible, we open sourced that code to support our team’s mission of openness and collaboration, so that others can use it, learn from it, or just satisfy their curiosity by working out how we did something. But GitHub isn’t the easiest place to discover which code does what, so for posterity, here’s a quick rundown of the most noteworthy code we’ve published, with notes about what each does.
Service Workers
worker-commands
This is a JavaScript library that bundles together a collection of useful commands you might want to run inside a service worker, through an interface that allows you to execute code in response to notification clicks and push events. Some form of this code was used on the vast majority of our web notification projects (the ability to update the worker in response to a push was a lifesaver).
node-service-worker
This is a node module that allows you run basic service worker scripts. We created it to help solve the “first load” problem with service workers — the first page load a user makes won’t run through the worker, because it isn’t installed yet, so you have to duplicate page rendering code in the worker and on the server. By wrapping the worker code, you only need to keep this logic in one place.
If you wanted to use it for something similar, you might find a use for…
node-service-worker-proxy
Another node module that attaches node-service-worker to an HTTP server, sending HTTP requests directly into the worker and returning the response generated by the worker. It comes with a Dockerfile, letting you run self-contained instances of the server on any container service. We used this to provide the first page load during our Shifting Lenses experiment.
SWWebView
This is a module for iOS development, written in Swift, that adds service worker functionality to the WKWebView class, and can be used as a drop-in replacement for it. Since we worked on this, Apple announced that service worker support was coming with iOS 11.3, but those worker environments are not accessible outside of a webview, and don’t support features like the Push API. If you’re interested in extending the service worker environment, or interacting with its APIs natively (for example, adding items to CacheStorage preemptively) you might want to take a look at this.
Push Messaging
pushkin
This is a collection of AWS Lambda commands for sending push messages to a lot of Push API clients simultaneously. It uses the Serverless framework to bundle and deploy all of these functions.
After using it for a few experiments (Brexit results and monthly US jobs report) we found that it wasn’t as reliable as we wanted. Some pushes took a long time to be delivered (if they were delivered at all) and we struggled to work out why. So in the end, we abandoned it in favour of…
pushkin-firebase
Once Firebase added support for web notifications alongside their iOS and Android offerings, we switched away from Pushkin to a new iteration that wraps the Firebase API, leaving them to handle the difficult part of actually delivering push messages at scale. It uses topics — letting users subscribe and unsubscribe to different topics, and you to push to one or many topics on demand.
It is Node based, and comes with a Dockerfile for those who want to run it on something like the Amazon Elastic Container service, which is what we did.
pushkin-client
A client library for the above. Wraps operations like subscribing and unsubscribing into a simple Promise-based API.
Image manipulation
png-pong
This is the library used for our UK general election image notifications. Service workers don’t have access to the Canvas API, so we had to create png-pong to manually create images by manipulating the array buffer of a PNG file. It can draw lines, rectangles and copy sections of other image files.
It was structured to take advantage of the ReadableStream API to use as little memory as possible, but we ended up not being able to get that functionality ready in time, so for now it has to load the whole image into memory.
png-pong-font
A plugin for png-pong that allows you to add text to image files. It uses a custom bitmap font file format (and has a tool to help you create those source files).
Developer tools
mock-aws-sinon
We needed to run automated tests on our code that used the aws-sdk node module, but we couldn’t stub the functions because aws-sdk is written in a strange way that sinon (the stubbing library we were using) doesn’t expect. This library lets you use the two together.
It’s also had contributions from a few developers outside of the lab, so hopefully it has life in it yet.
treo-promise
IndexedDB is an irritating API to work with, and Treo was the smallest library I could find that wraps it into something sensible. But its promise support is broken, so I extracted it and fixed it. Treo hasn’t been updated in years (but also doesn’t need to be — the IndexedDB API isn’t changing!), so this library should continue to work.
Additional developer resources
During the lab’s two years of experimentation we also published other resources for news developers who were interested in learning about, or replicating our work. A collection of those developer resources are here, and also below.
- Taking Web Podcasts From Hack to Production: The highs and lows of building a new podcast interface, by Alastair Coote.
- How to replace the content of an iOS notification: How to make the most of iOS 10’s new notification features, by Alastair Coote.
- Generating Images in JavaScript Without Using the Canvas API: And putting them into a web notification, by Alastair Coote.
- Best Practices for Using Notifications and Service Workers on a News Site: Tips based on our initial experiments for the UK election, by Alastair Coote.
- Podcasts Without Apps: Building a fully featured podcast player for mobile web, by Alastair Coote.
- News App, Justify Thyself: How and why we built a Guardian Mobile Innovation Lab app, by Alastair Coote.
- Structured, Queryable Logging with Postgres and Bunyan, by Alastair Coote.
- What We Learned Developing the Good News/Bad News Notifications: Some technical learnings about interactive notifications, by Alastair Coote.
- Creating our Web Push Service: A technical look at the building process behind the Guardian Mobile Lab’s web notifications, by Alastair Coote.
- Creating Our Service Worker: Building the system behind the Guardian Mobile Innovation Lab’s web notifications for news, by Connor Jennings.
The Guardian US Mobile Innovation Lab was set up as a small multidisciplinary team housed within the Guardian’s New York newsroom from 2015–2018. Its mission was to explore storytelling and the delivery of news on small screens, and share what they learned. It operated with the generous support of the John S and James L Knight Foundation.
