Scaling Offline First with Envoy
At Offline Camp, fellow IBM Developer Advocate Bradley Holt gave a Passion Talk on Cloudant Envoy. As I am more involved in this project than he is, Bradley asked me to write this summary of Cloudant Envoy in his place.
The “one database per user” design pattern makes things very easy for an Offline First application developer. Simply create a database on the mobile device and one in the cloud and get your app to read and write from its local copy. When there is an internet connection, data can be synced between the the device and the cloud.
CouchDB 2.0 and IBM Cloudant are built to scale massively on the server side and each mobile device only needs to store a single user’s data. We can use PouchDB for web apps and Cloudant Sync for native mobile apps on the client side and use the CouchDB replication protocol to sync without loss of data.
The problem comes as the number of users increases: backup, reporting and change control become problematic when there are hundreds / thousands / millions of individual databases — one for each user.
Earlier this year, faced with this scaling problem, some IBMers armed with a flip chart, some Sharpies and a code editor, set about building something to address the scalability problems with this approach.
Envoy is a Node.js micro-service that sits between the mobile devices and the Cloudant or CouchDB 2.0 cluster in the cloud, acting as CouchDB replication target. It proxies the replication requests between the client and server replicas, subtly changing the documents on the way through and storing the data in a single, server-side database.
Each mobile device still has one database per user but Envoy seamlessly stores the server side data in one database — or in two databases if you count the database of users too. Having a single store of data in the cloud makes querying, backing-up and managing the data set a breeze.
We think Envoy has potential, but it’s early days for the project and we’re looking for folks in the Offline First community to try it out, provide feedback with comments & suggestions and hopefully contribute to the codebase. It’s published under the Apache-2.0 license so we’d be more than happy for folks to get involved.
In future posts to the IBM Cloud Data Services Developer Center blog, we’ll delve into some of the technical details but for now I’ll leave you with some links:
If you have any questions then you can leave comments here or ping me in the Offline First Slack community.
Editor’s Update 12/14/16: If you’re interested in getting started with Cloudant Envoy, check out this newly published tutorial from Glynn, which will walk you through adding third party authentication (such as a Facebook login) to a static application that writes data locally using PouchDB, using Cloudant Envoy to store user data in a Cloudant database.
