Mobile syncing with Couchbase

Syncing offers a better UX than client-server for iOS/Android apps. Couchbase has an intriguing technology stack for mobile syncing.

The first stop on my European startup journey is a deep-dive with reproductive health app Clue. This week I’m working with their engineering team to explore possibilities for how to add a backend to the app.

Mobile syncing technology

So it’s 2014, and well-behaved mobile apps that connect to a backend should use a syncing model rather than a network call for every user action. Right?

Apps like Pocket, SimpleNote, and WunderList do a great job of continuing to operate normally when your internet connection is spotty or non-existent. They queue up your changes and stream them out to the network when it becomes available. The apps don’t become unresponsive or pop up weird errors when you use them on the U-Bahn (a common use case for those of us living and working in Berlin).

But much more common are apps like Facebook, Yelp, and Twitter, all of which behave erratically when your network connection is slow or unavailable. You could argue that these apps are essentially fancy web browsers to connect to a web service and syncing doesn’t make sense in these cases. But I’d prefer to see Clue’s app behave more like Pocket than Facebook when it comes to intermittent network capability.

Couchbase

Over the last few days, the Clue engineering team and I have looked at a number of services, open source libraries, tutorials, and general discussions on syncing (like this excellent slide deck). There are surprisingly few options given what a common use case mobile syncing. We did a handful of spikes to see how some of these different approaches would feel in practice.

One of these spikes had quite interesting results that I’m going to talk about here: Couchbase Mobile.

Couchbase is an open-source document database that has a confusing pedigree involving CouchDB and Memcached. But for the purposes of how the sync works, it seems to be conceptually compatible with the CouchDB multi-master replication model.

Client-server vs multi-master

It took some effort to get my head around the Couch syncing model. But once it clicked I found it quite elegant.

Client-server is what I’m most familiar with. Here you have a client, something like an iOS or Android app, or a good old-fashioned web browser. The client makes an HTTPS call to a server, probably some Ruby or Python or Go app that is running on Heroku or AWS or the Raspberry Pi in your closet. The web app can do whatever logic it wants, probably talking to a database (like Postgres, Redis, or MongoDB) for persistence. The web app returns its results and the client renders it.

This is the model that breaks down when network connectivity is intermittent. Any user action has the client waiting for the server to respond.

The Couch model differs in that it puts a Couch database everywhere you’re going to store the data. If you have an iPhone talking to a backend, that means you have two CouchDBs: one on the iPhone, and one on the backend. If you add an iPad to the mix, you now have three CouchDBs. And so on.

The lightbulb moment for me came by thinking of the difference between client-server vs master-master syncing as the difference between Subversion and Git.

With Subversion, you can’t commit changes unless you can talk to the server. Unavailable or intermittent network connectivity blocks your ability to work. Git (and other DVCSes), on the other hand, store a copy of the revision control database (as the .git directory) with every repository. So you can work continuously without network, committing as you go. Git pull & push do a sync.

Using Couchbase for mobile syncing is conceptually and architecturally similar, and offers many of the same benefits.

Couchbase Mobile

Couchbase Mobile is a stack of technologies using Couchbase as the backend data store and a series of embeddable Couchbase libraries (iOS, Android) for your clients. These clients are called “Couchbase Lite.”

A key point is that these aren’t CouchDB clients, like you’d think of with a Postgres or MySQL client library. They are complete databases which you embed into your app. (The “Lite” name was probably chosen to help you compare to SQLite, which is not a SQL client but a complete embeddable SQL database.)

These clients are usable as a local datastore all on their own. But these embeddable databases additionally have the ability to replicate themselves to a remote backend.

For our spike, I set up a Cloudant database. This worked flawlessly with the Couchbase clients, and as a bonus Cloudant has a really slick web admin dashboard that lets you run ad-hoc queries and introspect the database.

Open questions

While our team is feeling good about the Couchbase approach to syncing, many open questions remain.

Multi-tenancy

Naturally each user should only be syncing a copy of their own data.

One approach to this seems to be to do a single database per user, although this creates at least two more problems. One, queries across users become annoyingly difficult. And two, can the backend (Cloudant or a standalone Couchbase install) handle hundreds of thousands or millions of databases that will result from one database per user?

Another approach is a single database but with filtered replication. We got this working in our Cloudant + couchbase-lite-android prototype and were impressed with its simplicity and speed, but we need to do further investigation, especially around enforcing authentication and user partitioning.

User management

Following on from the above, how do you manage and authenticate users? The patterns and practices for this are very well-known when it comes to client-server HTTPS calls. It’s less obvious and well-documented with the CouchDB model. But that ties into the next point:

Backend logic

With a standard backend web app, it’s very clear where the backend logic goes. With the CouchDB model, you’re talking directly to the database. So if you want to do something like, say, send a notification to another user when a certain action happens, where does that code run?

Couchbase’s solution to this is a concept they call Sync gateway, and they have a piece of software you can use. It appears you could also write your own to sit in the same slot. Essentially it’s a proxy to the database that receives replication requests from the client and can decide what to do. Here you can do authentication or other user management, and pass on replication calls to the database when appropriate. We haven’t had a chance to try this out yet so I can’t speak to how well it works, but I like the concept.

Conclusions

Based on our few days of experiments and research, the Clue team and I have the following conclusions:

• Syncing produces a far better-behaved mobile app than client-server. Anyone making a mobile app should favor the former over the latter.
• …but syncing practices and libraries/frameworks are spotty. There aren’t a lot of mature, maintained, well-documented options.
• Trying to write your own syncing is probably a bad idea unless you have a lot of resources and syncing is pretty core to your business. It’s a hard problem that is likely a huge distraction for your technology team.
• The CouchDB multi-master replication offers a good model for mobile syncing that is conceptually similar to how Git works for revision control.
• Couchbase Mobile is a nice stack of technologies that sees to include everything you need to do mobile syncing.

But we have far more questions than answers at this point. We’ll be continuing to explore this area over the coming weeks.

Are you interested in this type of problem? If so we’d love your help. Drop me a line at adam@helloclue.com if you have any thoughts or comments on this article, or if you’re interested in joining our team as a backend engineer. CouchDB enthusiasts & experts particularly welcome.