Syncing reference data with Couchbase sync gateway

Here at Ryanair, we are using Couchbase Lite and Couchbase Sync Gateway to help us manage reference data in the mobile apps, without the need to deploy new versions and without the bandwidth penalty of requesting them from a REST API dynamically.

How we do that, is by synchronizing revisions of our resource documents from the cloud to the device, with the help of sync gateway. This approach is great for semi-static data, even if the device fails to sync for some reason (being issues with the infrastructure or connectivity), our applications are still responsive, with the tradeoff that we might have stale data on the device. In most cases this is not a problem.

You can find a working example here

Ryanair/resource-sync-example

If your application consumes JSON documents from a REST API, you can implement a more permanent cache relatively easy . I am going to share some of the things we found work good for us.

The recommended way of working with the database is to create a singleton object in your Application class.

One important thing to keep in mind is to separate the replication from the database handler. As you can see in the github repo above, the class StorageManager, takes care of the the database interaction, like creating, opening and replication, which can have unwanted side effects in production. The problem is with the replication.

Let’s say we have an app that lists all the coffee shops in the region. The list of the coffee shops is stored in the form of Couchbase documents and synced via the Sync gateway to our app. Because we want to have the most recent data as soon as it’s available on the server, we have set up a continuous pull replication, which is started in the StorageManager constructor. This is ok for applications that can run only in the foreground because as soon as the application is stopped, the replication stops with it and the connection with the server is closed. What happens if we want to add a sticky service that tracks our location and triggers a notification when we are close to a coffee shop? Because our database is coupled with the replication, we will have a constant connection with the server, which is definitely something we want to avoid. Here is an illustration of a better approach, that will split the replication from the database object and will give us better control over the process

public class AppController extends Application {
    private static final String TAG = "AppController";

    private static Database database;

    @Override
    public void onCreate() {
        super.onCreate();

        openDatabase();
    }

    private void openDatabase() {
        Manager manager;

        if (!Manager.isValidDatabaseName(Constants.DATABASE_NAME)) {
            Log.e(TAG, "Bad database name");
            return;
        }

        try {
            manager = new Manager(new AndroidContext(getBaseContext()), Manager.DEFAULT_OPTIONS);
        } catch (IOException e) {
            Log.e(TAG, "Cannot create manager object");
            return;
        }

        try {
            database = manager.getDatabase(Constants.DATABASE_NAME);
        } catch (CouchbaseLiteException e) {
            Log.e(TAG, e.getMessage());
        }
    }

    public Database getDatabase() {
        if(database == null) {
            openDatabase();
        }

        return database;
   }

}

And our replication controller

public class ReplicationController {
    private static final String TAG = "ReplicationManager";

    public void startContinuous(Database database) {
        URL url;

        try {
            url = new URL("https://sync.example.com:4984");
        } catch (MalformedURLException e) {
            Log.e(TAG, e.getMessage());
            return;
        }

        Replication replication = database.createPullReplication(url);
        replication.setChannels(Arrays.asList(Constants.channels));
        replication.setContinuous(true);
        replication.start();
    }

}

Now we can start replication easily by invoking the startContinuous method and passing the database object

View.OnClickListener onContinuousClick = new View.OnClickListener() {
        @Override
        public void onClick(View v) {
            ReplicationManager manager = new ReplicationManager();
            manager.startContinuous(app.getDatabase());

            Toast.makeText(getBaseContext(), "Continuous replication started", Toast.LENGTH_SHORT).show();
        }
    };

When our Activity gets killed, replication will stop, even if the database object is still in memory.

For this reason, we should treat replication and the database as two different entities. For more control, I recommend implementing a push notification service, that wakes up your app, syncs the data and then stops. A good example can be found on Couchbase’s blogs

Generally if you are dealing with semi-static data, that gets updated rarely, it’s an overkill to use continuous replication, as it might have a negative impact on battery life as well as increase infrastructure costs.

Another thing to consider when you are designing your resource repository, is to properly partition your data with channels. For our coffee shop app, a good example would be to have channels based on regions, which can allow you to sync the data only for the region the user is in. The perks of this technique are obvious, we don’t have to sync our full dataset to the user’s device, as the list of coffee shops we can have in our bucket may become huge.

Implementing sync for our mobile apps can be a very straight-forward task and significantly boost performance, if implemented properly. Planning the replication strategy and document structure are important and can bring a lot of value to the product. You can check the video below, for a side by side comparison of our old and new apps and the performance boost we gained after implementing this “smart” caching