Background Syncing — how to never miss any local change

This article is for you if 3 technical terms circle around in your mind: Concurrency, Synchronization, and Mutable Records.

When we were about to shift the syncing process of our mySugr App to the next level, we were looking for a simple, rock solid solution to never miss any concurrent modification of a locally stored record and make sure that every update gets pushed to our server consistently. Yes, there are many articles and papers out there about synchronization. And yes, TL;DR them all.

Architectural requirements of a responsive app

The first version of our app was very trivial and so was its architecture. Only a few data records were saved locally and then were transferred to our server. Nowadays, we gather hundreds and thousands of records from different sources like HealthKit, Google Fit, blood glucose meters, and manually entered values concurrently, which required us to rewrite our syncing process. Background fetching and uploading has become most important for optimal user experience.

Mutable vs. immutable Records

An immutable record is existent or not, but never gets modified. The synchronization procedure just needs to identify all unsynchronized records and send them to the server. Records that were created remotely need to be downloaded and stored locally. That’s all, no magic here.

On the other hand, mutable records can change values during their lifetime and therefore we need a method to identify any change to these records.

Naive approach

To identify changes of a mutable record, we introduced a status field which reflects the synchronization status of each record (basically, one of new/modified/deleted/unmodified). So, the synchronization procedure could fetch all candidates which are needed to push to the server. Every candidate is uploaded and afterwards its status is set to unmodified, indicating that the record is “in sync” with the server. This solution works fine as long as there are no modifications of the record during the syncing process.

But since we introduced background syncing and the amount of records increased, it is very likely that records get modified while these records are in the process of syncing. Which leads to 2 issues:

1. Missed updates: If a record’s status is set to unmodified after a successful push to the server, any concurrent local modification (i.e., modifications during the synchronization) is never picked up by the synchronization procedure and therefore never gets pushed to the server.
2. Lost updates: A pull from the server overwrites any local changes to the record. There is no option to determine if the record has been modified in the meantime.

A simple solution: Record Versioning

After realizing those issues, the idea of “every record needs a version number!” came up to our minds. With a version number, we can track all modifications made. But is this sufficient? Where should we store the version number of the modifications which are already synced? How can we make sure that nothing gets overwritten by concurrent threads?

Here, the principle of Unidirectional Data Flow will help! :-)

So we introduced two additional fields in the local database schema. These are associated to the record and never leave the local node (i.e., they are not synchronized from or to a server):

1. version: This field keeps an incremental version number that reflects local updates.
2. version_synched: This field keeps the version number that was last synchronized to the server.

The synchronization procedure then needs to follow these rules:

1. There is a separate thread for synchronization; all other updates to the records happen on other threads.
2. The version is never updated by the background thread.
3. The version_synced is only updated by the background thread to follow the Unidirectional Data Flow principle. It must not be updated by other threads.
4. Whenever a record gets updated, its version is increased by 1.
5. Periodically, the background thread fetches all records whose version is greater than its version_synced, and pushes these records to the server.
6. When the push is completed successfully, the version_synced field is set to the value of the version field.
7. After all local modifications were pushed to the server, new or updated records from the server are pulled.
8. If an incoming (pulled) record has been updated locally in the meantime, it is NOT overwritten. Instead, we rollback the pull, push all changes to the server, and then pull again (continue in step 7).

This principle works for any storage solution with a proper transaction management, like most SQL databases or Apple’s Core Data. We have been using this approach successfully in our mobile apps whenever we need to synchronize mutable records to a remote server.

Takeaways

These are the takeaways that should help you make sure that your synchronization procedure does not miss updates:

• If you are working with immutable record, just add a boolean flag to mark if it is already uploaded.
• If you are working with mutable records, use versioning with the Unidirectional Data Flow principle.

mySugr specializes in app-based, all-around care for people with diabetes — made by people with diabetes. Our apps, services, and access to the world’s best diabetes coaching at the tap of a finger all work together to ease the daily grind of diabetes. In short, we make diabetes suck less!

Interested in joining our engineering team? We’re hiring!