MongoDB Realm Flexible Sync: A primer by an engineer, for an engineer (Part 1)
Hi! My name is Tyler and I am an engineer on the Atlas Device Sync team.
I am incredibly happy and excited to finally put Flexible Sync into users’ hands. Although Partition Sync is powerful for a subset of use cases, we always knew that we wanted to ultimately remove the partitioning limitation and give users a more “flexible” syncing experience.
I am incredibly proud of the work that our team did over the past few years to make it to this point and want to share some tips/tricks for getting started.
Note: This article is split into two parts:
Part 1: What and Why
- What is Atlas App Services?
- What is Realm?
- What is Device Sync?
- Why do I want Device Sync?
- Configuring sync for your application (Schemas, Permissions, Subscriptions)
- Under-the-hood look at how changes flow through the system
Table Of Contents:
- Overview (Part 1)
- Schemas (Part 2)
- Queryable Fields (Part 2)
- Subscriptions (Part 2)
- Permissions (Part 2)
What are Atlas App Services and Realm?
Atlas App Services is a platform to make building applications with MongoDB fast, easy, and scalable. It contains features such as:
1. Authentication and session management
2. Database Triggers: run functions in response to modifications made in MongoDB
3. GraphQL Service
4. Device Sync
There are many other offerings on the platform, but those are just a few to highlight. To see the whole suite of services offered by Atlas App Services see the diagram below:
What is Realm?
Realm is an open-source on-device object database: a replacement for Core Data & SQLite. It is:
- Optimized for performance on a mobile or embedded device (IoT)
- Designed to boost developer productivity
- Built to run cross-platform (Android, iOS, Node, Xamarin, Flutter, etc)
It is a competitor to things like Core Data and SQLite if you have heard of those.
What is Device Sync?
Sync was designed from the ground up to provide a best-in-class experience for synchronizing data between mobile devices and MongoDB Atlas.
In order to build a good mobile application, you need to have a robust data layer. Handling the happy path of no latency reads/writes, no disconnectivity, and no failed writes is easy; however, that is not the world we live in. Modern mobile applications need to handle:
- Intermittent network connectivity
- Failed / Invalid Writes
- Multiple Concurrent Writers
Taking a quick step back, let's first define the goals of Device Sync:
1. Immediate propagation of changes to connected devices:
Any time a change is made either from a mobile device or from an external MongoDB client (shell, driver, etc), the change is immediately sent to all relevant mobile devices if they currently have a connection to Atlas App Services.
2. Online-Always Data:
The sync protocol allows devices to behave normally during network interruptions and automatically resume uploading / downloading changes from App Services when it regains internet connection in such a way that no changes are dropped or missed. This lets applications be designed to read and write data locally in Realm on the device and the data in Realm will be updated asynchronously when needed / possible.
No more tricky networking logic, retry logic, or backoff logic, it is all implemented for you with just a few lines of setup code.
3. Conflict-Resolution:
- If multiple devices are updating the same underlying MongoDB document, Sync performs conflict resolution using the Operational Transformation algorithm.
- For simple updates such as Update(“field1”, 123) we use a “last writer wins” policy; however, when using lists, more intelligent strategies are used to preserve the intent of the user’s change and incorporate both writes (think of two devices inserting an element into the same list)
- To see why conflict resolution is needed in the first place, trace through the example below
- For more information about the conflict resolution performed by realm see here
4. Seamless Integration into Mobile SDKs
Once the cloud is set up with schema models, permissions, and authentication settings, hooking up your mobile application to sync data is simple and magical.
5. Scalability and Tooling
Sync is architected to maximize the throughput of the system and allows developers to pay only for the amount of sync traffic that they generate. Additionally, when using sync you have access to logging, metrics, alerting, security, and a whole suite of other helpful features for building an application.
6. Flexible Sync vs. Partition Sync
The previous iteration of Device Sync was called Partition Sync which allowed you to specify a specific field as the “partition key” across all schemas and devices could sync on specific partition values. For certain use cases, this was very valuable but had serious limitations from a permissioning and data modeling perspective. If you have not used partition sync, feel free to skip this section but I will highlight a few of the main highlights of Flexible Sync:
- Expressive Subscriptions: Devices can leverage the full power of the Realm Query Language (RQL) to specify what data should be synced. Additionally, being able to query on values in a list/array is very powerful and allows for more intuitive data modeling.
- Fine-Grained Permissions Model: Instead of pre-partition permissions, can have per-document permissions as well as field-level permissions to control what data is sent to the devices and what data is allowed to be modified from devices
- Optimized Bootstrap Time: The architecture of the system was redesigned in order to optimize the time it takes to download the initial state of data to the device
- Simplified History Compaction: Instead of a complicated compaction scheme it has been made both simpler and more effective which reduces both your Atlas storage costs and reduces the load on the cluster
- Faster Initial Sync: Enabling sync for the first time will insert all data in your synced collections into the “sync history” and this process has become much faster
- Other: There are many benefits to some of the design choices made in Flexible Sync that will cut down on some pesky bugs (Bad Changesets), improve performance and throughput of the system, and reduce developer friction with getting started.
With that, you are ready to get syncing. Please see Part 2 for more details on how to properly enable and use sync, as well as some under-the-hood details on its implementation.
I am so happy that we can bring the power of Flexible Sync to developers and am even more excited about some of the many future projects we have planned, so stay tuned!!
I hope that you enjoy sync and that it can make you, your organization, and your customers more productive!
Tyler
