Synchronizing Online/Offline States

At Samsara, we deploy hundreds of thousands of sensors in vehicles and factories. One of the ways our customers interact with our platform is via a React Native mobile app. To support our customers’ operations in areas without Internet connection, our engineering team built SyncStore, a framework that the app uses to synchronize local and remote states as the mobile app switches between online and offline environments.

Introducing SyncStore

At Samsara, we use React Native to build our mobile apps and Redux to manage React states. Redux actions are dispatched when our users interact with our app.

If you haven’t heard of Redux reducers and actions, reducers are functions that map a state and an action to a new state.

On-the-road compliance is a critical function of our app and thus our users submit many reports. For an app that supports report submissions, an action could be an object with all data associated with a new report, and the state could be a list of all submitted reports.

Our drivers rely on our products to log their driving hours and perform vehicle inspections, and they need to be able to do them reliably at all times with or without an Internet connection to stay compliant with government regulations.

This is a hard problem because the driver on the road can mutate the state offline and then their fleet manager in the back-office can also update the backend state. This means that states can easily be out of sync.

Figure 1: States can go out of sync when drivers edit logs offline and admins edit the same logs online

Let’s see how SyncStore makes synchronizing easier.

When actions are dispatched, two things happen:

1. Idempotent reducer computes the new state (why idempotent? we’ll see)
2. The actions are put on a persistent queue backed by AsyncStorage

This lets users continue using the app offline and also lets the app recover the most recent state in case the app is closed or crashes. When the mobile device regains the Internet connection, the actions in the persistent queue are batched and uploaded to the backend over a WebSocket connection.

These requests are routed to the backend’s GraphQL mutation handlers, which compute the new ground-truth state by processing the uploaded actions. This new state is then persisted to the Aurora cluster in the backend.

Back to the mobile app: after the action queue upload succeeds, the app needs to download the new ground-truth state and finish the synchronization. At some point, we also want to delete completed actions to reclaim storage.

This turns out to be a challenge for two reasons:

1. WebSocket subscription is a push model — i.e. state download is a server-side push and can be triggered by an unrelated mutation.
2. State is written to the Aurora DB master, but state downloads could be served by any read-only Aurora replicas — i.e. the change may not have fully propagated to all the replicas.

With that context, a downloaded state can either be “behind” (server hasn’t yet applied the uploaded actions) or “ahead” (server has already applied all the uploaded action). Because actions/reducers are idempotent, we don’t have to worry about applying same actions twice when the state is “ahead”, but we do need to be careful about potentially deleting actions too early.

We get around this by forcing a download by re-establishing the WebSocket connection after injecting a time delay that’s 4–5x of the Aurora replication lag on successful action queue uploads. After this injected delay, the app can download the state from any read-replica and be confident that the downloaded state includes all changes of the uploaded actions.

This has worked well in practice and is much simpler to engineer than other distributed system approaches for synchronization such as vector clocks.

Developer Ergonomics

The beauty of using SyncStore is that all the details behind persisting actions, batching action uploads, and action deletion are abstracted away as we build the app’s business logic.

We have implemented a generic queueing uploader and a generic state downloader, so to build online/offline features, our developers effectively only need to define:

• Idempotent actions/reducers in the app and in the backend
• GraphQL query for downloading states
• GraphQL mutation for uploading the actions

… and the SyncStore framework does the rest of heavy-lifting.

Once the state is defined with SyncStore, developers can connect it to React components using regular Redux patterns. SyncStore’s easy-to-use APIs have enabled our engineering team to build and iterate quickly on mobile features that work robustly in areas without Internet connectivity.

Tradeoffs

Like with any system, using SyncStore comes with its own set of challenges:

Action Migration Difficulty
Because the action list and the state are persisted in the device, it can be challenging to migrate to different actions by writing additional code that gracefully handles deprecated actions.

Increased Maintenance Load
Because actions are uploaded (and not states), the server must also implement its own reducer. This results in creating duplicate logic written in Go in the backend and in TypeScript in the React Native app. Solutions such as GopherJS exists, but they come with their own costs such as suboptimal performance and low code readability.

For 2 years of its existence, SyncStore has been powering much of our mobile features such as Hours of Service, DVIR, and Documents.

Figure 2: Features built with SyncStore

Building an app that works robustly despite unreliable connectivity is difficult, but SyncStore has made it easier for us to move fast and not break things.

We’re working on cool technical problems in distributed systems, computer vision, firmware, and UI development too! Swing by our page to check out our open engineering positions! At Samsara, we welcome all. All sizes, colors, cultures, sexes, beliefs, religions, ages, people. If you enjoy learning and building things in a highly collaborative environment, we’d love to hear from you!

Special thanks to the amazing team at Samsara who helped build SyncStore. I’d also like to thank Kavya Joshi, Elisha Paul, Yosub Shin, Joy Chen, Bo DellaMaria, Andrew Monks, Emma Ferguson, Angel Lim, Sarah Wright, Jason Laska, and Andrew Robbins for reviewing and helping edit this post!