How we made our Chats filterable, locally-synced, and blazing fast

Beekeeper Chats

Beekeeper counts over 400 thousand weekly active users. These are primarily people who work in frontline jobs across manufacturing, retail, hospitality, construction, healthcare and 89% of them are accessing Beekeeper every week using their mobile devices.
The chat feature is fundamental for them to stay connected with colleagues and receive real-time instructions and information from their team leads or from location or corporate leadership teams.
In large and interconnected organizations, the number of open chat conversations can become substantial, making it crucial to quickly locate past chats and retrieve important information or to carry on the conversation. The ability to easily find your most recent conversation with a specific person or identify unread chats that may have been missed, becomes essential for instant access to relevant information.
To address this need, our team embarked on developing a filtering system for the chat inbox, allowing users to quickly find the relevant conversations.

The original Chats feature

The original chat feature implemented pagination, loading chats as the user scrolled down. This approach was logical because chats were sorted by their last modified timestamp, ensuring the most relevant and recent conversations appeared at the top. By not fetching old chats that were no longer relevant, this method saved bandwidth and provided users with the most current and pertinent chats in the shortest possible time.

Chat Filters

Solution Space

In order to implement filtering we identified two plausible solutions:

Solution #1: Sending the filter query to the backend which would do the filtering and return the results.
Pros:
+ Little work on the client side
Cons:
- Slow
- Chat microservice doesn’t store needed chats metadata
- Needs internet connection

Solution #2: Sync the chats locally and then simply filter the inbox
Pros:
+ Inbox will be available offline
+ Filtering becomes lightning fast as soon as all the data is stored
Cons:
- Inbox must be kept up to date at all times
- Would require a much bigger effort on the mobile clients side

We picked Solution #2 which changed the initial problem from “how do we filter inbox items” to “how do we sync the entire inbox locally”?

Risks and Challenges

The main challenge of filtering inbox items turned out to be the local syncing of all the chats in the inbox. Each conversation in the inbox is the composition of an inbox item entity with various metadata entities. For instance, displaying the avatar view of a correspondent in a 1-on-1 chat required separate fetching of user profiles. This metadata had its own lifecycle and needed periodic refreshing to keep the inbox items up to date.

Additionally, the new requirement meant that upon login, clients had to download potentially thousands of inbox items and their related metadata.

However, local syncing of the entire chat inbox provided one really important benefit for frontline workers: offline capabilities. In low-reception areas with intermittent internet connectivity, frontline workers could access chats and refer back to important information shared by peers or superiors.

Technical requirements

Functional:

1. Sync the entire inbox on the initial login
2. Keep the inbox synced after the initial login
3. Preserve the old sync functionality since the chat filters was to be feature flagged

Non-functional:

1. Reliability: The locally synced chats need to reflect the correct state of the system, i.g. it has to be up to date at all time
2. Speed: Chat syncing must happen in a timely manner

Solution

Initial naive approach

We identified two types of syncs:

1. The initial sync: syncs the entire inbox and its metadata when you first login and no chats data has been fetched yet.
2. The delta sync: each time the user opens the app after having done the initial sync, we need to retrieve the changes that happened since the last sync

Initial Sync
1. Fetch the entire inbox and store it
2. Fetch all the metadata for the inbox
3. Fetch the messages for the most recent chats

Delta Sync
1. Fetch all items that changed since the last sync
2. Update their metadata

We tested this first approach using a test tenant with mock data simulating a customer with a very large inbox.

The issues were the following:
1. Timeouts on big inboxes: For very big inboxes (4000+ items) the backend couldn’t handle the size, and the request would time out.
2. High latency: For slightly smaller inboxes, the call would take a very long time to return and after it returned we still had to fetch metadata and messages.
3. Live events: As we sync the inbox it’s likely that things get updated and we receive live events e.g. new message in one chat which makes it bubble to the top and require changing the message snippet we show. These need to be buffered and processed after the sync is complete to make sure that we are up-to-date.

After several brainstorming sessions and numerous iterations we came up with a series of optimizations that made our chat sync one of the most performant features ever.

Optimized approach

Our optimizations were driven from the following assumptions and realizations:

• Realization: Active inbox and archive can be synced in parallel as they are completely independent
• Realization: The initial sync is essentially a delta sync with the timestamp set really far in the past
• Assumption: The top of the inbox (most recent conversations and frequently contacted people) is more likely to be relevant than older conversations
• Assumption: The users will not care if they open the inbox and very old conversations are still loading, they care about seeing the recent ones. However, they will mind logging in and having to wait for 30 seconds until they can interact with the chat inbox.
• Assumption: Querying an incomplete inbox is unlikely to happen because. The time between app start and reaching the filter allows for a considerable number of items to be synced. Users are likely to filter for conversations that are somewhat recent e.g. frequently contacted people and those can be inbox items that we fetch first
• Assumption: Querying an incomplete inbox is also acceptable because in the event that a specific inbox item is not found, we can still show the loading indicator indicating the results are loading
• Realization: Incoming live events that update the inbox items will target either items we already have or items that we haven’t fetched yet. In the former case we just apply them, and in the latter we can drop them as we will fetch the inbox items anyways any time soon. As a result we didn’t need to add any handling of the live events.
• Realization: by starting the sync process right at app start, we will gain some time before the users enters the chats tab.

Optimizations

• The first obvious measure we had to take was introducing pagination to our delta sync end point. This allowed us to fetch inbox items in batches of 50, and to avoid time out problems
• We created one flow that handles both the case of an initial sync and the case of a delta sync since they are equal.
• We gave more importance to the “top of the inbox” (50 most recent items) over the bottom. That’s why we decided to first fetch the very top of the inbox and its relevant metadata to be able to present that to the users as soon as possible. Once the top of the inbox is fetched, this can already be presented to the user while showing a loading bar and loading the rest in the background
• Doing the complete syncing of the top of the inbox first and proceeding to the rest also bypasses the problem of having to buffer incoming update events and processing them post sync. The reason is that events relative to the top of the inbox will be applied right away, whereas the events relative to older inbox items can be dropped as we will fetch those inbox items later again.

This brought us to our final solution as shown in the following graph:

1. We fetch the timestamp of the last sync. If it’s present, it means that it’s not the initial sync otherwise this is just a delta sync (our inbox is not empty)
2. We start fetching the inbox in a paginated fashion and store each page right away.
   a. For the top 50 items of the inbox we also fetch the metadata right away. Fetching the date entails fetching:
   1) User profiles
   2) Group chat metadata
   3) Group chat members
3. At this point we show the inbox as the top is synced (without messages)
4. We fetch the latest messages for every inbox item of the top of the inbox
5. At this point all the inbox items are stored locally but only the metadata for the top was fetched. We therefore fetch all Inbox items that are missing metadata
6. For all the items fetched in step 5 we fetch the metadata just like we did in step 2.
7. The sync is complete

Results

The chat sync was developed in order to allow inbox filtering and was a great success for our product as it brought several unanticipated performance improvements.
The first is that chat filters are blazing fast once synced thanks to having the inbox items stored locally. Now scrolling through the inbox is very smooth and has no lag since there’s no need to fetch older inbox items as we reach the bottom of the current page. Users can now go through their chat inbox on poor network conditions or on airplane mode. For the top of the inbox they can even see the latest messages.

After the initial sync is performed, every time the app is started we only fetch delta changes, thus avoiding redundant calls and unnecessary loading of data. In the original chat sync, on the other hand, at every sync we were replacing all the local data with the latest version from the service, disregarding what actually changed. This consumed a lot more data.

Lessons learned

Multiple lessons were learned during the development of this feature, many of which confirm empirical principles in software development. One principle that certainly held true here is that the bigger the feature and the more unknowns, the more you should overestimate the effort. Additionally, it’s very hard to come up with a detailed solution from the very beginning; however, it is definitely worth it to plan ahead and sketch out what you want your solution to look like. The key for us lied in not dwelling too early on the implementation details, but still being thorough in sketching out all the possible requirements and edge cases. Due to the high uncertainty,two things were crucial in the way we worked: 1) constant iterations over the current solution and 2) very frequent syncs between the two platforms, iOS and Android, in order to make sure that the sync logic was 1:1 the same.
Decisions were always recorded in written form and every previous iteration was kept for future reference using a timeline graph.
Analogously the devil lies in the details. We realized that after implementing the core logic, one should dismiss the thought that most of the work is done. Error handling, edge cases and tracking forced us to re-think and tweak our approach multiple times and should not be underestimated.

Conclusion

The rewards of our efforts are evident in the transformative benefits brought to both the users and the company. The blazing fast chat filters have revolutionized the way users navigate and search their inbox, providing a seamless and efficient experience. The ability to access and interact with chat conversations even in poor network conditions has empowered frontline workers operating in low-reception areas.
The success of the chat sync feature serves as a reminder that with determination, teamwork, and an unwavering commitment to delivering a seamless user experience, even the most challenging obstacles can be overcome. We look forward to applying these experiences and continuing to innovate, making a lasting impact on the lives of our users and the success of our company.