How to Reduce Android Mobile App Data Sync Start-up Times to Delight Your Customers
At Intuit, we’re proud to solve hard problems and constantly strive to deliver unrivaled benefits to our consumer, small business and self-employed customers on our mission to power prosperity around the world.
In this blog post, we’ll explain our approach to optimizing Intuit’s QuickBooks Android mobile app during the onboarding stage, the challenges we faced, and lessons learned along the way. We’ll take a deep dive into our journey to finding a reliable solution that not only improves the overall efficiency of the onboarding experience, but improves customer delight.
We’re all familiar with mobile apps which take time to load data upon launch, making us feel frozen in time. Historically, mobile apps are architected to download/fetch as much data as possible from back-end APIs to provide an offline experience to the user. This is especially true for mobile apps that are database-centric, with a sole goal is to load up with data that might be used later.
Intuit’s QuickBooks Mobile app helps customers create and manage invoices, estimates, vendors, expenses, sales receipts, etc., all stored in a local database during app onboarding. When fetched from back-end APIs, all of this data may take significant time over 3G, LTE, even Wi-Fi networks, in extreme cases.
Apps typically show a spinner screen while fetching all the required data behind the scenes. If shown for considerably longer periods, this “infinite spinner” screen can lead to a bad onboarding experience for users, resulting in bad/low-star reviews in app stores.
Before we begin..
With the QuickBooks Android Mobile App, 225K Intuit customers track their finances, create invoices, log miles, and track their profit and loss reports each month, in addition to solving for many other accounting needs. The mobile app is architected to be database-centric and relies heavily on a relational database management framework/system provided by Android’s Room/SQLite framework to power most of the app’s critical data needs. The app builds entity data models (to support various accounting workflows like invoices, estimates, or vendors) by fetching domain models from corresponding back-end services.
The Data Sync Process
The app fetches about 25 entity data models and writes them to local database persistence storage. This process (internally known at Intuit as data sync) happens during customer onboarding at app launch and is a mandatory step for completion, after which the customer is taken to the dashboard/home screen. The app also triggers this process at a regular interval in the background to keep the app’s data in sync with server-side changes.
During onboarding, the app displays a “Loading company data..” screen (see Diagram 1 below) while the data sync process completes, so the customer must wait before continuing to use the app.
This data sync start-up process takes about 12 seconds to complete, according to our internal real user monitoring (RUM) metrics and collectively runs about 10,000 times per day, across all Android QuickBooks Mobile user devices combined (for TP90 metric, see Chart 1 below.)
So, what’s the problem?
Customers with a wide variety of company data sizes use the app daily. This mandatory data sync start-up process needs to run and complete quickly. But for customers with huge company data sizes, this can take far too long. At least that’s where our team decided to focus at the beginning. Suffice to say we had some aha moments along the way!
How did we solve it?
Let’s go over the approaches on the client side (Android app) and the roadblocks we hit.
#1 Deep dive into individual entity APIs:
To find the root cause of this problem, we dug deep to identify specific entities that took way too long to fetch from back-end services by collaborating with back-end teams here at Intuit.
This posed our first challenge: most of these entities came from one single back-end batch API and there wasn’t a predictable way to identify which entity endpoints in the backend were slow. Breaking this single batch REST (representational state transfer) API into individual APIs was doable, but came with other complex changes in the app’s data layer (a task for another day), so this approach didn’t prove viable, either.
#2 Fetch important entities first:
We then moved on to group the entities based on varying degrees of importance, fetching only the group(s) with the highest importance and deferring the rest to later stages. With this approach, we hit another challenge. Given how the app is used by customers with different business needs (small businesses, accountant firms) in our small business ecosystem, it was close to impossible to predict non-critical entities whose fetching could be avoided. So, unfortunately this approach didn’t see the light.
#3 Try adding pagination limits:
We looked into the entity data itself to see if we could ask for fewer entities from each of the batch APIs and introduced pagination for as many entity fetches as possible. As we looked at each entity’s request payload, we noticed that the ‘epoch’ field would ask the server to return only the changed records, while some entity batch requests had a pagination field to fetch the last 1000 records ever created. A few entities, however, didn’t have fields in the request payload. Upon further investigation, back-end APIs didn’t support either ‘epoch’ or ‘pagination’. So, we hit another dead end. Still, we persisted!
#4 Try bypassing the blocking data sync screen:
At this point, we concluded that the data sync process was unavoidable.
We wanted to see whether it might be possible to bypass the loading blocker screen display in the forefront — in certain scenarios when the data sync process could proceed unobtrusively behind the scenes. This was our aha moment.
With our overall QuickBooks mobile customer experience in mind, we pivoted to a different approach. We set out to dramatically reduce the total number of times data sync would run in the forefront across all Android devices each month, collectively, to improve customer experiences across a variety of small business data sizes and use cases, rather than explicitly reducing average data sync time per instance. In other words, we looked to solve the problem at a system-level rather than a per-instance occurrence level.
To answer the question of whether we could bypass the loading blocker screen display, we looked under the hood of the app’s data sync process and related database logic. After signing into QuickBooks Mobile app, the user would select a company file to work on. The data sync process began to fetch all related entity data about that company and to insert them into the database, and it was very common for a single customer to have multiple companies to choose from.
As we went deeper into the app code, we noticed there was a one-to-one mapping between the selected company and the database file created. This meant that whenever a user selected a new company, the app created a brand new database file on the device.
To efficiently let customers switch between companies, the app always kept previously created database file(s) intact and never deleted them (though the user could manually wipe out the app’s disk storage from operating system settings). With the help of Android’s SQLite Framework, we could reliably see when a database was created for the first time. Based on this first time creation, a flag (boolean) could be stored along with the corresponding company ID into a Java HashMap, and referred to later for the app to decide whether a data sync would be required.
Armed with these insights, we constructed a set of criteria to help us identify whether, for a given company, there was a need to begin the data sync process (and display the “Data Loading…” blocking screen) before letting the user go to the dashboard screen.
However, a new problem popped up with the above design. The in-memory Java HashMap object got wiped out whenever the app was shut down by the user or operating system (for example, due to low memory constraint) and wasn’t itself a reliable solution.
So, we augmented our criteria to check whether the database file existed on the device and, if so, we could determine that the entity data was already present (since the app never deletes tabular data from the database).
With this change, the criteria to trigger data sync was established in this order:
- Check if in-memory HashMap contains entry with flag set to ‘true’
- Check if the database file on the device is already created (final source of truth)
With the above criteria in place, we knew when to perform the blocking data sync operation.
But, we still needed to perform non-blocking data sync operations to fetch the latest data from back-end services, otherwise the app data would be stale. To achieve this, we triggered the data sync process behind the scenes and displayed a “data sync running” toast message, while simultaneously enabling customers to continue using the app (see Diagram 2 below).
Results
As seen in Diagram 3 below, we previously had about 121,000 instances of data sync each month, reflecting the total number of times data sync ran across all Android devices out there.
After applying this new approach to data sync optimization, total data sync instances dropped to 36,000 during the same period (see Diagram 4). That was a 70 percent reduction in total data sync occurrences for situations in which users already had a database (with the data) and could use the app immediately, without delay!
What did it mean?
Previously, there were about 10K data sync occurrences per day, each taking ~12s (TP90) to complete (120K seconds). With the above optimization, we observed a 70 percent reduction in total number of data syncs, which came down to 84k seconds (120K * 0.70) or 23.3 hours.
In other words, we saved a whopping ~23 hours per day for our customers, who otherwise would have to wait to use the app until it completed loading data!
So, any takeaways?
Below are a few learnings gleaned from this optimization that might be helpful to others looking to optimize data sync time for Android app onboarding.
- Change perspective as needed. If the selected approach didn’t help to solve the problem, we had to zoom out. Our initial approaches were all making sense when looking at improving the data sync process efficiency, while displaying the loading screen. However, looking at a higher altitude helped us answer an important question: Was the loading screen needed in the first place? Changing perspective helped us to pivot after plowing through various approaches that got tough along the way. And, it exemplifies a guiding principle we abide by here at Intuit: “fall in love with the customer problem, not the solution.”
- Pros and cons for every approach. We realized there will be pros and cons for every approach, so it was important to always keep the use case in mind. Specifically, having a single batch API to fetch all entities had certain advantages (e.g., single API call to back-end services per mobile client, single I/O thread). However, having a single API call also obfuscated critical details (e.g., the time taken by individual entity fetches), making it hard to identify and improve slow-running back-end entity APIs.
- Leave no stone unturned. While it made total sense to go deep into the software components we knew and were comfortable with, a simpler solution could be found in other cross-stack software systems. For example, we went over all entity fetch API request payloads to see if we could introduce pagination size limits, and noticed all entity requests were up to date, as supported by the corresponding back-end APIs. This investigation paved the way for other important conversations with back-end Intuit teams to improve the APIs, making them mobile friendly.
Wrap Up
Solving challenging problems can take considerable effort. Brainstorming and experimenting with a seemingly endless number of approaches can be discouraging — especially when many of them don’t prove viable. Using Android SQLite framework’s APIs directly, we are able to reliably identify and distinguish between database storage-creation nuances without any convoluted business logic. And, now the app’s database layer is clean, maintainable, and easy to understand and work with, leading to a significantly better customer experience.
Before you go
Thanks for reading! We hope this post provides useful insights and thoughtful ideas for solving mobile app challenges. If you are passionate about solving interesting problems and challenges, chat with us and join the team! The QuickBooks Mobile team is actively hiring.
Below are job openings in our team — join our talent community to learn more!
Software Engineer 2: https://jobs.intuit.com/job/mountain-view/software-engineer-2-android/27595/20948744432
Senior Software Engineer: https://jobs.intuit.com/job/mountain-view/senior-software-engineer-android/27595/20948744336
Staff Software Engineer: https://jobs.intuit.com/job/mountain-view/staff-software-engineer-android/27595/20948744128
