How we made our React Native tests run 5x faster
By Arthur Lee, a Chime Software Engineer
Every day, millions of members use the Chime mobile app to achieve financial peace of mind. At Chime, we use React Native to build our mobile app. We cut app releases from our main branch every week so that we can deliver the best experiences to our members as quickly as possible.
When moving at such a high velocity, we also have to make sure we aren’t causing any regressions along the way—which can negatively impact our members, who rely on Chime for necessities like groceries, gas, and paying bills. Despite having a QA process that validates every release, we want to catch issues early so they can be addressed without disrupting the release cycle. We have a Continuous Integration (CI) suite with automated tests that run for every mobile Pull Request (PR). A majority of these tests are end-to-end (E2E) tests written using Detox, which tests our app’s features in an iOS simulator against a mocked backend.
Problem
As Chime has grown, so has our mobile engineering organization. In the space of 12 months, a once ten-person team quickly grew to over 30 developers across multiple teams. As we wrote more code and shipped more features, our CI times ballooned.
Last year, running the basic test suite took approximately an hour, and running the full suite took as long as two hours! This led to a myriad of issues: Developers were afraid to add more tests because they worried about “slowing everyone down,” resulting in lower test coverage, and developers couldn’t reliably predict how long it would take for them to get their PRs merged—causing frustration for product managers. Plus, last-minute fixes and improvements didn’t ever make the release cut. Developers were constantly context switching, lowering productivity.
From one hour to 35 minutes
At the time, our CI tasks had been set up to run serially, which made sense given the team’s former, smaller size, but it soon became clear that we would need to run tests in parallel. However, this wasn’t as straightforward as it seemed. Unlike unit tests, which are lightweight to bootstrap, our Detox E2E tests take over 20 minutes to bootstrap (because Detox needs to build a custom version of our iOS app before tests can begin).
While we could theoretically build the app N times across N workers for every PR, that would be a waste of resources, especially given the high costs of macOS workers. Instead, we were able to leverage CircleCI’s workspaces feature to build the app once and share the built app with multiple test-running workers with minimal overhead. As a bonus, we were able to use lower-cost linux workers for our lint, type-checking, and unit tests.
With these techniques, we lowered the runtime of the full test suite to 35 minutes — a 40% reduction in runtime.
From 35 minutes to 15 minutes
While our initial speed increase was significant, we weren’t totally satisfied. No matter how many parallel workers we added or how few tests we ran, the suite still took longer than 20 minutes because of the iOS build step. Ideally, developers shouldn’t have to wait longer than 20 minutes to get feedback on their PR.
To make matters more interesting, a bulk of the 20-minute build time was spent with Xcode building the native portions of our app. But over 95% of our PRs only change React Native TypeScript code. Our CI system was building the same native code over and over again!
It turned out we can skip the iOS build completely for a majority of our PRs. Using the git hash of our ios/ directory as a cache key, we were able to create a cache of our iOS Detox builds. If a cached build was available, we would use a powerful (16 CPU) linux worker to build our React Native JS bundle using Metro — only taking about a minute. We would then copy the bundle into the built app before it is loaded onto the simulator. With these improvements, a 20-minute build process has been reduced to one minute in most cases.
Results: Happy developers
A process that used to take over an hour– as long as a flight from San Francisco to Los Angeles– now takes 10–15 minutes. In the time it takes to grab a coffee, developers get feedback about their PR, enabling them to ship more code safely and with less context switching. More importantly, these improvements have helped prepare our infrastructure for the years ahead — with the ability to scale up to hundreds of workers, our test suite can grow without any adverse effect on CI times. This lets us keep our codebase safe and our teams efficient and focused on delivering the best experience to our members.
