Reduce iOS CI time using test parallelization
In Mercadona Tech, we define ourselves as an XP team; all we do are feedback loops, which means that we move fast in small safe steps. This post describes how we used Xcode tools to reduce the e2e test suite time of our iOS e-commerce application so that we can be safe and fast 🚀
Motivation: Why do we need this?
In Mercadona Tech, we have three e-commerce clients (iOS, Android and WEB), and every client has its own e2e test suite that tests all the golden paths of the applications. We can say that e2e tests are our seatbelts.
After some failed integrations with new Backend releases and after 2 or 3 rollbacks, the shop team in Mercadona Tech decided to include the three-client e2e test suite in every backend release pipeline.
This made us much safer, but on the other hand, this made us slower also (remember…XP). At this time, our iOS e2e lasted more than 15 minutes to run, and Backend releases are usually more than 1 per day (small steps). It made us focus on reducing our e2e time as it was the main bottleneck, and test parallelization was the way to get it. Still, before enabling test parallelization, we had to make some changes in the structure of our tests.
Starting point: Where were we?
Before this process, our e2e was included in one single file with a description (we use Quick framework) for each feature that we wanted to test (Create a user, create an order, rate an order….)
We use a docker image with the last backend release, pre-filled with some fixtures, as our server to run the test. So all our network requests run against this “fake” backend.
These tests run sequentially and always in the same order, and here we faced our first (and foremost) challenge. Some tests had dependencies with a previous one, f.e. you cannot edit order if the order doesn’t exist.
So we needed to find out how to divide the whole ApplicationSpec file into one file per feature, independent of each other.
The mean total time to run all the tests was about 15 minutes.
First step: Know your tests
So our first action was to write an e2e test catalog in a simple XML table. We divided the tests into features, annotating in each test if it has any dependency on another test and what data it would need to run in isolation.
It will help us know how many files we can divide the whole test suite and what fixtures we must create in the backend to run the test files in parallel and random order.
Once we fully know everything we need, it’s time to get in hands with the code.
Migration: Do it in a lean way
One of the core principles of Mercadona Tech when delivering software is doing it in a lean way, adding value with each short step.
It makes no sense to do the complete migration at a time, creating all the fixtures on the backend side and breaking the whole Application spec file into shorter files in one step.
What we did: begin with the simplest feature, and once we had it isolated,😅 take learnings from this step and go ahead with the next one.
This step brought us some side benefits for the whole team. As we needed new fixtures to be created in the backend project, it was a perfect time to get our hands dirty and learn to make them ourselves with the help of the backend team.
This led us to cross-pairing between different guilds, reducing the knowledge silos inside the team, improving our T-shaped skills, and, as the whole team, benefiting from the tests. We spread the ownership of the creation and maintenance of the e2e test to the entire team, not only the iOS team.
The result of the migration gave us the following file structure:
We moved from one big ApplicationSpec file containing all the tests to 11 independent files!
Parallelization: Embrace Xcode test plans
Test plans were introduced in Xcode 11 and are a great tool to organize and configure your tests, and there is a lot of documentation on how to create and configure test plans. And it’s fully integrated with automation tools such as Fastlane, which is great for us because we are Fastlane lovers.
In our e-commerce app, we defined 3 test plans:
- Integration test plan
- e2e test plan
- All tests (runs both unit test and e2e plans)
To parallelize the test plan, just enable this option in Xcode and BANG! Magic happens!! Each e2e test file will run in the first available simulator.
And this is a short clip of the tests running in parallel:
As you see in the GIF, five simulators are running at the same time so that we can run 5 test files simultaneously; this was achieved on a MacBook PRO with an M1 pro chip. We use intel i5 or i7 as our CI nodes, and we get three simulators simultaneously.
You can change the number of simulators running on parallel in the Xcode preferences menu:
We use the auto option as default because there is no significant difference in time having more than three simulators for our current test suite.
And the big thing here is that we passed from a mean of 15-minute run time to a mean of 6 minutes, that’s more than a 60% percent of time reduction.
Extra ball: The hidden benefits
After the parallelization and test plan adoption process, we got some collateral benefits almost for free.
1. Now, we are able to debug e2e tests a lot faster than before!
If you’ve ever written UI tests, especially e2e tests, you may agree that the word that best defines them is slow. They’re slow to run, debug, and hard to maintain. Whenever you see a red in the UI tests pipeline, you can spend a lot of time figuring out what and why it failed.
If you remember, in the beginning, we had dependencies between the tests, which meant that to debug a failing test, you may run the whole 15 minutes suite every time. Fortunately, we divided the tests into independent files. If some test fails, we can focus only on the file or the test that failed. So this hidden gem saved us a lot of debugging time.
2. We sometimes experienced some randomly flaky tests that were solved just by re-running the pipeline. Thanks to test plans, we can mitigate this by enabling the test repetition option in the e2e test plan, so if a test fails, retries it one more time, and if it passes this second time, Jenkins will mark this build as unstable. So we can notice how frequently we get these flaky tests.
3. To run our CI pipelines, we use dedicated nodes, basically mac computers. These computers need to be maintained and configured from time to time (new Xcode versions, Fastlane updates…). Reducing our whole pipeline time makes these computers able to run more jobs, so we need fewer dedicated nodes; this means less maintenance from our side.
Conclusions: Our job here is done
As you can imagine, after reading this post, in Mercadona Tech, we spend a lot of time writing and improving our test suites. And specifically, the e2e tests require a big part of this time on the shop team. This is done like this because we get a lot out of them. They saved us from going to production with mistakes that were not visible on the unit tests. This is even more important when talking about iOS applications (Appstore review time, version adoption…)
Parallelizing our e2e tests brought us a lot of benefits, not only as iOS developers but also as a development team; in addition, we followed and practiced most of our organization’s engineering principles.
On the other hand, there is still room for improvement. For example, there are still dependent tests inside some files, and this means that we cannot run the tests on that file randomly, and we had to enable de alphabetical run.
If you liked what you read here, we invite you to visit our medium page and we are sure you will find something interesting for yourself.
And last but not least, if you’re interested in joining our fantastic team, check our open positions!!
