Unleashing the power of efficiency: Dramatic reduction in test build times
Why unit testing is important at Qonto
Qonto’s iOS app has hundreds of thousands of users. It’s a standard in Qonto’s iOS team to ensure 100% crash-free sessions, overall, for every weekly release. We treat any regressions as an absolute priority and we respond immediately.
Occasional regressions may occur in our codebase — which is typical in large projects. However, our primary focus is on proactive prevention rather than reactive resolution. Through diligent practices such as comprehensive unit test coverage, rigorous UI testing, thorough peer reviews, and by embracing the Qonto Way with dive-ins, we strive to minimize these regressions.
It’s worth noting that our target is to reach the highest possible level of test coverage. While that coverage is 70.7%, we recognize that there’s room for improvement to reach our goal. The number of project tests is growing every day, causing our CI runtime to increase as result.
Here’s how we reduced test build time by more than 45%.
Optimizing processes for unit test compilation
When I wrote this article, the Qonto iOS project had 70.7% code coverage with more than 9,000 unit tests run on the CI, 20 to 25 times per day.
Every integration on our CI runs unit tests for 19 minutes on average; however, building tests on a developer’s MacBook (with M1 Pro chipset and 32 GB of RAM) takes around 9.5 minutes.
Qonto’s iOS project has a 100% protocol-oriented codebase. We can, therefore, rely on protocol mocking for unit testing. This makes the verification, stubbing — and other operations involved in writing unit tests efficiently — available and easy to use.
SwiftyMocky helps us meet these requirements. There are more than 1200 mockable protocols in the main app target:
Distribution of SwiftyMocky version 4.2.0, which we use at Qonto, includes a default template file used for generating mocked protocols, and depends on a compiled distribution of Sourcery.
Based on the given defaults, a file named Mock.generated.swift is created, but it has its own issues.
The total build timeline is 9 minutes and 37 seconds, out of which 312 seconds are spent on compiling the Mock.generated.swift file, and the build timeline looks like this:
Problem statement
#1: Large generated file for mocked protocols
When we want to mock 1000+ protocols, a single Mock.generated.swift file contains more than 300,000 lines of code.
When using the “Incremental” mode, Xcode compiles multiple files at the same time, which helps make the whole compilation process faster. Having a single file with all protocols turns off this compiler workflow, and all files depending on Mock.generated.swift need to wait in the build queue until the compiler finishes processing this file. The compiler can’t handle this in parallel.
#2: Poor developer experience in Xcode
Since generated symbols are available and indexed for searching, it’s common for a developer to eventually misclick on a mock implementation definition instead of an actual one.
As mentioned above, the file Mock.generated.swift contains more than 300,000 lines of code. The Xcode editor can’t handle such large files. Neither smooth scrolling nor symbol lookup works without the spinning beach ball appearing for up to 15 seconds (and sometimes longer) even on the M1 Max chipset, depending on the CPU load:
With this problem, Xcode was hanging multiple times every day. This occurred when developers — who were researching the codebase by looking up method definitions or using Search — would accidentally open a generated version of the protocol definition via Quick Open (⌘+⬆+o) and Jump to Definition (⌘+click).
To summarize, large Mock.generated.swift not only limits the functionality of the Xcode source editor but also hinders the Swift compiler from doing a parallel compilation of files.
Possible solutions
It all started during a local hackathon that took place at Qonto in March, 2023. The hackathon unleashed a wave of creativity, making a big impact on many different areas of the company, including the duration of unit test compilation for Qonto’s iOS project.
Multiple solutions were available; however, not all of them were suitable for each situation. In our case, only one worked out to be easy to achieve and fast to deliver.
1️⃣ SwiftyMocky support of splitting generated files into multiple files
According to this issue on GitHub, it’s possible to annotate a declaration with the following comment to make Sourcery use the pre-existing declaration to fill in definitions for functions and variables:
// sourcery: mock = "ItemsRepository"
class ItemsRepositoryMock: ItemsRepository, Mock {It does require pre-creating such files separately, either manually or via automation, for every mocked protocol. There are more than 1200 such cases, so this solution couldn’t meet our needs.
2️⃣ Manually split Mock.generated.swift into multiple files
This solution would work, but it requires custom scripting and might eventually fail if the scripting didn’t work correctly. It might also lead to build errors because of a bad “cut” between mocked definitions, if we weren’t relying on SwiftSyntax and using custom logic for splitting files.
Even though this option seemed reasonable, we chose to use the native solution provided by Sourcery as a more scalable and customizable way of configuring code generation results.
3️⃣ Modify Mock.swifttemplate
SwiftyMocky has a set of templates used by Sourcery to generate mocked versions of protocols. Since Sourcery supports per-file code generation, we considered it to be the right solution. The team decided to go for it.
The change would be to add a loop responsible for controlling the output filename:
- <%_ for type in all { -%><%_ -%>
- MOCK DEFINITION HERE
- <%_ } -%>
+ <%_ for type in all { -%><%_ -%>
+ // sourcery:file:Mock+<%= type.name %>.generated.swift
+ MOCK DEFINITION HERE
+ // sourcery:end
+ <%_ } -%>With this change, the template was already generating multiple files. Each Sourcery configuration file would then specify the output attribute as a folder, and not a file like it was before:
output:
- ./Sources/Mock.generated.swift
+ .Sources/Mocks/GeneratedThere was a caveat, though, that whenever we added a new mocked protocol to the project, we had to modify the Xcode project in a specific way — by adding the file to a particular group and the Build Phase containing all compiled source files.
Sourcery supports settings to link files to the specified project automatically:
output:
path: <output path>
link:
project: <path to the xcodeproj to link to>
target: <name of the target to link to> // or targets: [target1, target2, ...]
group: <group in the project to add files to> // by default files are added to project's root groupAfter testing this way, it became apparent Sourcery would not remove protocols we moved to another module or target, nor would it remove protocols from which we deleted the “//sourcery: AutoMockable” attribute. We wrote a custom script using XcodeProj and Swift Argument Parser to address this issue.
Another caveat was that mock generation could not be a part of the Build Phase because it might eventually add or remove compilable sources. This would cancel the build in Xcode since the build plan is constructed during the first Xcode pipeline operations.
Solution 3️⃣: Before and After
After some work and customizations, we established file processing and pbxproj modifications. We did tests and took measurements before introducing this change, and as soon as we implemented it, we saw these results:
- Xcode stopped freezing when developers missed the desired item in search results or used Quick Open (⌘+o) to open the generated mock version. We’re already saving a lot of time because of this and team morale is high.
- Mock files compiled in parallel changed the entire build pipeline:
Of course, this parallelization has drastically improved iterative builds, as well as clean builds.
Generated mocks compilation time metrics
⚠️ * Mocks generation takes more time than it used to because there are 1000+ files generated, compared to before when there was just a single file. However, it’s essential to note that we changed the workflow. So mocks are generated outside of the build pipeline rather than within a build phase of Xcode.
Build time metrics
This advancement attained a remarkable arithmetic mean reduction of approximately ~46.36% overall. A simple approach to a non-trivial problem adds to its significance. This shows that, even though a situation might seem difficult, the solution doesn’t always have to be as tricky.
Note that all values are approximate, within a range of ±5 seconds.
Outcome
The results exceeded our expectations. The duration of pipelines decreased, from 39 minutes down by ~37% to 25 minutes on average:
We used the climatiq.io API to calculate CO2 emissions with the average input values.
- Before: 14 kg of annual CO2 emissions on average, caused by CI running unit tests.
- After: 8.7 kg of annual CO2 emissions on average, caused by CI running unit tests. This is approximately a 38% reduction.
It was also a big win in terms of developer experience. Now Xcode doesn’t hang multiple times a day whenever search results are browsed or generated definitions are accidentally clicked on.
Qonto is a finance solution designed for SMEs and freelancers founded in 2016 by Steve Anavi and Alexandre Prot. Since our launch in July 2017, Qonto has made business financing easy for more than 350,000 companies.
Business owners save time thanks to Qonto’s streamlined account set-up, an intuitive day-to-day user experience with unlimited transaction history, accounting exports, and a practical expense management feature.
They stay in control while being able to give their teams more autonomy via real-time notifications and a user-rights management system.
They benefit from improved cash-flow visibility by means of smart dashboards, transaction auto-tagging, and cash-flow monitoring tools.
They also enjoy stellar customer support at a fair and transparent price.
Interested in joining a challenging and game-changing company? Consult our job offers!
