AB testing Xcode build settings
by Ben Yohay
At Lightricks, our iOS developers are busy working on multiple apps. We wanted to see if we could reduce the average build time to boost our developers efficiency, both locally and in a CI environment.
To achieve this, we first needed to gather information about build times. Then we used this information to test whether we can speed up the average by changing build settings — all without interfering with the developers’ regular work.
To gather the information we used Spotify’s excellent XCMetrics tool. We then augmented it with support for AB testing to allow us to compare different configurations of the build settings. We’ve controlled these settings in a way invisible to our developers (even when running on their machines) by harnessing the power of Xcode’s xcconfig files.
Before going into the design of the experiments, let’s briefly recap on xcconfig files:
xcconfig files
In a nutshell, xcconfig are text files that declare key-value pairs of build settings mapped to their values. At Lightricks, we use xcconfig files for almost all build settings, as they’re more flexible and readable than Xcode project build settings. Furthermore, they can be shared across projects. NSHipster has a great introduction on them here.
In order to conduct experiments though, let’s look at manipulating Xcode to load build settings on-the-fly.
Loading build settings dynamically
Loading build settings dynamically is not supported natively by Xcode, so we had to figure out a way to do that using existing capabilities. We used the fact that xcconfig files can include other xcconfig files, plus the ability to run pre-build actions in Xcode.
Pre-build actions are shell scripts invoked before the build takes place. As it turns out, if you create an xcconfig file in a pre-build action, then include it in an xcconfig file that the app uses, the build settings from the created xcconfig file will apply. Let’s look at a simple example:
Assume that our target uses a configuration file Application.xcconfig that’s in $SRCROOT (The path to the directory containing the Xcode project). The Application.xcconfig contains the following lines:
Choosing our app from “Provide build settings from” and adding this line to pre-build action:
will turn optimization on by creating a new xcconfig “just in time” with a SWIFT_OPTIMIZATION_LEVEL build setting.
Selecting variants
Now we know how to create xcconfig files on-the-fly, we still need to assign different build settings to different participants. For this, each variant of an experiment is mapped to a set of build settings that are written to the xcconfig file. When a variant is selected, its corresponding build settings are written to the xcconfig file included by the app’s xcconfig file. Continuing with our previous example:
Writing the file atomically ensures it won’t produce invalid build settings in case the script was stopped abruptly.
If more than one experiment is running, they can be aggregated together and written once:
Sending the results
To actually compare the performance of the variants, the results need to be sent to a central location. After the script is done selecting all the experiments, it creates a JSON file that contains the selected variants. The format of the JSON file is quite simple:
After the build finishes, XCMetrics is run in a post-build action. XCMetrics runs in the background, therefore it doesn’t affect the build time. In addition to the regular metrics that XCMetrics collects, we instruct it to read the variants JSON file and send it as build metadata, as specified in the custom metrics collection section.
Hiding experiment files
In order not to disrupt developers workflow, we needed to find a way to create the xcconfig and JSON files without them appearing in git status.
The challenge is that the include path to import other xcconfig files is very limited. In particular, it cannot include an xcconfig path that’s based on a variable (except for DEVELOPER_DIR). We can’t write the xcconfig file to derived data (since its location may differ between machines), so in order to not litter the developers’ filesystem outside the repository, we created the dynamic xcconfig file in the repository, and had the application’s xcconfig include it using a relative path. We added a folder named .xcode_experiments to .gitignore, which is where the experiment files will be written. Additionally, for the experiments’ build settings to take effect, we included the dynamic xcconfig file from the xcconfig file of the app:
“#include?” is used in order to fail silently if the file isn’t found, e.g. when no experiments are being run.
This line is also checked into the repository.
Problem: When to re-select variants?
To ensure reliable results we need enough samples for each variant. The problem is that we only have a few dozen users, and assigning a permanent variant to users might result in insufficient samples for some of the variants.
On the other hand, we don’t want to disrupt developers’ workflow, so we need to be careful when variants are re-selected.
That’s why we chose to re-select variants only on clean builds. We identify a clean build by checking if the project’s derived data folder exists:
Experimental results
Let’s look at the results of some of the experiments that we conducted.
Linker experiment
We set out to improve incremental builds, and linking is one of the slowest steps in such builds. Linking is done after the build system finishes compiling all the files. It takes all the compiled files and produces a single executable or library file. It doesn’t depend on the number of files that changed — it would take roughly the same amount of time if a single file changed as from a clean build.
We experimented with an alternative open-source linker called zld and with some linker optimizations, to check if it can provide a speedup in the linking step. Here are the results of the experiment:
We tested the results with standard statistical tests and deduced that the “zld-with-optimizations” variant is significantly faster than the default build settings.
Disclaimer: The test is only relevant to Xcode 13 as Apple claims they have improved the link time on Xcode 14.
Swift build system integration
Apple introduced a new setting in Xcode 13.2 that could speed up compilation times when compiling with Swift. It’s enabled by executing:
and affects all the compilation with Xcode 13.2 and above.
As you may have noticed, this setting isn’t related to Xcode build settings nor xcconfig files, so we had to change the behavior of the experiment a bit. Each time a variant is selected (a clean build), instead of writing an xcconfig file, we simply enable the setting if the user is given the “build system integration” variant, and delete the setting if given the “baseline” variant.
Projects that don’t participate in the experiment will be affected as well, but if they don’t send metrics it won’t affect the results. And assuming this setting always produces valid builds, this shouldn’t affect the developers’ experience.
The results are displayed in the following graph:
The results show that there was no significant difference between the variants — so we decided to continue using the baseline.
Summary
At Lightricks, we strive to base our decisions on actual data rather than our gut, which is susceptible to cognitive bias. In this post we’ve shown an example of how you can collect actual build data and run experiments, without disrupting your developers’ workflow. I hope this will encourage you to collect data and make more informed decisions.
Let us know in the comments what kind of results you get from the experiments you conducted!
—
Create magic with us
We’re always on the lookout for promising new talent. If you’re excited about developing groundbreaking new tools for creators, we want to hear from you. From writing code to researching new features, you’ll be surrounded by a supportive team who lives and breathes technology.
Sound like you? Apply here.
