Continuous integration for iOS with Nix and Buildkite
Austin Louden | Pinterest engineer, Core Experience
Tens of millions of people use the Pinterest app for iOS every month. To ensure a high-quality experience for our users on Apple devices, we use continuous integration (CI) to test incoming code before it gets shipped. We recently adopted Nix and Buildkite in our CI infrastructure to make this process faster and more developer-friendly. By implementing these tools, we created a sandboxed development environment that allows engineers to make changes locally and expect identical results on CI. It also makes onboarding new engineers easier, saves CI capacity and increases developer productivity. In this post we’ll share challenges we faced with Jenkins, why we migrated to Nix and Buildkite and how we made adopting these services more seamless.
Gathering developer feedback
Our pre-existing CI setup was difficult to understand and maintain. Using Jenkins, we had code written in Groovy and bash spread throughout the iOS repository. We also had a substantial amount of code stored in Jenkins configurations, which wasn’t under version control.
With this setup, we identified concerns our developers had about the current CI, including:
- “My build passes locally, but fails on CI.”
- “It’s difficult to reproduce what’s running on CI on my machine.”
- “It’s difficult to add new pipelines or new build machines.”
- “Builds are not fast enough.”
- “It’s difficult to tell why my build failed.”
At the root of the first two issues was the idea that our builds weren’t reproducible enough. The lack of reproducibility can be a major detractor from developer productivity. Finding bugs turns into a goose chase for differences between two environments — there’s the dreaded non-starter, “well it works on my machine!”.
The second class of issues had to do with Jenkins and how we were using it. While Jenkins is known for being a dependable open-source CI system, the problems we faced inspired us to look for alternatives.
Irreproducible environments
When addressing reproducibility, we realized the scope of the issue wasn’t limited to our CI system. We wanted to give our engineers the capability to run the exact commands that are run on CI in an identical environment, so we examined the way developers set up their own local environment.
At any company, new engineers need to install all the necessary dependencies before they can start contributing. For iOS engineers, this meant running a script called setup_environment.sh. This was an imperative, catch-all script that installed the bare minimum necessary to build the app.
setup_environment.sh was only run on developer’s machines, not on CI. Those machines had their own set of scripts and dependencies. Over time, the setup for engineers and CI diverged further apart. It became almost impossible to run commands locally that were run on CI. After some discussion, we came up with our challenge: With one command, can we make an engineer’s local environment identical to CI, to the extent where an arbitrary developer’s laptop could become a CI machine?
Initial attempts
Our dependency problems stemmed primarily from Python and Ruby. For example, we use the Ruby gem fastlane to handle code signing and the Python package boto to send artifacts and build data to Amazon S3.
We needed to ensure all machines were capable of using specific versions of several pieces of software. Our web team relies heavily on Docker for problems like this, however running macOS inside a Docker container is not supported.
For Ruby, we tried rbenv and bundler. For Python, we tried virtualenv and installed other miscellaneous dependencies with Homebrew.
Ultimately, we weren’t happy with this approach for a few reasons:
- Several layers of depth meant that any mistake in the dependency tree could result in issues. For example, if rbenv had the wrong version of Ruby selected, that would affect fastlane.
- In the end it wasn’t easy for developers or CI to use. Developers weren’t sure which commands were needed to run a specific virtual environment, while CI had to activate rbenv and virtualenv in every build.
- Homebrew wasn’t designed with reproducibility in mind. We considered creating an internal Homebrew tap as a fix, but ultimately decided to look elsewhere.
Driving reproducibility with Nix
Enter Nix. Nix is a package manager that’s popular in the functional programming community. It allowed us to create the reproducible environment we were looking for. We’ve since replaced our setup_environment.sh script with one that simply installs Nix. This is used when onboarding new developers and to provision new CI machines.
The key difference between Nix and other package managers is that Nix treats packages like immutable values — once a package is built, it will never change. Nix takes advantage of that packages-as-immutable-values idea to derive several features.
Here’s how we leveraged these features to create our shared environment.
Nix runs all packages out of the /nix directory
Software like virtualenv, rbenv, bundler and brew introduce a lot of complexity into the PATH. We ran into plenty of issues in the past involving a faulty PATH locally or on CI. Using Nix, the PATH is automatically set based on which dependencies are needed at the time. For example:
Before Nix:
𝝺 which ruby/Users/alouden/.rbenv/shims/ruby𝝺 which python/usr/bin/python
Inside the nix-shell:
[nix-shell:~/ios]$ which ruby/nix/store/gxkwkw3g8m3mpb524rssfsg4m7q1jnzk-ruby-2.3.3/bin/ruby[nix-shell:~/ios]$ which python/nix/store/6hn46zk4fs9v80ablii3zkr3xal42fp6-python-2.7.13/bin/python
The garbled text before the package name is a cryptographic hash of the package’s build dependency graph. This allows us to easily have multiple versions of the same package installed. We don’t need to worry about overwriting what an engineer may have installed locally. Once we’re in nix-shell, it’ll be our specific version.
Single commands can be run in the nix-shell with “--run”
Before entering the nix-shell, Nix ensures all required dependencies are installed and in your PATH. It’s like saying, “run this command, but with all the dependencies I’ve specified.”
𝝺 which fastlanefastlane not found
For example, there’s no fastlane installed globally (above), but it finds the correct package (below) when I run a fastlane command in the nix-shell. If Nix has fastlane already installed it uses that version, otherwise it downloads a pre-compiled binary from Nix’s shared cache or a pre-specified location.
𝝺 nix-shell . --run “which fastlane”/nix/store/6d39qpkan48l075iwkvd2vapk7ih5rrv-pinterest/bin/fastlane
New dependencies can be added everywhere by modifying one expression
Since we run most commands out of the nix-shell, the CI or developer machines automatically install any new dependencies. This continually saves us time and CI capacity because:
- We no longer need to re-provision the CI machines every time we add a dependency. This was an incredibly time-intensive process, because it required building a new virtual machine and deploying the updated VM to all Mac Pros manually.
- We no longer need to take down CI machines while we re-provision.
- Engineers never need to rerun a “setup_environment” script. They can simply run the command inside the nix-shell and it will just work.
Our Nix setup
The multi-level dependency tree shown above has been replaced by this short Nix expression we keep in the root of the iOS repository.
If we revisit the first two points of feedback:
- My build passes locally, but fails on CI. Now that the CI environment is identical to the developer’s local environment, we’re confident that failures should be consistent.
- It’s difficult to reproduce what’s running on CI on my machine. Every command run on CI can be run locally if it’s done through the nix-shell. This means engineers are capable of running a full test with their patch before submitting a pull request, saving time and CI capacity.
It’s worth noting that while Nix helps us create environments, an environment is only one part of reproducibility. We’re also exploring technologies like Bazel to make builds entirely reproducible, which we hope to discuss in a later post.
Continuous integration services
With confidence in our new sandboxed environment, we began to address the remaining issues with CI:
- “It’s difficult to add new pipelines or new build machines.”
- “Builds are not fast enough.”
- “It’s difficult to tell why my build failed.”
Considering the amount of work required to get our Jenkins setup to meet these needs, we decided to evaluate alternative build systems (including Jenkins Blue Ocean). After some research we started testing Buildkite. Adopting a new service is a difficult decision, so here’s a look at our setup and the advantages it has over Jenkins.
Any machine can become a Buildkite Agent in seconds
This will install and start the buildkite-agent.
That’s it. This machine can immediately begin processing jobs. Unlike Jenkins, we no longer have to worry about connecting master and slave nodes, configuring the launch agent or installing anything else.
Now that our environment is sandboxed, any machine can become an agent in seconds. While we were transitioning from Jenkins to Buildkite, we even turned a few of our personal iMacs into build machines to add capacity.
Buildkite builds are parallel by default
Buildkite runs each step on a different agent by default. This led to a decrease in build times because it encouraged us to parallelize as much as possible. Here’s the pipeline that runs on every pull request in the Pinterest iOS app:
Buildkite parses the YAML file and displays a pipeline like the one shown below. Notice how each step is run on a different machine and run within the nix-shell.
Build steps can run on different agents
Our local Mac Pro cluster is limited compared to the fleet of Linux agents that build our Android and web apps. This means we only run the parts of the build on macOS that have to run on macOS, such as building and signing the .ipa. Other less essential parts of the build can be farmed out to Linux machines.
It’s easy to tell where the build failed
Thanks to Buildkite’s UI, it’s easy for developers to find where a build failed.
The failed step can be expanded.
Then we can locate the error.
Potential downsides
Before officially switching over, we looked at the potential downsides to Buildkite and developed strategies to mitigate them.
Service vs. open-source software
Buildkite is a company, not an open-source project like Jenkins, so the future of our CI system is, to some extent, tied to the future of Buildkite. On the other hand, Jenkins isn’t dependent on a specific company.
After review, we felt it wouldn’t be too difficult to switch back to Jenkins, or another service if we needed to, in the future. Since all build code is now in scripts within the iOS repository, we can continue to use those scripts on any service.
Security
One great aspect of Jenkins is that it can be kept entirely internal. Buildkite, on the contrary, requires remote execution. Our hardware runs commands received via the open-source Buildkite agent.
To mitigate risk, we isolated our build machines from the rest of our network. We also take advantage of a Buildkite feature that prevents agents from running console commands. This allows scripts within a repository to be evaluated, providing a safety net against remote execution vulnerabilities.
Evaluating success
Over the past couple months, we’ve run thousands of jobs on Buildkite. In that time, Buildkite has exceeded expectations.
Looking back at our earlier concerns:
- It’s difficult to add new pipelines or new build machines. We’re now able to add new machines in seconds. Adding a new pipeline is as simple as adding a YAML file.
- Builds are not fast enough. Fully parallel builds have made a measurable difference in build times, sometimes as much as 50 percent for certain jobs.
- It’s difficult to tell why my build failed. The Buildkite UI makes it easy to locate build failures and identify where it went wrong.
We chose to take on this project to help developers move faster, while making sure to keep quality high. However, there’s still a lot of work to be done. If you’re interested in building the tools and frameworks that power Pinterest’s apps, from the UI to the build system, come work on the Core Experience team.
Acknowledgements: Thanks to Brandon Kase, Harry Shamansky, Kevin Grandon, Rahul Malik and the rest of the Core Platform team for their contributions.
