Improving our CI/CD with GitHub Actions
Introduction
This year, the Turo iOS team embarked on replacing our CI/CD system with a combination of Xcode Cloud and self-hosted GitHub Actions on a lightning-fast M1 Mac Mini.
As a result of this move, our build and test job now take ~10 minutes per run compared to ~30 minutes on our old hardware.
Motivation
From start to finish, our unit tests were encroaching on ~30 minutes per run (not including time in the queue!) which, as we doubled the size of the iOS team this year (we’re still hiring!), was quickly becoming unacceptable.
Our previous CI/CD tool was not user-friendly, resulting in only a few engineers understanding how it worked. It was also disconnected from our version control system, making it challenging to review changes to build scripts and view recent changes properly.
When evaluating new CI/CD systems, we found that GitHub Actions stood out. It’s free, compatible with Apple Silicon, and would allow us to move our CI/CD scripts into version control, enabling code review and file history. What’s more, GitHub Actions has a more user-friendly UI when compared to our previous tool, and more developers have a working knowledge since it is integrated into GitHub’s familiar UI.
How we did it
Our GitHub Actions integration comprises two self-hosted M1 Mac Minis, each running two runners. One runner we label as `normal-priority` and use this for most workflows, like running tests or uploading translations. The second runner is labeled `high-priority` and reserved for time-sensitive workflows we want to run immediately, like releasing a new app version.
This year, we created in-house scripts to help ease onboarding, allowing developers to configure their entire development environment with one terminal command (we’ll share more on how these scripts work in a future blog post). We configure dependencies on the runner machines using the same script that we use to set up our own iOS development environment. This tactic makes it much easier to debug problems in actions, as you can just run the commands in your own terminal and see the results. In addition, if the self-hosted runner is down due to maintenance, it’s easy to run the commands locally and unblock yourself.
As our team grew, we decided to add a new Mac Mini to reduce queue times; these scripts, combined with how easy it is to set up a new runner, allowed us to get going in just a few minutes and cut CI/CD queue times in half.
Workflow template
Generally, our workflows are all one job per workflow file based on the following template:
- Provide a descriptive name for the job. We like to include the word `Automatic` if a job is triggered by an external event like a merge in another repo or a cron job.
E.g., “Automatic Translation Update” periodically checks if any new translations are available from our third-party translation vendor, and
“Automatic CodeGen” detects when we need to regenerate our model classes when a change is detected in the YAML file in our backend repo. - We’ve found it useful for all jobs to have the ability to be triggered manually with `workflow_dispatch` in addition to other triggers. You don’t have to wait for a specific condition to debug if your script is working properly.
- For some jobs, like running tests, we only want to run them on the latest commit to help keep job queue times down.
- Here we select between our `normal_priority` runner and `high_priority` runner.
- By default, jobs will not run under zsh on macOS self-hosted runners, leading to `command not found:` and other `PATH` related errors. Here we set `zsh` as the shell on which to run the job.
- Currently, self-hosted runners do not clean out the work folder between jobs. As the first step in all our jobs, we manually clean the folder, so each run starts with a fresh state.
- Add any more steps after the `Clean Workspace Folder` step.
Improvements we would love
It’s been great integrating GitHub Actions with our current workflows. There are a few improvements we would love to see in the future:
Ability to bulk delete old workflow runs so only relevant workflows appear in the side menu on the Actions page.
It would be great to have the ability to bulk delete workflow runs, so they don’t appear in the sidebar of the Actions page.
As we iterated on actions, this menu became cluttered with irrelevant workflows. We were able to clean it up with the following script, but it would be great to have this as a supported feature.
zsh as the default shell on macOS self-hosted runners
Since macOS Catalina 10.15 zsh is now the default shell on macOS, it makes sense that the self-hosted runner should default to zsh as well.
Most tooling (Homebrew etc.) installed on the machine works best with zsh. We ran into a few issues with commands missing from our $PATH before we figured out we needed to add shell: zsh {0}
This issue has been discussed already (1, 2), but we’re hoping this can be addressed in the future.
Clean up self-hosted runners after each run
It would be great if the runner cleaned up the workspace folder automatically after each job runs to allow us to remove our “Clean Workspace Folder” step from each workflow file.
While you can accomplish this by adding a script to run before/after a job (described here), this isn’t ideal for us, as these scripts are added outside of version control and manually added when setting up the runner.
Ability to bump a job to the start of the queue
Our previous CI/CD solution had the ability to manually bump a job to run at the start of the queue. If we had similar functionality in GitHub Actions, we could remove our `high_priority` runner and manually bump these time-sensitive jobs to the top of the queue.
Going forward
We’re delighted with our move to GitHub Actions. Their documentation is stellar and made moving our CI/CD system trivial. We’ve been keeping an eye on new features and are excited to try out the new Markdown Job Summaries next!
We’ve also been very impressed by Xcode Cloud, and last year we transitioned a number of our workflows, like internal build distribution, to use Xcode Cloud and TestFlight. Currently, we’re exploring using the App Store Connect API from GitHub Actions to trigger and report on Xcode Cloud Workflows to automate our release process further.
We are also comparing performance on running tests in Xcode Cloud vs. self-hosted runners as Xcode Cloud runs tests in parallel with other actions, such as analyzing, archiving, and building, so we might get results quicker.
We look forward to sharing how these tools complement each other in a future blog post.
