Provisioning Engineers with ChirpStrap

A story of using Ruby to glue together system tools

“A [person] is only as good as [their] tools.” is, I think, how the saying goes. Perhaps a more appropriate quote for engineering would be “An engineer is only as productive as their tools allow them to be.” For a software engineer, this encompasses the ability to have a productive development environment. In an organizational setting, this is especially important. Ad-hoc coordination on environment configuration or application setup between developers might work for a small team of two or three, but as more people are added, the ability to have everyone using the same conventions prevents headaches arising from divergences. Having to spend hours or days troubleshooting environment setup can quickly derail your ability to deliver value to users counting on the next big feature or bugfix. At Iora Health, we went from a complex, fragile, and time-consuming environment setup process to a simpler, faster, more reliable process by leveraging existing tools and writing some Ruby.

Boxed In with Boxen

When I first started working for Iora in February 2016 my first task was to provision my new MacBook Air using Boxen. (Aside: Always get the Pro when given the choice. I learned this the hard way.)

MacBook Air fans while running five Ruby on Rails applications at once.

Developed at GitHub for provisioning their developer machines, Boxen wrangled the power of Puppet and Homebrew into a package that could be endlessly configured and extended to fit a myriad of potential provisioning requirements. It was immensely powerful. However, powerful software tools can quickly become complex and unwieldy.

There was a standard refrain here among the engineers at Iora Health that Boxen almost never ran successfully the first time you ran it. I was told I’d need to re-run it at least once and it was likely I’d need to do some debugging and tweaking in order to get it to finish successfully. For this reason, when getting a new engineer set up with their laptop, we’d offer up another engineer for sacrifice at the Altar of Boxen to work with them to get their machine set up. Once the mission was complete, the new engineer’s first contribution was usually fixing whatever had broken in Boxen (often something having to do with a broken dependency or OS X and Xcode upgrades.)

When Boxen fails for the third time

After that ordeal, you’d try to avoid running Boxen ever again. This was the opposite effect Boxen was probably after. It wanted you to run it with some regularity, to ensure your environment stayed in a known good state (and presumably so you’d find and address problems earlier.) It was far from ideal, but it at least worked well for centralized configuration management. It was certainly better than ad-hoc setup processes or centralized dotfiles, methods the team had utilized before Boxen.

In July 2016, GitHubber and Homebrew project lead Mike McQuaid wrote a blog post entitled Replacing Boxen, announcing the deprecation of Boxen and detailing GitHub’s new approach to provisioning developer machines. The post was a spot-on summarization of our own problems with Boxen. GitHub’s solution resonated with us as well.

Our biggest problem with Boxen was that it took control out of the hands of the developer to manage much of their machine and environment. It used special Boxen-specific formulas for various Homebrew packages, non-standard ports and paths, and it did not react well to things outside of Boxen impacting pieces of the system it managed. Everything had to be done The Boxen Way.

GitHub’s new approach to provisioning leveraged standard Homebrew packages and tooling, and a small bash script for establishing a system-level baseline configuration. The simplicity and flexibility sounded like a refreshing proposition, compared to the overwhelming complexity and opaqueness of Boxen and Puppet.

What We Wanted

When discussing replacing Boxen, there was consensus among many of us that whatever solution we landed on, it should meet the following requirements:

• Allow us to install arbitrary Homebrew packages, and track upstream Homebrew versions (no forking)
• Be written in a language most engineers would be familiar with or could pick up relatively quickly
• Use a limited amount of Bash or other shell scripts that would be difficult to maintain
• Run on a new, blank machine quickly. (Without installing anything / preferably using tools available natively in macOS)
• Keep everyone on the same version of PostgreSQL
• Do things that Boxen did that we liked, such as installing required Ruby and Node versions, cloning and setting up all of our projects, and installing other dependencies we use

Searching GitHub for “bootstrap macOS,” there are many projects that attempt to provide a good provisioning tool for development purposes, but nothing really met these standards. Most were either too prescriptive, abandoned, written in Bash or Zsh, or used a tool like Ansible that would require too much upfront installation on a new machine to get going.

Invent It Here

In late 2016, we started incorporating what we call “Engineer’s Time,” where each engineer can devote up to 30% of their time each sprint to engineering initiatives. Engineer’s Time encompasses anything from small features or bugfixes, to technical debt reduction, or developer tooling, open source work, and this blog post.

For one of my first Engineer’s Time projects, I decided to spike on something that could replace Boxen. I spent a few days hacking together a Ruby CLI tool that wrapped Mike McQuaid’s strap.sh, provided some standard Rails app setup (mostly via homebrew-bootstrap), a Brewfile, NGINX configs, and some shared environment config to be sourced into the user’s shell (you can never truly escape writing some shell script.)

Soliciting feedback from Rusty (Director of Engineering) and Patrick (Staff Engineer)

I reached out to my coworkers Rusty (Director of Engineering) and Patrick (Staff Engineer) who had previously expressed opinions about our development environment setup. They provided a lot of good feedback on the spike, which was then integrated into what we ended up calling ChirpStrap. (Our main application is called Chirp. Chirp + Bootstrap = ChirpStrap. We are creative.)

ChirpStrap

Class diagram for ChirpStrap

There are a few aims of ChirpStrap:

• Targets the system Ruby version (on whatever the lowest macOS folks are still using)
• No external dependencies (Only stdlib gems)
• Idempotency
• Minimal vendoring. We vendor strap.sh but make no modifications. If something changes upstream in strap.sh, homebrew-bootstrap, or Homebrew itself, we change ChirpStrap, rather than resort to forking.
• Avoid non-standard service configurations (Use default ports, paths, etc.)

Ruby ended up being an especially good choice for a system utility like this, as Ruby’s system method and backticks make it easy to call out to other programs without having to do too much setup. The rich standard library and powerful templating library (ERB) make it easy to write powerful and flexible configuration steps.

ChirpStrap is organized into three main concepts: Steps, Projects, and Migrations.

Steps

Example of our ConfigStep, which sets up some standard environment variables, SSH, and tbm config.

Steps are small, focused classes for provisioning one part of the environment. They might be a small logical grouping of related configuration items or might be a wrapper around one command.

They’re required only to have one method, bootstrap , and an initializer that takes a hash of options.

They should also be runnable in isolation.

Our current steps:

• ComposeStep — Wrapper for docker-compose that uses our stack’s configuration
• ConfigStep — Adds some standard environment variables, SSH config, and tbm config
• DependenciesStep — Runs strap.sh , sets up hub , runs brew bundle with our Brewfile (Installs a common set of tools and applications, from jq and rbenv to Code and iTerm2)
• ElixirStep — Installs latest Hex and Rebar
• MigrationsStep — Runs all applicable environment migrations
• NodeStep — Sets default node version and installs various npm globals for all installed Node versions.
• ProjectsStep — Clones all configured projects and runs the project bootstrap process for them
• PythonStep — Installs python and various pip globals
• RubyStep — Sets default Ruby version and installs various gems for all installed Ruby versions
• ServicesStep — Installs Docker for Mac and sets up a number of services using docker-compose
• UpdateStep — Checks for updates to ChirpStrap and pulls them in

Projects

Our RailsProject class, which overrides our default Project configuration by using a UNIX socket and uses some code to run the project bootstrap process under the project’s version of Ruby, rather than the version of Ruby running ChirpStrap

Projects define common configuration for various types of web projects being worked. The default Project class has some common conventions around using GitHub’s “Scripts to Rule them All” approach, to delegate to a script/bootstrap , if present. Otherwise, it does standard setup such as installing the required version of Ruby or Node, required dependencies, and setting the application up in NGINX. We have different project types for Ember CLI, Rails, and Phoenix applications, which handle additional setup specifics for those frameworks (port vs. UNIX socket, etc.)

Migrations

Migration for removing our old .dev domain resolver and NGINX configuration files.

Just like database schemas evolve throughout time, so do development environments. Rather than circulating a message over email or Slack with commands everyone needs to run to migrate from some config to another or switch packages, or some other migration task, our preference is to write automated migrations.

A migration is a Ruby module that uses the module_function macro to define two public methods: applicable? and migrate! . We have a MigrationsStep which runs as the last step, which runs all applicable migrations. Writing migrations this way allows us to write environment-sanity restoring scripts and target them to specific configuration situations. For example, we used to use .dev domains, but needed to migrate away from this once Google announced plans to add .dev to the HSTS preload list. We updated ChirpStrap to use .localhost domains and wrote a migration to remove all the old .dev configuration, allowing a seamless upgrading process. Once we’ve ensured everyone is using the new configuration, we can remove the old migrations.

Configuration

Projects and various other settings are made configurable in a simple YAML file.

For projects, we maintain some conventions on naming of repos and paths, so in many cases we just assign a cname (Canonical name) to the project, and the rest of the attributes we need, such as the repository name and hostname, are inferred.

However, we have some older projects that don’t meet these same conventions, so we have the ability to specify those properties directly via repo and hostname directly.

Getting up and running

The process for getting up and running using ChirpStrap are fairly simple:

• First, an engineer adds their new SSH key to GitHub, and grabs a GitHub access token with read access to their repos.
• Next, they run chirpstrap -a $ACCESS_TOKEN
• They’ll be asked a series of questions, such as their name, email address, Iora username, and GitHub username that are used for by strap.sh, ChirpStrap, or other internal tools.
• ChirpStrap runs a limited set of configuration steps, until it installs some environment variables. Then it prompts for a machine or shell restart.
• Finally, running it again will run the entire set of default bootstrapping steps

After this, the machine is successfully provisioned and the engineer can begin working on features. The total process for a new machine is about 30 mins to an hour. chirpstrap can be run at anytime afterwards, and typically completes within a few minutes if all dependencies and projects have already been fetched, making it easy to keep one’s environment up-to-date.

How it has worked out

Overall, ChirpStrap has worked out well for the team. People are not afraid of running it, as they were Boxen, and risking their environments being hosed for the rest of the day. Though it has not been without some challenges, we’ve been aided by our ability to swiftly make small and drastic changes to our development environments without losing velocity on the work that matters for our care teams, patients, and operations.

Before we used Docker for our service dependencies like PostgreSQL and Elasticsearch, we relied on versioned Homebrew packages, which are few and far-between. They did not necessarily match the exact versions we use in production, making it difficult to maintain dev-prod parity. We had resorted to forking some versioned formulae to our own homebrew tap and maintaining those builds ourselves. We’ve since reduced our maintenance burden by only using Homebrew for packages for which we don’t require a specific version.

We’ve been able to stand on the shoulders of giants with the work of Mike McQuaid, Misty DeMeo, and others on strap.sh and homebrew-bootstrap, and are very thankful for their continued maintenance of those two projects, as well as the wider Homebrew ecosystem. Eventually, we may find that our needs diverge, but for now relying on those sets of tools has limited the scope of what we need to maintain ourselves. After all, we’re a healthcare delivery company and not a developer tooling company :). ChirpStrap has allowed us to maintain our focus on building our platform to support our great patient care and population health management.

Thanks for reading!

Does this sound relevant to you and what you’re doing with development environment provisioning or do you want to know more about how ChirpStrap works?

Feel free to reach out to me on Twitter (Disclaimer: views expressed on my Twitter are my own, not those of my employer, etc.)