How we do Lightweight Deployments at Treehouse
At Treehouse we have a simple workflow for deploying software. This simplicity makes our work more enjoyable and boosts developer happiness. Additionally, it allows us to stay productive while multiple features are shipping.
In general we consider our master branch to always be deployable. For every new feature we create a branch from master and then open a new GitHub pull request when we’re ready to get feedback on the changes. After a code review and some testing on staging, the code is ready to ship. That branch is then merged to master. Finally, after continuous integration is green, we deploy to production. We use Capistrano to deploy most of our code.
This happens with:
$ cap production deploy
All of our designers and developers can deploy to production. We place a lot of trust on our engineering team. Giving them the ability to deploy to production builds confidence. It also ensures that they can quickly fix things if something breaks.
Creating a branch for every new feature has the benefit of isolating those specific changes. We can be confident that our changes can ship independently of other features that are being developed at the same time.
We usually create a branch from master for every new feature, but this isn’t always the case. Sometimes we have a larger set of features that get merged into a long-running branch. These branches are useful when there is a lot of churn that might cause instability in master. Merging between a long-running branch and master can result in merge conflicts that are difficult to resolve. Because of this reason, we usually try to avoid long-running branches if we can.
Feature toggles are a good alternative to long-running branches. We can review and deploy lots of small branches behind a feature toggle. Features can quickly be enabled or disabled through our admin interface. During development a feature is typically just enabled for Treehouse staff. When the feature is eventually ready, we enable it for everyone.
While some parts of the Treehouse application have been extracted into services, the majority of our code is still in a single codebase. Initially this process worked really well with a small team; however, engineering has grown significantly over the past couple years. We now have multiple teams totaling over 30 designers and developers. With multiple people deploying several times a day, we found our existing workflow difficult to manage. Without some guards in place, we ran the risk of having inconsistent code running on production. We needed to scale our deployment process as well.
Locking + Queueing
The first step to scaling this process is to only allow one person to deploy at a time. We do this by locking our servers during a deployment. This ensures that our deploys are consistent and helps prevent downtime. Capistrano has a deploy_lock plugin for managing locks. The plugin creates a temporary lock file on each server during a deployment and removes it at the end. If someone tries to deploy during this time frame, the plugin detects that the file is there and prevents them from deploying. All of this happens automatically, but there’s a also simple set of tasks for managing locks.
Occasionally, we need to remove a stale lock with deploy:unlock.
Now that only one person can deploy at a time, we need a way to queue up deployments. We first tried to manage this manually in our #dev channel. Someone would announce that they were deploying and others would need to wait until they were done. This sorta worked but it wasn’t a great solution. What we came up with instead is a release command that our chat bot responds to for managing deployments.
The release command usage:
- release — Print the current queue
- release enqueue <branch> — Add a deployment to the queue
- release dequeue <branch> — Remove a deployment from the queue
- release promote <branch> — Promote a deployment to the front of the queue
Here’s an example of how it’s handled with the bot:
When someone dequeues a release the next person gets an @mention letting them know it’s their turn. We have a dedicated #release channel in Slack now, so our other channels don’t get cluttered with these messages. This also prevents deployments from being queued in one channel and people in another channel not realizing it. The release command has also had the benefit of adding more visibility around deployments.
Our current workflow hasn’t changed a lot from the original process. Adding locking + queueing has worked pretty well for the past year or so.
Now when we deploy to production, we follow these steps:
- Branch from master for new features
- Code review with a GitHub PR
- release e <branch>
- Merge feature branch into master
- Wait for CI to pass.
- cap production deploy
- release d <branch>
There have been a few tweaks, but the core functionality has stayed the same. We’ve avoided adding stifling procedures. These simple changes have allowed us to keep a deployment workflow that we’re still happy with even as our team has grown while providing us with consistency and reliability.
If you enjoyed this post, we’d be thrilled if you clicked the ♡ below! And be sure to give the Treehouse Engineering publication a follow. 👋