Setting up Dry-Run Deployments on Heroku 🚀

Our team recently pushed a significant change to the data architecture of an application we support. This involved a schema change, a significant data transformation, and changes to application code.

Deploying this code would require us to take the app offline while the migrations ran. Because of that, we wanted to be extra sure that the build process would be quick and smooth. We wanted an easy way to perform dry-runs of our deployment against an environment that closely mirrored not only the state of the production app — in terms of application code and data.

The app is hosted on Heroku, a managed hosting platform. A bit of relevant background — Heroku allows you to have multiple instances of your application running. Normally we have a production instance running which the public has access to and a staging instance where we internally test out new features we’re working on. Heroku also offers a convenient command-line utility for controlling apps right from the developer's machine

Instead of taking over our staging instance for testing this large migration (we auto-deploy our develop branch to staging so this could get weird for other members of the team testing features that don’t yet incorporate these architectural changes) we decided to create an entirely separate app instance for testing the deployment.

Our first step was getting our new app (we’ll call it feature-staging) into the current production state.

After creating the app we needed to give it the same add-ons and configuration as our staging app. Copying all of the config variables over was going to be a pain, so we ran a quick script to do that:

This script leverages the Heroku CLI to grab details about one app and set details on another app. We run this from our local machine.

Note that config vars with the HEROKU_ prefix are set by Heroku, so we don’t want to copy those over. There were also a couple others, like DATABASE_URL that will be unique to the feature-staging app and those were worked out manually after running the code above.

Now that the environment is set up we need to give it some data. We needed this data to reasonably match the state of the production database, so we captured a backup from production and restored our feature-staging app’s database from that backup:

$ heroku pg:backups:capture --app=production-app  
$ heroku pg:backups --app=production-app 
=== Backups
ID Created at Status Size Database
──── ───────────────────────── ───────── ──────── ────────
b001 2018-08-01 14:21:51 +0000 Completed 1.62GB BLUE
$ heroku pg:backups:restore production-app::b001 --app=feature-staging-app

Again, this runs on our local machine and is sending instructions to our database which lives on Heroku’s servers.

Since we’re testing this deployment against production data we want to be extra sure that we don’t fire out any emails to real users. So we updated all users email domains to example.com

$ heroku pg:psql --app=feature-staging-app
=> UPDATE users SET email = concat(left(email, strpos(email, '@')-1), '@example.com');

(Note that the api key for our mailing provider was removed from ENV and background workers were never started, too)

Finally, we’ll set the feature-staging-app as a git remote and push master to it:

$ git remote add feature-staging-app https://git.heroku.com/feature-staging-app.git  
$ git push feature-staging-app master

This causes Heroku to build and deploy the current master branch to our feature-staging-app. Now, feature-staging-app mimics the current state of the application.

Now we need to test the deployment. We wanted to get some idea of how long the build and deploy steps would take, so we baked in some time logging into a dry_run rake task:

Again, we’re running this locally to effect changes on our app instance living in Heroku.

Note that Heroku by default will only trigger a build and deploy when changes are pushed to master. Our task, then, calls git push feature-staging-app feature-branch:master so that we push our local feature-branch to the remote’s master and the build and deploy are triggered.

Boom! Now we’ve arrived at our deployed state and have some feedback about how long this is going to take!

Of course, the whole purpose of this was to tweak and optimize. So to get back to our current state we have a reset rake task:

Now we can easily perform as many a dry-run deployments as needed, and easily get back to a clean current production state!

~ Dan, Software Developer