Software Deployment Pipeline w/ Heroku

Published in

UpLaunch

9 min readJul 22, 2017

In the software engineering world, there’s a TON of super smart people who share their knowledge readily and without compensation, and it’s helped us immensely while bringing the UpLaunch Platform to life. The way I see it is now that we’ve figured a few things out, it’s up to us to help pay it forward.

If you’re not a nerd, read on at your own risk ;)

Our Growing Needs

Our software has been rapidly evolving — started out as a proof of concept, then to a prototype, and then to a production-ready application that’s about to start taking on live customers. Similarly, our development workflow has also evolved. In the beginning, it was just me — coding, pushing directly to master, breaking stuff, no tests…cowboy coding at its best.

Once the development team doubled in size (yes, there are TWO of us now!), we needed to formalize the process a bit. We started actually organizing our work into epics and user stories, which at least gave us a way to measure how close we were to being able to go to market. We then created a process where each user story would have its own branch, and conversely, that any branch we created would have a corresponding user story. The program we use to manage that is incredible (total JIRA killer)…but I’ll save it for another post.

We eventually got a test suite built out, and I protected the master branch of our repo by requiring that all branches be merged via a pull request with an approved code review.

Sounds legit…but something was missing.

Protecting The Customers

We understood that we needed a “staging” copy of our app — a place to test things out, make sure that the features we released weren’t going to break things for our customers, etc. I wanted to have very granular control over what got sent to production — in other words, I’m cool with merging a PR and it getting automatically deployed to staging if the tests pass. But when it came to production, not only did I want to have to press a button, I wanted to know exactly which feature branches were making the jump.

Enter Heroku Pipelines.

You can read their docs here, but basically think of it as a collection of Heroku apps that fall into three categories — development, staging, and production.

Once configured properly, every open PR will spin up a “review app” — so you (and/or your team who is reviewing your code) can check out how the new feature will work in an environment that more closely represents actual production conditions.

After the PR is merged into master, we have it set up where the test suite will run (also on Heroku) and assuming it passes, it’ll automatically push to the staging app.

Last but not least, Heroku conveniently supplies a “Promote To Production” button on the staging app — with a single click, you can push the already-compiled code to your production app, and all is well.

The Devil Is In The Details

If it was as simple as I just wrote to get it up and running, I wouldn’t be spending time writing this blog post. It took a bit of messing around (due in large part to my own inexperience) to get everything working right, and I had to track down a few other people’s blog posts for guidance — so I decided to aggregate it all in one place.

I’ve posted the links to the blogs that got me across the finish line in the footer — huge thank you to them for the work that they did, and a lot of what I’m sharing with you came from them. I’ve simply updated it and put it all in once place.

The problems that I ran into are as follows:

Heroku Setup via app.json. Although Heroku generates a super helpful app.json as a starter, I needed to make some tweaks to it in order to get things working properly, which took some trial and error.
Subdomains. We use Basecamp-style multitenant authentication, meaning that an account gets assigned its own subdomain, and each account can then have multiple users. This doesn’t work for review apps, which are spun up as a subdomain of herokuapp.com (for instance, http://yourapp-name-pr-263.herokuapp.com). Sub-sub domains on herokuapp.com are a no-no.
Data. We knew that starting off with an empty DB would be counter productive, as making our QA guy create a brand new account, new user, new contacts, etc just to test a simple feature would be a HUGE time waster. However, we weren’t sure the best way to go, or how to do any of it.
Migrations. Heroku doesn’t automatically run database migrations, including when promoting from staging to production.

I’m proud to say that everything is up and running, after some trial and error and lots of bad words. The rest of this post will talk about how we tackled our four main problems.

Heroku Setup / app.json

When creating new apps that are based on existing ones (like review apps), Heroku uses an app.json file to allow you to set some specifications. This file should get saved in the top level of your Rails app and committed to version control. Like I said, I started with the Heroku-generated app.json, which puts every config variable you have into the schema.

Any config variable that you leave in the JSON schema as “required” will automatically import its value from the parent app. That’s really useful for things like API keys of essential services, but make sure you strip out the ones that don’t make sense — for instance, HEROKU_RELEASE_VERSION.

I also edited the app.json to include separate schema for the test environment, and to run rails db:test:prepare in the test-setup phase of Heroku’s CI workflow.

Honestly, it just took some good ole trial and error — Heroku’s builds will error out if your app.json is jacked up, so feel free to use mine as a reference to help get you started.

You can find an example app.json here.

Subdomains

Disclaimer: if you don’t use subdomain-style multitenant authentication for your app, this step won’t apply. But it has some cool stuff in it regardless :)

Setting this up was a big hurdle, and I NEVER would have gotten it up and running without Sebastian Korfmann’s article about handling this exact problem. The rake task that he provided needed updating in order to get things going, so I’ll provide the details in this article instead of simply sending you to his post.

I, like many of you, have spent the past number of years purchasing domains on GoDaddy. It’s one of those things that you know isn’t a great idea, but you’re not really sure what the best alternative is so you just keep doing it.

I’ve found my solution, and it’s DNSimple. The reason? You can programmatically create DNS entries for your domains via their API…and they have a Rails gem! This capability was essential to make our sub-sub-domain requirement a reality.

First things first: get yourself a domain that you want to use for review apps, or transfer one in. After that, you’ll need to install a couple of gems and set up your rake task.

Gems: dnsimple and platform-api.

Also grab your API key from DNSimple and Heroku.

Next, create a new file in lib/tasks and call it staging.rake. You can grab my copy here — just make sure you set line 12 with your review app domain, and set up the config variables with your API keys. In my example app.json, I added the config variable names in the comments.

Basically, this task, which is called at the end of the postdeploy scripts in app.json, calls DNSimple and adds two CNAME records to your domain which point to the review app, and also calls Heroku and adds your DNSimple domain as a custom domain for the review app.

In short, instead of accessing your review app at http://your-app-name-pr-267.herokuapp.com, you’ll now be able to access it at http://pr-267.your-review-domain.com. And it happens automatically, every single time a review app is built for the first time.

The nice thing for this is that you can then use your account subdomains — for instance, if your normal account was http://mike.your-production-domain.com, you could access the review app at http://mike.pr-267.your-review-domain.com.

Problem solved…except your account doesn’t exist on the review app. Yet.

Data

We talked a lot about this one. Our initial plan was to just share the staging database with all the review apps. We thought we were soooooo smart — all of our accounts would be accessible, blah blah blah. It worked great until we needed to make a database change in a PR.

What would happen if you run the migration on the review app?

Of course, now I was dealing with my deflated ego AND a database with a jacked up migration, so we kept looking for a better method.

Enter EpiphanyMachine’s answer on this StackOverflow post. Basically, on your staging (parent) app, set your postgres DB URL to a config variable STAGING_DATABASE_URL. Then, add pg_dump $STAGING_DATABASE_URL | psql $DATABASE_URL && bundle exec rake db:migrate to your postdeploy script in app.json. This will copy the staging database over to your review app’s fresh DB and run migrations.

Now you have your accounts, data, etc ready to rock and roll, but you can F it all up without consequence.

Note that in app.json, we now need to run the above code to set up the DB as well as the code to run the rake task. When it’s all put together, the postdeploy script should look like this:

pg_dump $STAGING_DATABASE_URL | psql $DATABASE_URL && bundle exec rails db:migrate && rake staging:publish_dns

Migrations

Last but not least. The postdeploy script above runs migrations when the new DB is set up, but remember — postdeploy only runs ONCE. So what happens when we promote from staging to production?

Enter the Release phase script. For some reason, I had trouble finding good documentation on how this works. Even though it’s probably my own fault that I couldn’t find it, I’ll hit it here as well to make sure everyone’s good.

If your app is already on Heroku, you should have a Procfile in the top level directory of your Rails app. For most people, it likely contains one or two lines; one starts with web and sets some config options for your web dyno, and if you have a worker set up as well, a second line that starts with worker.

We’re going to add a third line: release: bundle exec rails db:migrate. This will essentially run rails db:migrate every single time a new VERSION of your app is created. If you update a branch with an open PR, Heroku will run your migrations when it updates your review app. When you close the PR, the migrations will run on your staging app. And, of course, when you promote your staging app to production, it’ll run again.

Problem solved.

Conclusion

That’s a lot of stuff. However, it’s also a huge time saver once it’s up and running. As of now, all of our tests are running inside Heroku CI, and every time we open a pull request, we can all test it without having to pull down and run it locally. We’re still automatically merging closed PRs with green tests into master and onto our staging app, and finally, when I’m happy with the features that have been merged into staging, I can push it to production with one click.

Hope this all helps — please let me know if there are any questions.