Rogue bug: A DevOps Story

typing`$ git push heroku master` for the first time makes you feel like a DevOps Ninja


The DevOps methodology covers a wide variety of Agile software development and operations practices. On a basic level, it involves constantly deploying code when new features are available to multiple environments and servers, all while taking care to manage source control of an application or a set of applications and services.

For a more in depth guide about DevOps, check out:

My Experience

I assumed my first DevOps role while I was a student in my fourth module at the Turing School of Software and Design. My group and I were tasked with improving a live Turing enrollment application that communicates with another service for authentication. The app was written in Rails, was deployed in production on Heroku, and had a domain name.

Being students, we had limited access to the private repo code and to the production environment on Heroku. Therefore we had to create a staging environment that would essentially act as our “production” code, and when that code was approved, it could get merged into the active production application.

Some functionality regarding private information like payment and credit card processing had to be stripped out for staging; the database also had to be scrubbed so that our staging environment did not have access to sensitive production data. Thanks to the well-written Heroku docs we didn’t have too much trouble creating two new apps with domains and deploying them to “production” environments with “staging” in the app name. The problem was that once they were up, the apps did not communicate in the same way they did locally in development. We were experiencing redirects even though we were certain that we had all the information we needed to authorize the app, verified by testing, investigating params and following the Heroku logs.

This whole process took a week and a half to resolve, and was the source of significant frustration for my pair and I; we were used to debugging code and working with frameworks that have verbose errors, not this mystical land of Dev and Ops.

Working on a “brownfield” project

This project also marked my first time working on a true “brownfield”, or pre-existing, codebase. The repo had many contributors, services, and custom gems with limited documentation. We had to dig into the codebase and understand as much as we could before being able to implement new features.

The first task was fixing the test suite, which had a number of failures to track down. This type of work on a brownfield project is generally enjoyable, and was for us; we solved most tests in a day or two and got to understand many parts of the app. We also discovered some areas of the app that we were unsure about how they worked. One such area was a custom Ruby gem that fetched data from the authentication service for the enrollment app. The gem stored user data in a session cookie in the browser and used that for authorization on the main app.

The Bug

We assumed that this gem was the root of all of our problems; it was difficult to follow, and was the culprit of most issues in the test suite. We even ended up replacing most of its logic with our own service, but it was not the true source of our communications issue between apps.

As it turns out, applications in the domain are prevented from setting cookies for *

Both of our staging apps had domains and thus cookies on one could not be set by the other. In development, the app worked because there was nothing regulating the session cookies. The active production app was also in the clear, it was on the domain. The fix was simple, we set a custom domain name for the apps and suddenly they worked as expected!

What I Learned

I already didn’t know much about DevOps, but I learned that there is even more about DevOps that I didn’t know than I thought. Errors and complications can arise easily when doing this kind of work because of the integration of many different technologies and applications into one system. The more points of potential failure, the more failure you should expect. These failures can be frustrating, but the wins are worth it.

Some Tips for newbies

  • Use the Heroku docs for configuration and deployment, very well written and maintained
  • Use the Heroku logs for debugging, this grants you insight into what is going on between your app and the server
  • Take advantage of Heroku CLI Toolbelt commands like heroku config to check and confirm environment variables in your app
  • Use the Figaro gem to protect your secret keys when deploying to git and heroku