Implementing Heroku Review Apps with Rails

Heroku’s Review Apps are an important part of shipping code at Opendoor. Gone are the days of “works on my machine” — instead, code reviewers, PMs, and anyone looking at your work gets a working link to a server running your code. Review apps also let us test infrastructure tweaks (like build tooling) that can be harder to iterate in local environments.

Review Apps in your pull request

Before you rush off to enable them for your teams, we found that a few subtle implementation details can make or break adoption of review apps. They come with their own tradeoffs that one organization might see differently than we have — we’ve done our best to document them below.

At the end of the day, we chose to think of review apps as “staging, but running your code,” which not only makes implementation simple but helps engineers have an intuition for the limitations of review apps. Technically this means at runtime Rails.env.staging? is true for review apps, and all review apps share the same database and services as our normal staging server.

Rails Environment Configuration

You might already have a central staging server, which will lead you to wonder “Should ‘review’ be its own Rails environment, distinct from staging?” If you choose to have a distinct environment, which would be mutually exclusive to staging, there are a few downstream consequences.

Thinking of review apps as a different environment adds another runtime permutation of your code which everyone may not remember as they add new features or during code review. This is what we went with having our review apps “quack” like staging, for which we already had well established practices.

For example, your code might also have runtime logic that forks based on Rails.env.staging?. Depending on the size of your codebase when first enabling Review Apps, you might have to audit a lot of code to figure out where to inject new instances of Rails.env.review?.

Additionally, all of your YAML configuration files (think secrets.yml, database.yml) would now need a distinct section for review. You could use YAML-style inheritance to share configuration and avoid duplication between environments, but we found that the YAMLs became increasingly hard to grok once we went beyond the usual development, staging, and production environments.

In the very few cases we did need to fork the logic at runtime differently between staging and review apps, we used code like this:

class DeployInfoService
  def self.review_app?
    ENV['HEROKU_APP_NAME'].to_s.include?('-pr-')
  end
end

Heroku Configuration

Again, you might have a staging environment set up as a Heroku app. If you want to think of staging and review apps as distinct entities, it would mean creating a new app for your Heroku pipeline from scratch (or cloning it from your staging app). Much like the Rails-level environment and code configuration, there could be a strong chance your Heroku settings between staging and review drift apart.

The other Heroku-level consideration for setting up review apps is how you decide to spin up plugins and services for each app. Do you spin up a new database for every PR, or do you share one? What about services like Elasticsearch or Redis?

Dollar cost is one consideration — it can be expensive to run many more instances of these services all the time. And then there’s time it takes to maintain whatever mechanism you use to load data into these services — do you copy it from staging or some other persistent source? How long in minutes would that process take? Or would you write some scripts that generate data?

Ultimately we chose to point our review apps to a single set of shared persistent data sources. They all look at the same primary databases, which are available instantly upon review app creation — no need to wait for data to copy to your app. We have yet to encounter a case where this has been an issue (i.e. someone deleting another person’s staging data unexpectedly) — additionally, many of the data sources are rehydrated on a regular basis, so in the event of an issue it wouldn’t be a permanent problem. We do spin up a new Redis instance for each review app, primarily so that our asynchronous jobs don’t collide.

Your team might make different tradeoffs — it might be easier for you to maintain seed data scripts, or the cost mechanisms might work in your favor. YMMV.

Domain Complexities

Each review app is its own Heroku app, so they all end up with their own domains like my-review-app-*.herokuapp.com. Depending on your product, this might be a no-op or have some frustrating side-effects.

If you use OAuth to authenticate to external services (Google, Facebook, etc), you’ll probably need to think of an alternative login path for review apps. Most OAuth services enforce a whitelist of allowed domains and do not support wildcards.

In our case we came up with a workaround for the one feature that requires external authentication, but a more scalable solution might be to have a proxy service between your review apps and the OAuth domain. Using the OAuth state parameter to store your originating review app URL and route accordingly, your proxy service could transparently pass OAuth information back and forth.

Another domain gotcha is around browser cross-origin policies. You might have resources like fonts or iframes that are configured only to be retrievable by your staging or production domains, but your new cluster of review apps presents a complication. In the case of static resources, S3’s policy syntax does allow for wildcards and makes it relatively simple to support review apps.

Those are the two classes of domain problems we encountered. It may be hard to predict the failures that will happen due to domain settings, so you might have to do some exploratory debugging the first time you setup a review app.

Worth It?

Despite some of the above subtleties, we think review apps are absolutely a worthwhile investment and have paid the initial setup cost back in spades. It took a week or two to iron out the kinks and get feedback from teams on what features weren’t working as expected, but since then they have been self-driving.

We’re looking for engineers of all backgrounds: it doesn’t matter what languages you work with now, we’re sure you’ll ramp up fast.

Find out more about Opendoor jobs on StackShare or on our careers site.