We successfully upgraded a large Rails application from Rails 3.2 to Rails 4.1. It took us 20 months, with around 24 developers working across 4 teams.
We did this without having a separate feature branch for the Rails 4.1 upgrade. The two tools we found that enabled us to do this effectively was forming a team, and using a Rails version toggle. The rest of this post will tell you our story.
Hard Date
Around October 2014 our company’s flagship product, PackManager, was still running Rails 3.2. The engineering team realized this was an issue since this Rails version would reach end of life on Q2 2015, and therefore, it would no longer receive severe security fixes.
Given that security is one of our top priorities, we decided to start the upgrade right away.
For more context, PackManager has ~400 ActiveRecord classes, 530k lines of code and 216 gems. We have 4 teams working on this code base (we call them PM teams) with four to six developers on each team.
No Progress
After seven months, little progress had been made. This upgrade was a big daunting task that nobody owned. This was a recipe for failure — something had to change.
Insight: Form a Team
We decided to use the best tool for a big project: a team.
I created the Rails 4 team composed of representatives from our original PackManager (PM) working teams. This cross-sectional Rails 4 team met bi-weekly (sometimes weekly) to split the upcoming upgrade work into smaller tasks so that the PM teams could finish them during an iteration. During these meetings we also shared our learnings from previous tasks.
The Rails 4 team helped to coordinate the work done by the PM teams and kept them focused on a common goal.
Rails Version Toggle
When we upgraded our app to Rails 3 a few years ago, we used a long running branch. Given our app was already very big, this upgrade took a few months, making every git rebase very painful after a few weeks. On top of that, once the long running branch is merged and deployed, it is almost impossible to rollback the Rails version in the production servers.
This is why we decided to use a version toggle this time around.
This means upgrade code changes were going to get merge to master all the time. The implementation of this version toggle was using an environment variable. Here are some examples on how it was used:
if ENV['RAILS_4'] == 'true'
gem 'rails', '~> 4.1.14'
else
gem 'rails', '~> 3.2.22'
end…
if ENV['RAILS_4'] == 'true'
has_many :bookmarks, -> { order("name") }
else
has_many :bookmarks, order: "name"
endThis way, by setting the RAILS_4 environment variable before running the application, we can decide what Rails version to use without needing to change the codebase. This has the bonus of quick rollbacks in production servers if a bug is found during the version upgrade.
To implement this, we had to also keep two Gemfile.lock files, one for each Rails version, since they use different gems. We had Gemfile.lock for when running Rails 3 and Gemfile.lock.rails4 for Rails 4. Then, instead of just running bundle install, we used the following script:
#!/bin/sh
if [ "$RAILS_4" = 'true' ]
then
cp Gemfile.lock.rails4 Gemfile.lock && bundle install && cp Gemfile.lock Gemfile.lock.rails4
else
bundle install
fiWhenever we wanted to add a new gem or switch to Rails 4, we would run the script:
RAILS_4=true ./bundle_rails.shAnd to start the server:
RAILS_4=true rails sBuilds and QA Servers
During the upgrade, we duplicated all our builds and QA servers so we could keep our test suite running green in both versions, and QA could easily switch between Rails version when testing a feature. Creating all these builds and servers was very easy because it was only a matter of setting the environment variable RAILS_4 to the correct value.
Resource allocation
We didn’t stop feature development during this upgrade, so it went very slow as expected. Every team would fix a few tests every week. However, the company decided to allocate the first 2 weeks of 2016 to work full time on the Rails upgrade. During these two weeks we fixed most of the tests, completed the upgrade to Rails 4.0 and started the upgrade to 4.1.
Coordinating the work of 20 developers on the same project is not an easy task. During these two weeks, a focus on communication and collaboration was important. I sent daily status emails, visualized progress using burn up charts, and used Trello and Google Sheets to distribute the work. It was an intense 2-week sprint, but a lot of progress was made which is shown in the following charts:
These two charts show progress towards fixing all the failing tests. These charts were included on the status emails and displayed on a big screen. They helped a lot in motivating everyone to work toward crossing the finish line.
Tests Pass
Once all the tests were passing, we were ready to start doing some manual testing of the application. Each team had sessions of 1–2 hours where everyone, developers, designers, product managers and testers, allocated time to test different parts of the application.
Last Minute Catch
When we were only a few days from upgrading our production environments to Rails 4, we found two issues. One performance related, the other one around new assets not being updated. The two issues were hard to resolve and pushed our planned release date for Rails 4 to production back. But thanks to our version toggle we could easily delay the switch in production until these issues were fixed. This was a big win for the version toggle!
Final Deploy!
The upgrade in the production servers went very smoothly. We did it during work hours and it was very quick since it is just a matter of setting an environment variable, running bundle install and restarting the application. All that without incurring any downtime. We were also ready to downgrade if necessary although there was no need for it.
Summary
If I have to pick the best two tools during our Rails upgrade, I’d pick “forming a team” and “version toggle”. The Rails 4 team was the locomotive that kept the project moving forward and on track. The version toggle was our protection system that gave us the options to delay or rollback the upgrade in our servers.
Enjoyed this read? Click the♡ ! Want to learn about how we make major engineering changes? Follow us.
This article was written by Nulogite, Arturo Pie.
