Reflections on an “Impossible” Upgrade

Upgrading Ruby 2.1.5 to 2.5.5 and Rails 4.2.8 to 5.0.7.2

Contributed by Talia Trilling, Senior Software Engineer @ eatsa

Most engineers have experience working on legacy codebases. Unless you’re an early hire at a startup, you probably don’t have much of a choice; even those of us asked to build out new codebases are usually first asked to cut our teeth on an existing codebase. These facts are in direct tension with technology culture on the whole, and the obsession with anything new and radical. I’ve always been a bit of a skeptic on that front (in that sometimes I think people overly favor what is new versus what is established), but my recent experiences as a backend engineer at eatsa led me to challenge this assumption.

eatsa was founded in 2014, within a very different technological landscape. Ruby 2.1.1 was used with Rails 4.1.1, which at the time were the most recent releases of the language and framework. As our codebase grew rapidly over the next couple of years, other than to move ahead two minor releases (to Ruby 2.1.5 and Rails 4.2.8), our stack stayed on the original versions. By the time I joined the company in mid-2018, my team was actively developing with this setup. After not much time, I became pretty sick of working on something that followed outdated standards and was difficult to troubleshoot. Additionally, it was a common problem to want to use a specific gem that was not supported by our version of Ruby, which led us to have to use less practical implementations.

My predecessors had investigated updating Ruby and Rails, but had come to the conclusion that it was unlikely that our stack would ever be updated given the size of our codebase, and the fact that we couldn’t take time away from feature development to invest in a potentially insurmountable improvement. Still, I had a nagging feeling that the issue could be solved, with the right approach.

When my boss announced we were doing a tech debt sprint and we could tackle anything we thought could improve our monolith’s performance, I jumped at the opportunity to upgrade our Ruby and eventually Rails versions. Although I found a large number of articles discussing literally how to increment the configuration files for an out of date Ruby version (as in, change your Gemfile to reflect the new version, run `rvm install NEW_VERSION`, re-run `bundle install` and ta-da, you’re done!), I didn’t see much out there about the more cumbersome reality of updating a massive application to actually be compatible with a significantly newer version of Ruby, which made me less confident I would be successful in my attempts.

However, after a couple of months of confusion, and interruptions to work on features, I have brought our system from running on Ruby 2.1.5 and Rails 4.2.8 to using Ruby 2.5.5 and Rails 5.0.7.2. I’ve learned immensely from this journey, and I wanted to share some of the lessons I’ve gleaned from this process (beyond that yes, you will need to update your Gemfile and re-run `bundle install`). There’s definitely a certain sense of shame associated with working on outdated tech, especially working at a startup, which I think keeps people from talking about it, so I am really excited to have this opportunity to discuss the issue openly and share some of the things that helped me.

Make sure you understand your deploy process

Part of my job is running code deployments for customers, and I considered myself to be pretty familiar with how our build process worked. Upgrading our Ruby demonstrated to me just how much I actually misunderstood, or took for granted, in our build process. We have a custom build framework written in Bash, and while it worked well when we never updated our Ruby or Rails, it quickly became my enemy. Our Travis CI instance seemed incapable of pulling the correct Ruby because of the changes, and then once that hurdle was overcome, some of our boxes started crashing because of an issue with how our build framework communicated with AWS Code Deploy (TL;DR, symlinks do not always play well with Code Deploy, see https://github.com/aws/aws-codedeploy-agent/issues/152). Prior to working on this upgrade, I had never written bash or even really read much of it, and I can’t exactly say I’ve become a pro in this department, but with some help from others, I was able to take out some of the more hacky logic that broke when the Ruby version changed, and make it so that in future upgrades only our `RUBY_VERSION` or `RAILS_VERSION` would need to be updated for our build process to pull the correct version.

This part of the process was particularly painful, and at times, I found myself frustrated with my lack of system-level knowledge. I do think that putting in the work to understand what was broken (and when that failed, reaching out to those around me for support) made me a stronger engineer.

Don’t assume your build passing on local == passing on a production (or production-like) environment

A somewhat related discovery for me was that my upgrades successfully functioning on my local environment did not actually indicate success in production or production-like environments in AWS. Prior to this upgrade, I had worked primarily on backend feature development, and while once or twice I had written code that worked locally and not on the server, it was usually because of faulty data or another external factor. These upgrades were an entirely different beast, and I learned that the hard way when I declared victory prematurely. At minimum, I now try to put these types of changes on our internal testing server before opening a PR, so that I can look for the most glaring errors and compatibility issues.

Accept that you may have to throw out some of your gems

An unfortunate reality of Ruby development is that if you are keeping your tech stack up to date, you are likely to discover that some of the gems you rely upon have been deprecated or left unmaintained, resulting in them not working as expected on newer versions of Ruby or Rails. The bad news here is that you have to become a little bit of an archeologist. Sometimes deprecated gems will link to gems that fill the same purpose (which is greatly appreciated on my part), but some gems seem to die a silent death, leaving you wondering, “did someone decide that this gem hit the epitome of greatness in 2013 and has thus left it as is? That can’t be right…” This can lead to a general sense of panic and confusion, but I can promise you that someone else has already run into the same issues, and you can usually follow their lead. For every gem I found that was no longer usable, I found a replacement gem, often one that captured the previous gem’s functionality fully without requiring changes to our codebase.

Start small, or, embrace incremental improvements!

Part of what delayed people on my team from working on these upgrades was that by the time the idea was raised, not only the current version of Ruby was deprecated, but the following version as well, with the version following that on track to be EOLed soon. However, the approach that I took was to try to incrementally upgrade, even with the knowledge that if we only got from Ruby 2.1.5 to 2.2.x, we would still be working with a deprecated stack. This piecemeal approach is what made the work fathomable, instead of a giant undertaking with no end in sight. In the end, I split the work as going from 2.1.5 to 2.3.8, then 2.3.8 to 2.4.4, and finally 2.4.4 to 2.5.5. Each of those splits was based on trial and error, and discovering the distinct section of work required for each upgrade, so that I wouldn’t drown in the work (and perhaps more importantly, so that the peers reviewing my PRs wouldn’t hate me).

Monitoring and testing are key

This should go without saying, but once you’ve made these huge upgrades, it is crucial to put the work in to test the functionality of your app, and to make sure that monitoring is in place in case of that process failing to capture issues. In our testing, we even moved the code to a production environment and created a complete transaction, including payment processing, in order to feel confident in the changes.

When all else fails, just give it a genuine effort

At the end of the day, the only thing that distinguishes me from the people who said the upgrade was impossible is my willingness to try — I had no idea if the upgrade was going to be possible, but I tried anyways, and eventually succeeded. As an engineer, learning to push myself to solve problems that feel impossible without letting the threat of failure dictate my actions has been immensely valuable, and I would challenge others to do the same if they are ever in a similar situation. At a minimum, you will learn more about your code base and increase your knowledge of Ruby, and you may actually find the situation is simpler than you feared. I now look at our code base with a less fearful eye — as absurd and self-indulgent as it may sound, I feel like the person who wrangled the Hydra, and I feel more ownership over all future development in this codebase. It is not an uncommon issue for a company to have an outdated tech stack, but I hope my experience can show people that what matters less is the problem and what matters more is the effort put in to fix it.