3 Reasons Why You Shouldn’t Outsource Your Rails Upgrades

(from someone who gets hired to handle your upgrades)

Does the following sound familiar?

You found product/market fit and have since grown a small-to-medium size software engineering team. A few of your developers have been around for a majority of your application’s lifespan, but most of your team is somewhat new.

You hired additional developers because the cycle time was slowing. Your team is experiencing some growing pains. They’re focused on what’s next in the Product Backlog and keeping things stable for your end-users. They’ve been doing a pretty good job at that.

As new team members have come in, it’s been a challenge to solidify some of your processes around handling tasks like:

• Pull-requests
• Documentation
• Improving your tests
• Formalizing your QA process
• Mentoring
• Debating what tools you’ll use for what

Phrases like “technical debt” and “legacy code” show up in planning meetings but it’s not clear what should be done about that. You’ll find some time to deal with that one day. Right?

If we’re being honest with each other, just how important are those concerns anyhow?

Your stakeholders (or product owners?) have heard these phrases a number of times but don’t seem to “get it.” Your team doesn’t know how to best advocate for dealing with them. I mean…if they really cared about the underlying technical problems your team is experiencing, they would obviously prioritize them, right?

So, it continues to get tossed into the someday/maybe pile.

Someday.

Someday…your team will have fewer priorities on their plate and they can spend that spare time to address some of those issues.

The team stops raising these concerns. They assume that nobody really cares, or that it’s someone else’s responsibility.

A big problem — your Rails version falls behind

Eventually, someone raises a very specific problem. Your Ruby on Rails application has fallen a few major versions behind the latest stable release ( currently v6.0.x). Unlike most technical debt concerns, an overdue upgrade to your application’s programming language and/or framework is easier for engineers and stakeholders to get on the same page.

“The version we’re running on is no longer actively supported.”

“We will no longer receive ‘security patches’ until we update.”

“We should see some performance gains and ability to use newer features, too!”

Nobody wants their application hacked and/or data stolen and shared on the dark web — so the security angle is often the easiest-to-understand approach for getting buy-in.

The team agrees to start exploring an upgrade. Before you get too far ahead of yourselves, an engineer on your team volunteers to do a deep dive into exploring just how big of a project a Ruby on Rails upgrade might be. We always appreciate when someone raises their hand to be a trailblazer.

They create a new branch in git, begin bumping the versions, start trying to make sense of the configuration changes, wait…are all of our 3rd-party Rubygems compatible? Why are most of our tests failing? Wait, do we even have automated testing on that one critical component in our application? After a day or three of digging into this, something else pops up that your engineer needs to switch over to.

They can pick this upgrade up again next week. Next week comes and goes. It’s unintentionally fallen back on the someday/maybe list, again.

Sound familiar?

Upgrades are complicated…but perhaps not for the reasons you believe they might be.

From a technical level, there is an abundance of valuable information available online to help engineers work through coding updates that are required to perform an upgrade.

The primary problem tends to be with your team being able to gain momentum.

Upgrade projects are difficult to jump in and out of. We’ve heard many stories of how 1–2 developers got a week or two into an upgrade, then had to switch over to something else for a while. When it’s time to look back at the upgrade, they’re not able to quickly merge code that they and the rest of the team have been working on in parallel.

Reality sets in. Can you be successful at continuing to work on the application under the older version of Rails while also trying to get everything working in the upgrade Rails branch? Aligning these types of projects, in parallel, is difficult. Especially if the engineer(s) working on the upgrade are also jumping back into helping with the next priorities in the backlog.

At this point, your team searches online for someone who has more experience working on upgrades.

This leads them to find an agency like ours (Planet Argon) to ask the question, “Can your team handle the Rails upgrade for us? We don’t have the time to do it, ourselves.”

While we can respond with, “Yes, yes we can help you with that” …we’re becoming increasingly convinced that you shouldn’t delegate your upgrade to a team like ours. (At least, not quite to the degree that you might be thinking.)

Let’s dive into our arguments.

1. You’re underestimating the time commitment needed.

It seems straight forward. Your team has X hours/week to work on the Product Backlog. If you outsource the upgrade to another team, you might believe that your team will only need to spend a few hours each week collaborating with the external team. The conversation with stakeholders to discuss budgets is that it’ll have very little impact on your internal team’s velocity.

On the surface-level, it seems like a win-win for everyone. However, if you haven’t gone down this path before — you might not realize just how involved one or more of your engineers will need to be to help dig through problems that pop up during the upgrade process.

This is particularly true if there isn’t much in the way of accurate documentation or automated tests in your application.

“We have 150 failing specs related in this area of the application. How exactly is this supposed to work?”

“We merged in your recent upstream changes and noticed that your team is using syntax that will not work with the newer version of Ruby.”

“This upgrade is going to require that we replace the following Rubygems. Is that something that your team will be able to start transitioning to or should we try to do that ourselves? Again, how should this feature work?”

Knowing that some level of this will likely happen — you assign one of your more Senior Engineers to act as a Technical Lead/project liaison. It’s not their only project to oversee.

Is it a safe assumption that they can’t be 100% focused on the upgrade — because if they could, wouldn’t you just have them lead this themselves?

So…after the initial “We delegated this to an agency” honeymoon period, your Technical Lead is voicing concerns.

“This is taking far too much of my time.”

“I’m not able to focus enough on my own coding commitments.”

“I feel like we could have done this faster, ourselves.”

“I feel like a bottleneck.”

While every project is a little different, it’s been our experience that it is rare for our clients to accurately reserve enough time for them to also be involved in the upgrade project.

I wish we could say, “You need to be available at least 10 hours/week for our team” but it really depends on what state your application and team’s process is in at the moment.

This tends to be an ongoing discussion throughout each of our client relationships. It’s an ongoing struggle to achieve perfect harmony.

2. The great merge is going to hurt.

A logistical challenge with any upgrade project is going to be when the upgrade gets fully merged, deployed, and is considered done. Depending on how rigorous your testing procedures are (automated or not) when you hire an external team, there is a grey area where the upgrade is handed off to your internal team to take ownership of.

There might be changes to your hosting infrastructure, your deployments, your third-party tools, possible data migrations, etc. This shared area of responsibility, if not carefully planned out, can create some frustration points.

We’ve encountered several scenarios where our team will get the major upgrade bump essentially finished, but the client’s team will be focused elsewhere at the moment. They don’t have the time to have their QA folks focus on this area, in-depth, nor does it seem like the right time to get the project pushed to production.

Why not? Risk.

“There might be new bugs introduced…so we’ll wait for a quieter month in our schedule to roll it out.”

“This requires data changes and we need to test that out more before we push it out as rolling back will not be a good option.”

While we have a number of clients who have been able to efficiently roll out an upgrade, we’ve also several sit on an upgrade branch that we “delivered” for a few quarters.

As I’m typing this; I am thinking about a client we wrapped up working with nearly a year ago. We’re still not certain they finished rolling that upgrade out. Their DevOps team wanted to handle that and from a stakeholder perspective, “it’s nearly finished” seems to be equivalent to “no longer a big problem”.

“Thanks, we can take it from here!”

…but did they?

3. You have a process problem; not a technical problem.

Your team of Ruby on Rails developers is capable of handling an upgrade — when it comes to the code. There are many articles available to help guide them through the common changes between each major version of Rails.

It’s quite possible that they’ve even participated in a few Rails upgrade projects before.

What they might not have a lot of experience with is…

• Explaining to stakeholders why upgrades are important.
• Being able to track, measure, and report their progress throughout the upgrade.
• Knowing how to best divvy up upgrade work amongst multiple team members so they aren’t all trying to solve the same problems, in parallel.
• Getting multiple team members up-to-speed with the project.
• Figuring out a healthy way to rotate who is involved in the upgrades.
• Fighting the urge to debate a, “it would be easier if we just rewrote it.” conversation (hint: no, it wouldn’t).
• Handling the scenario when we need to temporarily hit PAUSE on the upgrade to shift gears to focus on another company priority (without losing too much momentum).
• Finding sanity when there are over 1000 failing specs and trying to debug and resolve that.
• Exploring ways to make upgrades safer and faster in the long-run. (ProTip: Setting up a dual-boot approach)
• Making sure you aren’t reliant on a single developer to see it all the way from start-to-end.
• Avoiding burnout when it ends up taking far longer than your team expected it to.
• ..and most importantly, not having the next upgrade get put off for another 3–4 years.

Depending on how your upgrade project pans out, it could very well lead to introduce some residual trauma amongst your team and stakeholders.

This is, ultimately, why teams like yours reach out to companies like Planet Argon. You want your organization and team to get over these hurdles as painlessly as possible.

There’s an illusion that once you get caught up — your team will be able to successfully be able to keep up going forward. If that were true, wouldn’t you have been able to keep up with the latest version of Rails from the start of the project, all along?

Unless you make a commitment to making tactical changes to your regular development cadence, you will end up in this painful position, again.

Okay, so what should you do?

First, you need to admit that you have a bigger challenge here. There is a localized goal of being able to cross off a sizable task from your todo list. It’ll probably feel great but if we’re being honest, we both know that you’re going to find yourself in a similar position, again, in the next 1–2 years.

This is where we strongly recommend that you approach this as a global challenge with your team. That is if you want to avoid having to come back and try and sell an upgrade to those controlling the budget in the not-so-distant future.

“Did we not just pay for a big upgrade a little over a year ago?”

We both know that conversation isn’t going to be met with a blank check. So, you’ll opt to wait another year. The price tag increases.

Focus on the processes that led to your team falling behind

What your team needs is a cohesive strategy and processes for your team so that this just becomes part of your team’s regular workflow and routine. What if we could automate some aspects to upgrades while also making sure that your team has the authority to keep moving things forward, without it needing to put the rest of the Product Backlog on hold?

What we recommend here is to outline a process blueprint for your team. A guide that will help your team determine who is responsible for what, the frequency of when certain updates should be made, and to maintain a regular scorecard to help highlight whether your software is starting to fall off the rails, again.

Pick a strategy for technical debt repayment — and stick to it

With the constant go, go, go of new product development, technical debt is the unglamorous work that gets sidelined indefinitely. We have heard stories about this over and over from engineering teams of all sizes on our technical debt-focused podcast, Maintainable.

A few recommendations:

• Mariah Howard: How To Discuss Technical Debt With Product Managers
• Upgrading Ruby on Rails with Eileen Uchitelle
• Making is Easy, Mending is A Challenge with Sandi Metz

Part of solving your ongoing problem is to give your developers time to fix these issues as they pop up. This might require specific members of your team to focus on technical debt or to have some sort of backlog system for this type of issue. Whatever strategy you pick, stick to it, and hold your whole team accountable — including looping in your stakeholders to set expectations.

Hire consultants to focus on the long-term problem, not a quick fix

Outsourcing the bulk of your Rails upgrade is a temporary solution to a long-term problem. If you want to avoid needing to outsource this type of work every other year or so, we encourage you to seek a team that can focus on helping guide your team through this process, yourselves.

We believe that your team is fully capable of getting these issues ironed out. Having worked on ~100 Ruby on Rails applications since 2005, we have a lot of experience tackling the upgrades. As much as we do enjoy rolling up our sleeves and digging into a big gnarly upgrade, we really believe that your team will benefit more from doing more of the underlying work themselves.

We’d love to show you how.

Originally published at https://blog.planetargon.com.