Wave Invoicing Migration — Part 1

There is a special kind of pain that comes from working on a legacy system.

At Wave, we have many production systems that are critical to managing millions of small businesses. Some of these systems are only months old, while others date back to 2011. We’re always evolving and updating our apps to meet the needs of customers, engineers, and supporting staff.

We generally want to approach these changes in small, incremental steps. Delivering software incrementally reduces delivery risk–it makes it less likely that we’ll make mistakes that go undetected, or that we’ll build something that nobody likes. You may have heard the old adage of eating an elephant one bite at a time, and we often use that macabre analogy in architecture discussions.

There have been times, however, when the elephant almost ate us. In this series of posts, we’re going to tell the story of how we did a large rewrite of Wave Invoicing. We hope that this tale will help you in your own re-architecture journeys.

The Wave Invoicing Story

We built Wave Invoicing in 2012, and from that time onwards it was hugely popular with service-based businesses who wanted to look professional and get paid quickly and easily.

By the summer of 2018, though, it was starting to show its age:

• The first version of Wave was built as a monolith (unlike the distributed Wave architecture of today), and different apps and systems from that monolith had concerns mixed in with the invoicing-specific code.
• It was very difficult to incrementally improve this, as there were many issues entangled with one another. Database access was permitted anywhere, business logic was often duplicated across modules or on both the server and client side, and dead code was hard to discern from live.
• Many of the early contributors to this version of Wave had moved on, and there was little documentation or context available on some critical parts of the application. This led to difficult-to-diagnose bugs that created a bad experience for our customers.
• The developer experience was poor because the remainder of the original Wave monolith was still quite large, which resulted in slow test runs, builds, and deployments.
• We were using old versions of many key dependencies, many of which had complex upgrade paths, not the least of which was Python 2.7, which was near its end-of-life date of January 1st, 2020.

These problems weighed on us, and it was becoming increasingly difficult for us to build new features in this environment–especially compared to other Wave teams who were working on newer products. We also worried about reaching a point where it would become almost impossible to maintain the quality of service that our customers expected from us, even if we devoted most of our time to “keeping the lights on.”

However, the prospect of rebuilding Invoicing was equally scary.

• Because much of this legacy system was unknown to many of the current contributors, it was hard to estimate how long a rebuild would take.
• In addition to rebuilding Invoicing, we’d also need to migrate all of our customers to the new version while maintaining the integrity of their data and without disrupting their businesses.
• This decision also meant creating some organizational debt. Wave Invoicing was a key product for Wave, and we had dedicated product managers and designers working on it. While they would be critical in supporting the rebuild, they wouldn’t be as engaged in this project as they would have been in delivering features. This meant that they’d likely be focusing on other things outside of Invoicing, which increased the risk of losing critical Invoicing product knowledge if the rebuild went on for a long time.

We’ve had to do a couple of these rewrites before, and this is always one of the most difficult things about it: It’s scary to leave things the way they are, and it’s almost equally scary to move forward! Perhaps you’ve felt this yourself on your own projects. It is a very demotivating place to be.

We were, however, encouraged by the things we had going for us.

The existing Invoicing experience was well-loved by our customers, and the web frontend itself was fairly modern. We weren’t looking to change the product experience at all: We needed to focus on the aging backend that was making it difficult to further improve on the product. While rebuilding a new backend to be feature-equivalent to the old one was still daunting, it was a simpler problem to specify than having to change the experience in the process.

Further, we had already rebuilt and migrated smaller parts of our Invoicing product, such as the engine that enables business owners to schedule recurring invoices for their customers. These projects went well, and helped us gain practical experience and confidence that we’d be able to do the same thing on a larger scale.

We were also able to draw on the ideas and strengths of other Wavers who had been through comparable experiences at Wave and elsewhere, and this diversity of perspective helped us break the larger problem up in a way that we were confident would help us reduce the considerable risks involved.

In the posts that follow, we’ll describe how we tackled this daunting project, and the lessons we learned in the course of our journey. We hope you’ll follow along! Perhaps you will learn something that will help you eat any present or future elephants that may be standing in your way.

Co-authored by Jason Colburne and Emmanuel Ballerini, edited by Michael DiBernardo. Thanks to Joe Pierri, Nick Presta and Joe Crawford for the reviews.