Migrating the App Directory from Monolith to Microservice

If you’ve ever worked with an old legacy codebase, then you probably understand the value of refactoring. Code refactoring is the process of restructuring existing code without changing its external behaviour. The goal is to improve code readability, and more importantly, reduce complexity. Both of these allow developers to add and repair code more easily, improving throughput and productivity.

Background:

The Hootsuite App Directory was built using MongoDB and deeply coupled with Hootsuite’s legacy codebase (aka “The Monolith”). To decouple from the monolith, the team created the App Directory Service (ADS), a Scala microservice using a MySQL database. MySQL was chosen over Mongo due to it’s strongly typed fields and relational structure. Mongo’s type restriction features are much simpler and we have encountered errors caused by inappropriately saved data (read more about MySQL vs Mongo). Moreover, the MySQL database could also support Organization Apps (Apps that can be launched to a specific organization and only accessed by members of that organization). The goal of the migration was to reduce the dependency on Hootsuite’s legacy codebase by migrating to a microservice owned by the team, and also consolidate apps and organization apps. It is also part of a long term project to consolidate several representations of the same integration concepts into one cohesive solution. The migration of the App Directory data allows us to combine existing “social networks” into a single schema, significantly reducing the complexity of Hootsuite’s architecture. The project involved refactoring the monolith’s code to read and write from the microservice. The process was broken down into three major milestones:

1. Refactor the monolith to read and write from both Mongo and MySQL. MongoDB was still the primary database, meaning users were served data from Mongo and not MySQL.
2. Refactor the monolith to read and write from MySQL first, making MySQL the primary database and Mongo the secondary database.
3. Refactor the monolith to read and write only to MySQL, thereby removing the need for Mongo.

The original plan drawn out on the whiteboard

Challenges:

The plan evolved as we explored the codebase and realized we underestimated the complexity of the project. We had several forks in the road, forcing us to make a decision on how we should move forward. In each case, we documented the possible approaches and had a conversation about which one would be best. One such case was when we decided to remove our dependency on a ID generation service called Snowflake. We used this service to auto generate IDs in MongoDB, a feature which Mongo lacks. MySQL, on the other hand, can auto increment primary key IDs upon insertion so we decided Snowflake was no longer needed.

Overcoming these challenges:

There were five models that required refactoring. We decided to do it one model at a time, reflecting and learning from each iteration. We refined our process, learning to not prematurely refactor the monolith’s code and investigating and consulting the rest of the team for non-trivial changes and decisions. We also maintained a checklist of items that needed to be completed for each model and updated it as we moved forward. Refining our process and documenting along the way has improved our productivity and made it easier to complete this long migration. Even though I won’t be here to see it to completion, it has been an amazing learning experience and taught be a lot about the intricacies of a legacy codebase.