Moving mountains: How we succeeded in migrating huge datasets into Birdie
This piece is the first of a series of pieces covering the technical challenges Birdie’s engineers faced when migrating enterprise clients into our system. Watch this space for more!
Picking the right tool for a task or job is essential. In fact, the difference between a job done right and a mess to clean up could rest solely on the tool used.
A key factor in making the right choice of tool is an understanding of the job at hand. For example, if you were cutting through metal, you would need a different tool as compared to if you were cutting through wood.
I’m sure you’re probably wondering if this is a DIY article or an engineering blog by now but stay with me for a moment.
Birdie is an all-in-one home healthcare technology that has been growing tremendously to help improve the lives of older adults. We provide a software solution to care providers all over the UK. This allows them to assess, plan, roster and finance the care they deliver to older adults.
We often provide our software to large partners like Home Instead who deliver care to tens of thousands of people. They have come from existing systems and therefore require that their data is effectively migrated over to our system.
The process of migrating over this existing data as you would appreciate presents a few challenges
- Migration implies care agencies start using birdie instead of their existing system and this is a significant operational change for their entire staff (think moving from Windows to MacOS)
- We needed to have a migration process that allows us to incrementally migrate data, as opposed to a one-off file transfer to accommodate scenarios where larger agencies with multiple branches and different migration dates per branch could do so with little stress
- There was every possibility that we would not have access to their existing system’s API. Instead, the customer will have to transform their own data to conform to a schema expected by birdie. Every care agency might have to have different flavours of mappings due to their existing implementation details.
- Whatever system we generated would need to be very reliable. It requires a high level of automated testing, the necessary observability (logs & artefacts’ archives) and appropriate monitoring in place.
To solve the challenges outlined above, you guessed it — the right tool had to be chosen.
Another factor to consider in making the right choice of tool is the ergonomics of the tool. You would want to have a tool that fits comfortably and allows you to execute the task easily.
My team was tasked to solve this data migration challenge. We had to make the hard decision of picking the tool to use.
The team settled on using Elixir for this purpose and there are a couple of observations that I made that confirm the choice of tool was the right one.
- Simplicity: the language offered uniformity not just for the sake of it but it promoted; simpler development, simpler testing, simpler deployment, simpler maintenance.
All the things needed to address the challenges mentioned above were self-contained within Elixir’s ecosystem. We could literally define the system in one language. Background jobs, queues and data persistence (plus much more!) had adequate implementations in Elixir. We did not have to include external solutions which would come with some extra overhead.
- Improved collaboration: this simplicity allowed two-thirds of the engineers, who had never used the language before, to contribute equally. We mobbed (group coded) on parts of the system the day after the Principal engineer suggested that we were going to use the language. The only prerequisite was an open mind.
This article would be painfully long if I included the detailed architectural decisions and technical considerations we made using Elixir to solve the data migration challenge but watch this space for articles on the topics enumerated under the simplicity observation.
To tie it up, clearly defining the task at hand to be solved goes a long way toward determining the tool needed. Once that box is checked, assess the tool that gets you to start delivering value with speed and ease, then go for it.
This piece is the first of a series of pieces covering the technical challenges Birdie’s engineers faced when migrating enterprise clients into our system. Watch this space for more!
We’re hiring for a variety of roles within engineering. Come check them out here:
