Using Azure Data Factory to migrate data to microservices
By Rui Elói
Quality at Nmbrs isn’t restricted to the service we offer our customers — we also want to achieve the same high standards for our processes, to ensure our employees can work effectively. This will subsequently help us to make our overall solution better for our customers.
Our current goal is to migrate from a complex monolith to microservices on all parts of our product, which my squad has been working on. The aim is for each microservice to be in control of its own data. Our current product is a very complex monolith, so we wanted to make sure that after the migration, everything runs as it should and that data isn’t lost or corrupted during the process.
“We first tried the approach of migrating data by exposing an endpoint on the destination service and using an HTTP request to send the data from the monolith. However, that means coupling with the deployment cycles, and as a result, changes must go through code so it didn’t bring the flexibility we wanted. Moreover, it pollutes the code with the migration and data transformation logic.”
With this approach, for each tenant (customer), if we wanted to fix an issue we had to put logic into the code. If we’re migrating data across, and we wanted to filter this by entity type or customer, putting this into code is a laborious task. This consisted of turning filters into code, validating it, and checking that we haven’t hindered the overall service for other tasks. As we had to put in code logic — which is not business logic — only for the migration, we would have to do this recurrently, putting stress into the services which are used for the business itself.
We didn’t want to break current functionality so we’re carrying out these changes step-by-step. After creating the microservice, we’re focusing on migrating the data from the monolith — which is the master of the data — to the service. We found that Azure Data Factory (ADF) helped us as it established a connection between the origin and destination databases, and we could put in our own logic of extracting data from one to another without impacting any of the other services in the business logic. This has been the biggest benefit of ADF. We can also repeat this process with ease.
All transformations can be carried out in ADF without creating a single line of code in our web apps or APIs, and since it is an external tool it doesn’t pollute the code. Apart from the ADF, we also have in the current code a Message/Event pattern that helps sync the data between the monolith and the microservice.
ADF has helped us to find a lot of the issues as otherwise if we had used code there would have been no way to validate the data; this way we can compare the two sources and see what’s wrong. The validation pipeline can be triggered daily or in a set number of days, we can also do it manually, per tenant or customer. This means that if we found an issue we can focus on that first and then ensure that it applies to our other tenants or customers.
While ADF has been hugely beneficial to us, there is a steep learning curve and it can be costly so developers need to proceed slowly step-by-step. Once they do this they can use ADF to monitor changes in a much faster way than code, and can also use ADF for other purposes.
