Microservices

Lessons on solving the problem in a distributed manner

The Definition

Here is one of the simplest definitions of what Microservices are:

Microservices is an architecture in which different component pieces of a software design are created and housed as individual, isolated services. Each is deployed separately and they communicate through well-defined network-based interfaces. — Michael Douglass

Normally in what is called a Monolith Setup,an application would be structured like this: Client and a Server — with the server responsible for all the processing that the client needs e.g user management, payments and other things. In a Microservice setup, you would have several mini servers or services, each with its own implementation and database that specifically handles one of these tasks.

Practical Problem

We recently had an issue at work, we wrote an application using PHP’s Laravel Framework and one of the features it needed to do was upload a huge chunk of data from excel files daily. After the initial upload of data, the follow up daily uploads would require that we find the following before inserting into the database.

1. We needed to find which records from the new upload that are also found in the previously imported data.
2. We needed to find which records are no longer appearing in the new upload data but were in the previously uploaded data.
3. We needed to find that which records were in the new upload that didn’t exist in the previous upload.

All these tree things have to happen at each upload and the best part of these all.

Infinite Loop Problem

The first plan was the usual, just get these records and loop through each one of the, comparing their details against what is already in the database using the search function — but there was a huge flaw with this approach. The number of records at each upload was enormous — how enormous ?— 20,000 records enormous. This meant the application would running an enormous amount of times. For example, if the new upload contains 10,000 records, the application would get each one and then compare it against 20,000 records from the previous uploads. We tried to do it with a small set of dataset and Laravel & MySQL failed us interms of performance.

Set Theory to the Rescue

We decided to focus on the other features of the application,ck on this later. Well later came sooner than had we had thought. It needed to be done and quickly. Being lead on this project meant, I was tasked to get this out. So I reached out to a friend, he mentioned something about vectorization in python using numpy but the problem is that we had built this application in PHP using Laravel and we had no time to rebuild it using python.

I didn’t dismiss his idea though, with some research, I run into an article on the concept in Javascript and further reading revealed that we could archievethe three data comparison operations using Lodash Js which is based on Set Theory.

So we built a simple React Application that would read two files and get the records as JSON arrays. These Arrays would then be compared using Lodash’s IntersectionBy and DifferentBy functions. The former would find records that appeared in both arrays whilst the latter would find what records are different between the two arrays. We tried the application- to our delight it was able to compare arrays with 20,000 records and giving us the intended result in less than a minute. We refined the app and wrote tests and they all passed, confirming our theory.

Solution As A Microservice (SAAM)

After all this, we then had to answer the question of: how do we rebuild the application to capitalize on what Javascript had done? We didn’t have time to rebuild the whole app in Javascript. So what then?

Well that is when the concept of Microservices was pitched to me by a friend of mine, Fidelis from Nigeria — who I know from the Developer Circle Program. So instead of rebuilding the whole app, we could just take this part of the application and have it developed and deployed on its own langauge and connect it to the main Laravel application via APIs.

And that is what we did, we built a small Microservice using express JS, with its own database using MongoDB. We used Express JS as the framework for developing the Microservice(we tried Koa and Hapi)and MongoDB cause it offers bulk insertion via its insertMany API using Mongoose JS and its great performance when handling large data. For the records comparisons — we kept the use of Lodash JS. After all this, we exposed the Microservice to main application using APIs.

In the main application, we rebuilt all of the UI using React JS and only used Laravel’s Blade Template to serve the index page, where the React’s main component would be rendered. We then programmed the File Upload component to send data to the Microservice via its APIs for processing and storage. With this new setup, the MySQL database would only store user account information whilst MongoDB handled storage the rest of the data.

The New Architecture after implementing the Microservice

Impact of Using A Microservice

Performance was one of the things we noticed when we implemented this architecture. In our initial test setup, using PHP and MySQL database, we found that it took roughly 10 minutes to insert just 2,000 records without the addition of all those comparisons I mentioned earlier. Yet with the new setup, the application could process 20,000 records with all the comparisons in less 2 minutes on our Staging Server.

The main application also didn’t need to wait for the Microservice to complete its task before the user can interact with it—the user can do other things and only be alerted via a push notification when the microservice is done processing the data.

Challenges and Lessons of using the Microservice Architecture

Well the major issue with this approach is that, we now have to maintain another system which is implemented in language different from the main application. But the advantages this approach has give are worth it and it saved us on time and I personally enjoyed learning how to implement this.

Are Microservices Always Necessary?

Not really, if your application can do all its tasks without any issues, I would say stick with the Monolith design. Before jumping on this architecture, you should really understand why you want to use Microservices and what are the business benefits of doing so, some of which are covered in this article below.

The Benefits of Microservices Architecture for your Company!

That’s all for now forks, thanks for reading and please share if you can.