Our Journey with MicroServices

Growth Team@Hike Messenger

This is NUX from Mad Max: Fury Road, but at growth team in Hike Messenger it means a complete different thing which is “New User Experience

So, when the infamous NUX was launched to a some of new users, there was a downtime for status update service, which had nothing to do with NUX. This was because of the reason that status update service and NUX had shared code base. NUX was also trying to perform a full table scan during a particular scenario which had affected our API servers. This could have been avoided if there was service which abstracted out the table safe guarding it, instead of NUX directly accessing the table.

The Growth API server at hike is a giant monolith. It is like a swiss army knife which performs all the functions of organisation. Analogically giant monolith is like pond, if a stone is thrown into a pond, the disturbance produced propagates through out the pond in the form of ripples. Similarly, one service misbehaving can hurt other services hampering the uptime of overall monolith server.

It also has many such issues, major ones that can called out are

Issues with giant monoliths

Resilience issues, one systems outage affecting others like NUX

Scaling a single service of giant monolith is difficult, it usually involves spawning instances of entire system which is resource heavy and then routing the traffic of service that needs to be scaled to the newly spawned instance

Huge code bases intimidate developers, over burn IDE’s

Technology heterogeneity is not really possible because the developers are constrained to work within the existing stack. Less flexibility in choosing the right tool for tasks at hand. One would not want a plumber to perform a surgery right ?

Since all the developers work on shared code base, deployment is a hassle, requires a lot of coordination. At times there are unintentional dependencies, which leads to less confidence during the deployment. Thus, a rule of for deployment of API server allowed only during Tuesday and Thursday unless some critical fix has to be shipped.

Thus, MicroServices!

Microservices are small, i;e small in terms of scope, functionality and responsibility. They do one thing, but do it really well. This makes them extremely easy to deploy and maintain.

As long as the interfaces for consuming a service is defined and honoured, any changes made to it’s internals wouldn’t impact consumers. This gave us freedom of experimenting with different technologies and choosing right technology for any service. While breaking the giant monolith API server at into microservices we had noticed that every service had to:

  • Log data into analytics
  • Send packets to users
  • Interact with sharded-redis

This lead to a lot of code duplication and this duplicated code has to be reviewed and tested for every service. Since the code is getting copied, any bug fixes to added for the duplicated code had to be copied to all the services which made us realise that code is liability. As we all know that Necessity is the mother of invention. Hence, this lead to the development of a service called Pipy, whose primary objective is to extract out common code from all the services as modules and installing these modules should be as simple as

$pip install <module-name>

Thus, Pipy

Pipy, inspired from pip is a service that manages the repository of modules developed at Hike. It was developed with these design goals in mind.

  • Extracting out common code from the services to avoid code duplication
  • Optimising for developers productivity
  • Modules should be available only inside VPN
  • Easy installation, no devops dependency, which means installing these modules should be as simple as $pip install <module-name>

Working of Pipy

Working of pipy

Developers who want to add their modules, upload them to particular github repository. A codeship watcher, which listens to the changes of the repo, executes the test cases for that module. If all the tests are passed, a tar.gz of that module is created and uploaded to s3.

Users who want to use these modules access it via HTTP server which proxies S3.

Right now, pipy handles only python, but we as planning to extend support to other languages as well.

Then we have discovered some more issues with service discovery and monitoring which will be discussed in subsequent posts. :)

[ Keep Watching for knowing the issues we had encountered awesome tools we had developed during the process]

TL;DR

API server of Growth at Hike is a giant monolith, which had various problems. So, we started breaking the monolith into microservices. During the course of development of microservices issues such as code duplication, service discovery popped up. we had overcome these issues by developing awesome tools. One among those which was mentioned in this post is Pipy.