Better late then never : Time to replace your micro-service architecture with Kafka…

Kafka already spawns and facilitated many organizations on micro-services architecture world. If Kafka is still not part of your infrastructure, its high time for you to go with it. I am not promoting Kafka better then any other message queue systems as many articles are already floating on the internet about this subject.

Kafka’s uniqueness is that it provides both simple file system and bridge functions. A Kafka broker’s most basic task is to write messages to and read messages from the log on disk as quickly as possible. Queue message will not be lost after the persistence, which is the core of the entire project.

Queues are called “topics” in Kafka and share one or more “partitions” on the same topic. Each message has an “offset” — an offset representing the location of the partition where it resides. This allows consumers to record where they are currently read and request the broker to read the next message (or messages). To read more on this go to my previous article in-depth-kafka-message-queue-principles. Multiple consumers can read data from the same partition at the same time, and each consumer reads data from one location independently of the other.

Use case when you don’t have Kafka in your micro-service architecture.

Let us understand Kafka’s use and impact, imagine a system that receives data externally through the REST API, transforms in some way, and saves the results in a database.

This architecture divided the system into two services — one to provide an external REST interface (Alpha service) and the other to do data transformation (Beta service). For simplicity, the Beta service is also responsible for storing the transformed data.

Now let us say as shown in below image we have two type of systems:-

1. Micro-service architecture that does not contain Kafka and coupled many other services with REST Interface(Alpha) and coupled with Data Storage(Beta).

2. Using Kafka’s and a pair of services.

Kafka reduced coupling

A system consisting of a pair of microservices(above point 2) is a little better than a system offering a whole service(above point 1) (which is often bad). So the Kafka, an interface for sending data from the Alpha service to the Beta service to reduce coupling.

Kafka reduce wait time

When data is submitted to Alpha’s service, it needs to ensure that the data is stored securely somewhere or has failed before responding to the client — the client needs to know because it can resend (or use some other Way to recover). With the REST interface services, the Alpha service will need to wait for the Beta service’s response until it responds to the client until the Beta service stores the data in the database.

Kafka, improved performance with independent modules

The Alpha service now requires the Beta service to be up and responding to requests — its normal operation depends on the Beta service and the Alpha service waits until the Beta service responds to the client — its performance also depends on the Beta service!

Without Kafka in core of micro-services architecture, you would be asking questions like below:-

• How is the Alpha service coupled to the Beta service?
• Beta service failure frequency?
• How often does it require downtime?
• What is its peak performance?
• Is it really only after the data is stored safely before it can respond, or early response is not feasible?
• If it depends on other services, the corresponding will need to further extend the coupling chain?

Most importantly the quality of a system like this depends on its weakest service.

If we use Kafaka Message Queuing as the interface replacement to above, you will get completely different results, Kafka has a trick: a Kafka message queue is persistent storage. When the data is safely queued, the Alpha service can respond to the request; we can be confident that the data will eventually be stored in the database because the Beta service is dedicated to handling the messages in the queue. Alpha services now rely solely on the normal operation and performance of Kafka — both of which can be far better than other microservices in the system. It is so loosely coupled to the Beta service that it does not even need to know its existence!!!

Another thing is custom load balancer, you can add a load balancer in front of the Beta service instance and replace the Alpha service from the instance that points directly to the Beta service to point to the load balancer. Load balancers need to be able to automatically detect instances of downtime so that once an instance goes down, the load can be moved to another instance. But a Kafka message queue by default supports a variable number of consumers (for example, Beta services) without the need for additional infrastructure support. Multiple consumers can form a “consumer group,” simply specifying the same group name when connecting to the cluster. Kafka will share the data for all partitions of each theme for the entire group of consumers. As long as we have enough partitions for the topics we use, we can continually add instances of Beta services so that they will share a portion of the load. If some instances are not available, their partitions will be processed by the remaining instances.

What if you need to add more services

Now let’s say we need to add some new functionality to our system and we’ll put it in a new Gamma service. The way it relies on Alpha service data is the same as the way the Beta service depends on Alpha service data, and the data is provided on the same interface. With your use case the REST interface, the Alpha service will couple an extra service, exacerbating the problem of inter-service coupling discussed earlier. With the increase in downstream services, the more dependencies become.

Using the Kafka interface allows the new Gamma service to read data from the same message queue as easily as the Beta service. As long as it uses a consumer group different from the Beta service, the two will not interfere with each other. Because the Alpha service does not need to know what services it uses for message queues it writes to, we can add services continuously without any impact on it.

Now you should re-evaluate and help me in the comment section what extra functionality or data flow you are getting when you don’t have Kafka.

The limitations of Kafka Message Queuing lead us to design better systems and thanks for reading so far. :)