The five reasons why distributed architectures powered by Kafka overpass classical software designs
Nowadays, cloud solutions are gaining popularity day after day among giant companies that once relied on on-premise infrastructures and high-performance computer architectures, also known as mainframe-based systems. This trend was first boosted by big tech companies, especially the ones defined as FAAMG companies. With that mindset in mind, I’ve decided to create an article exploring some of the underlining benefits of event-based distributed systems using Kafka as the main message broker against classical monolithic applications and mainframe systems.
For this post, I’ve created a quick tutorial using two Spring Boot Java microservices, Docker, Schema Registry, and Kafka. I also included some of the history behind computing systems.
Why are big companies switching their stack towards cloud computing?
The mainframe is, without question, one the most reliable, performant, and cost-efficient machines ever created by humans. No matter your arguments against it, we have to acknowledge its importance, especially during the 1980s until the beginning of the twenty-one century.
With the advance of technology over the last years, however, this power was surpassed by many supercomputers using concurrent systems, and by new distributed solutions using several replicas of a given service in a scalable manner.
Companies with massive market cap and unlimited tech budgets were committed to building their systems using mainframes in the past, especially when we’re talking about the finance industry. After it all, its huge upfront cost wasn’t an issue at all. Take the largest financial institutions for instance. The massive majority of them were mainframe-powered institutions until the early 2000s.
It wasn't until the launch of companies like Amazon Web Services in 2006 that this scenario would drastically change. Since then, most of the banks, governmental institutions, air travel companies, and so many others want to ride this wave of cloud computing. No company wants to miss this boat of modern software architectures.
When it comes to the different cloud architectures we have nowadays, async architecture stands out as one of the future replacements for traditional software designs. Some of the reasons that this will be the future of large-scale systems are highlighted below.
1 — Cost
There is no question that cloud-based solutions are cheaper for start-up and medium companies than mainframe software. So, how about 100,000+ employees corporations like banks? And companies that have already robust and reliable mainframe systems? To answer these questions, we have to see the big picture here. In the long run, sure, the cost of distributed systems is lower than paying for MIPS (millions of instructions per second) in a mainframe architecture.
On the other hand, the upfront cost in terms of creating the platform itself and the infrastructure required to support business-related services can be quite expensive. Even though it is a long shot, once they have this up and running, the cost will be drastically reduced over time. Overall, the cost of these systems is lower than the mainframe systems.
When we talk about monolithic systems, many similarities are shared with mainframe platforms. Comparing it with microservices, we see the opposite direction for the cost. In other words, the long term cost is way higher than creating and deploying several microservices. This is true because we can split the features in a way in which business value is deployed into production constantly with microservices. This cannot be achieved if developers spend months to ship a single application, which is true for monolithic software since they requite way more time to be developed.
2 — Labor
Now we’re entering an interesting topic here. Back in the 1990s, COBOL was a real thing. Colleges and universities across the planet had this component on their curriculum as it was in high demand in the market at the time. This scenario shifted by the beginning of the 2000s when new programming languages were emerging as replacements for mainframe systems. In the present time, thousands upon thousands of new graduates enter the market every single year with well-trained skills in software development and cloud computing.
Those companies, which once relied on COBOL developers, had to adapt to this new normal. In summary, there are way more cloud professionals than mainframe specialists worldwide. To add no this, many of today's COBOL programmers are now retiring, and we are expected to have a shortage of these professionals in the near future.
3 — Agility
This is perhaps one of the main advantages of having a distributed architecture based on microservices with async communication. The reason for that is because developing and deploying small features in a microservice platform is faster than developing a new subprogram or routine on a mainframe. In completed automated cloud systems, agile teams are able to deploy features into production in matters of hours if not minutes.
4 — Independency
No company wants to have one supplier only in the supply chain, especially when it involves the core service of this company: software. This is the case of the majority of companies that rely on mainframes. IBM dominates this market, accounting for more than 90% of all mainframes in the world. Public cloud providers, in contrast, are competing against each other in a healthy way causing the price to drop significantly in the last five years or so.
Even if the company decides to have its own datacenter, which happens sometimes, it would have no ties to a specific supplier since the market is diversified in this field.
5— Complexity
Finally, when big companies migrate their backend infrastructure to the cloud using distributed systems, they can develop complex backend services in a timely way. What I mean is that many of the operations that are performed by the frontend services can now be completely coded in the backend. This is true because most of the time when they use only mainframe computers, they cannot complete operations like image analysis using artificial intelligence, for example, inside the mainframe architecture.
As a consequence, most of the time companies use the frontend to do these operations, or even specific services to provide that. When a microservice architecture is chosen, in contrast, this operation can be achieved without any major technical barriers.
Kafka
Now that we’ve talked about the importance of having a complete distributed platform, let’s talk about the types of open-source solutions available to support such design. The Apache Kafka is one of the most popular message broker components used for managing and orchestrating messages. With last than 10 years of existence, Kafka has become a key component in almost all large-scale and cloud-based architectures.
Long story short, Kafka is used as a message exchange by microservices, services, ou third-party applications, like Hadoop, for data analysis.
Schema Registry
The integrity and control of the data that is being produced and consumed through Kafka are made by the Schema Registry. All the governance, rules, and structure are defined in an Avro file, and the Schema Registry makes sure only messages that are in line with these standards are allowed on Kafka.
Docker
Most of the public cloud providers do offer KaaS (Kafka As A Service) option, like EKS, and there are companies like Confluent that provides customer support for on-premise data centers as well.
When developing microservices with Kafka and Schema Registry, most of the time we need to run these components locally to test our software. This is one of the main reasons why we use Docker(one of many, of course). With Docker Compose we are able to run several containers in our personal computer, allowing us to run Kafka and Schema Registry simultaneously.
In order to do so, we need to create a file named Dockfile with the following images:
FROM confluentinc/cp-kafka-connect:5.1.2 ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components" RUN confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:latest
Next, we’ll create our docker compose file named docker-compose.yml
Finally, let’s launch our containers locally.
$ docker-compose upWhen we perform the command docker ps -a, we can see our containers up and running.
If you prefer, I have included this code on my GitHub.
Alternatively, you can also run Kafka and Schema Registry embedded using this awesome Java component developed by Marcos Vallim.
Spring Boot Microservices
Now is time to create our microservices. The idea here is to create two microservices in Java. The one named kafka-holder will contain an API endpoint that performs a payment. Once it receives this HTTP request, it will send an Avro event to Kafka and wait for the response.
The service kafka-service will then consume this message from Kafka, process the payment, and produce a new event stating whether the payment was processed or not.
The Avro event for the payment request is defined below.
The architecture created here uses Kafka and Schema Registry to orchestrate the messages.
Note that both applications are consumers and producers simultaneously. By doing this approach, we can scale up and down our microservices individually. This guarantee we are not wasting any resources, and we are using our money where we really need to.
Another advantage this architecture brings is the fact that no message is lost in case of any downtime of a given microservice. This is counting that we will have DLQ (Dead Letters Queues) and a good resilience solution in our project though.
In case we need to find out what happened to a given transaction, we also can check the Kafka broker and find this specific event there.
This post presented some general overview of the history of computer power and software development over the last decades. It also showed some key benefits of using distributed solutions when dealing with a large number of services and data.
I hope this article helped you understand why this trend of cloud computing and event-based systems came to stay, and why it will replace most of the ancient mainframe systems in the long term.
If you have any questions, please feel free to drop a comment.
Thanks for reading it!
