Event-Driven Architecture at Scale Using Kafka — Part 2

How to decide on the right Kafka cluster option

Amit Sharma

Published in

Engineered @ Publicis Sapient

4 min readJul 3, 2023

Problem Statement:

With the increasing adoption of event-driven architectures by companies, as previously highlighted in our original Medium article, there has been a growing demand for a fully-managed and highly-available Kafka service.

Solution:

AWS MSK (Cloud Native) and Confluent Cloud (SaaS) are two of the leading providers of fully-managed Kafka services. However, it is imperative to carefully evaluate their offerings and choose the right service to ensure the future-proofing of your product, cost savings, and optimization of your data pipelines. This article provides a detailed comparison of the key components of MSK, MSK Serverless, and Confluent Cloud, in order to aid architects in selecting the ideal platform for their event-driven architecture requirements.

Cluster Info

What are the key differences between the cluster setup?

Cluster and Storage Resizing

What is required to alter the capacity of a setup?

Monitoring

How much insight into cluster operations do we get?

Security

Logging

What logs are available to users?

Schema Registry

What are the data governance options?

Cost & Billing

Kafka Streams

Does the streaming use case have any differences?

KSQL DB

Confluent created application, similar to the Schema Registry, that allows easy use of the Kafka Streams API through a microservice with SQL commands

Included UIs

How much information can we easily see?

Disaster Recovery

What options are available for disaster recovery and replication?

Conclusions

Both AWS MSK and Confluent Cloud are managed Kafka services that offer scalability, reliability, and security for data streaming. It’s important to note that the comparison between AWS MSK and Confluent Cloud is based on the specific use case of event-driven architectures, and that the ideal platform for your requirements may vary depending on your business needs. Depending on your use case, you may choose one over the other. For example, if you are using an AWS-only ecosystem, MSK IAM Authentication may be a desired feature. Alternatively, if you are looking for minimal involvement in Kafka Cluster setup and maintenance, then Confluent Cloud’s offerings may be preferred. Ultimately, you should evaluate both services based on the needs and goals of your project.