ANNOUNCING KAFKAESQUE, A MESSAGE STREAMING AND QUEUING SERVICE
Originally posted on the Kafkaesque blog.
We are happy to announce the arrival of Kafkaesque, an exciting new messaging infrastructure service. Kafkaesque brings the latest messaging technology and makes it available virtually anywhere, in any cloud provider. Since it is a fully managed service, you don’t have to worry about operating this messaging infrastructure because it is done for you.
APACHE PULSAR
Kafkaesque is powered by the open-source Apache Pulsar messaging system. Apache Pulsar is proven technology that has been used in production for several years at Yahoo!, who open-sourced Apache Pulsar back in 2016. To connect to Kafkaesque, you use the open-source Apache Pulsar clients. And with an active open-source community behind it, there are new features and capabilities being added to the Apache Pulsar clients on a regular basis.
NO CLOUD VENDOR LOCK-IN
Our service is designed to run in all major cloud providers (Amazon Web Services, Microsoft Azure, and Google Cloud Platform). Kafkaesque exclusively uses cloud-native technology, so it works seamlessly in all cloud environments. So no matter where your applications run, you can connect them using Kafkaesque. If you move from one cloud provider to another, that’s not a problem. If you need to send messages between applications running in different cloud providers, we can do that too.
And if you eventually decide to do it yourself, you can do that. Since Kafkaesque is based on Apache Pulsar, you can spin up a cluster yourself and your applications can be switched. The choice is up to you. No lock-in, here.
MESSAGE QUEUING AND MESSAGE STREAMING TOGETHER
Typically a messaging solution is either good at streaming messages, where you are dealing with a high volume of messages in real-time performance with simple pub-sub messaging patterns, or queuing, where you are dealing with a variety of complex message exchange patterns, such as competing consumers.
The technology behind Kafkaesque makes it adept at handling high-volume pub-sub messaging as well as the more complex messaging patterns typical in a message queuing system. And these complex messaging patterns are handled by the service itself, not left to the software developer to code around using a complex client library.
MESSAGE RETENTION
In a traditional messaging system, the system keeps track if a particular message has been consumed. Once the consuming client is done with the message, it acknowledges the message which tells the messaging system that message is no longer needed. A traditional messaging system will then delete the message from its persistent storage. After all, the message is no longer needed.
In a perfect world, that may true, but in the real world, things go wrong, applications crash, availability zones go down and being able to get that message back may be important for rebuilding your application state. That’s why message retention is important. If something goes wrong, Kafkaesque can replay the messages that have been published to a topic, even if they have already been consumed. Because you never know when you might need that message again.
The ability to retain messages also enables event-driven application architectures such as event sourcing, where it is important to record each change of state as an event in the order it occurred.
EASY GEO-REPLICATION
Replicating messages to remote locations is important to support disaster recovery or to enable applications to operate at a global scale. If the users of your application move around the world, you want them to have the same user experience no matter where they are. With geo-replication, applications can connect to the local cluster but can send and receive to clusters around the world. The messages are persistent, so messages that were sent during Airplane Mode can be picked up once the plane hits the gate.
With Kafkaesque, geo-replication of messages is built in. If you publish a message to a topic in a replicated namespace, that message is automatically replicated to the configured remote geo-location or locations. No complex configurations or add-ons, just a couple clicks and you are good to go.
TRY IT OUT
You can try out Kafkaesque right now. Just sign up for a free account. And if you have questions or feedback, contact us. We’d be happy to hear from you.
