Feb 22, 2022
Build Serverless Streaming Architectures with Upstash Kafka
How Upstash Kafka is making event streaming serverless while being compatible with Kafka APIs?
This article is a part of the EdU Technology Watch series where we highlight new technologies and products in the event streaming and analytics space. Contact us to get your product featured here.
Event streaming platforms like Apache Kafka and AWS Kinesis enable capturing and storing real-time events at scale. Consumed by downstream applications, these events help quickly derive timely insights and react to events without losing their value.
As the incoming volume of events increases, a significant effort must be put to maintain these streaming architectures, increasing operational complexity proportional to business growth. For example, managing a production Kafka cluster demands a dedicated skillset. However, a managed Kafka platform can compensate for that at the expense of operational costs.
Over the past couple of years, computational technologies have gone serverless, allowing more developers to adopt and migrate their workloads into serverless architectures.
Can we expect the same for event streaming? Yes. Upstash Kafka is making it a reality.
This article discusses Upstash Kafka at a high level, a serverless Apache Kafka offering that simplifies building streaming architectures. Also, we discuss its flexible pricing model, Kafka API compatibility and conclude the article by building a simple use case.
Upstash Kafka: Serverless data for Kafka
Upstash Kafka is a serverless Kafka platform where developers can quickly provision a Kafka cluster and connect to it using native Kafka clients (over TCP) and REST API (over HTTP).
Kafka infrastructure is managed for you
As a developer, you only care for producing and consuming messages to Upstash Kafka. The rest, including provisioning, monitoring, and patching up Kafka clusters, is taken care of by Upstash, giving you more freedom to focus on your streaming application.
Per-request pricing
Upstash is a true serverless data platform where the price scales to zero. Users are charged per-request basis, allowing them to only pay for what they actually use. That differentiates Upstash from other vendors such as Confluent and AWS (Kinesis and MSK), where a fixed hourly rate is charged for the underlying servers/shards whether you access it or not.
Simply put, you’re not charged when your data isn’t in use.
Getting hands-on with Upstash Kafka
Now that we got to know Upstash Kafka for a little. Let’s explore it further by building a simple streaming use case.
The use case is a simple one. We will create a Kafka cluster in Upstash, and a topic called orders. We will create a Python producer and a consumer to send and receive orders from the orders topic.
Before you begin
First, you need to sign up with Upstash using your Amazon, Github, or Google accounts. Upstash offers you a generous free tier with the ability to send ten thousand commands or requests to Kafka, message persistence, and REST API support.
Once satisfied with the free tier, you can upgrade to a higher tier like pay-as-you-go or enterprise.
Step 1: Create a Kafka cluster and add a topic
Upstash provides a control plane to manage the Kafka clusters associated with your account. Once you log into the Upstash console, you can create a Kafka cluster by clicking on the Create Cluster button.
While creating the cluster, you can specify the region where it will be deployed. For better performance, you should choose a region closer to your location. Then you should set the cluster type to either single zone or multi-zone. A topic created in a single zone cluster keeps a single replica only; a multi-zone cluster keeps three replicas.
Once the cluster is created, you can create topics there. While creating a topic, you can specify the partition count and some advanced properties such as message retention period, retention size, and the maximum message size it should accommodate.
Step 2: Produce messages to the cluster
Now that you’ve created a Kafka cluster, waiting idle and ready to receive messages. Let’s produce some messages now.
Upstash gives you several options to connect a cluster and produce messages. The first option is to use any language-specific Kafka clients. In the cluster details section, you will see some code snippets for a few popular Kafka clients, including Java, Python, Go, and Node.js.
For example, this Python code snippet constructs an order object and sends it to the orders topic I created in the cluster.
from kafka import KafkaProducer
import json
producer = KafkaProducer(
bootstrap_servers=['Upstash Kafka cluster URL'],
sasl_mechanism='SCRAM-SHA-256',
security_protocol='SASL_SSL',
sasl_plain_username='replace with yours',
sasl_plain_password='replace with yours',
)order = {
"id" : 124234,
"date": "2022-02-23",
"total": 49.95
}future = producer.send('orders', json.dumps(order, indent=2).encode('utf-8'))
record_metadata = future.get(timeout=10)
print (record_metadata)
producer.close()
The second option is to use upstash-kafka, an HTTP/REST-based Kafka client built on top of Upstash Kafka REST API. It comes as a Node.js library that you can include in your project.
import { Kafka } from "@upstash/kafka"const kafka = new Kafka({
url: "<UPSTASH_KAFKA_REST_URL>",
username: "<UPSTASH_KAFKA_REST_USERNAME>",
password: "<UPSTASH_KAFKA_REST_PASSWORD>",
})const p = kafka.producer()
const order = {
"id" : 124234,
"date": "2022-02-23",
"total": 49.95
}
const response = await p.produce("orders", order)
The third option is to use the Upstash REST API, which gives you the flexibility to use a producer written in any language with Upstash. Apart from that, you can use your favorite Kafka CLI tools such as kcat and kaf to produce a message quickly and test the functionality.
Once the messages are produced, you can view the message count and the usage on the Upstash console.
Step 3: Consume messages
Consuming messages is simple as producing. You can use any option that I mentioned above.
For example, this Python code snippet polls orders from the orders topic.
from kafka import KafkaConsumer
consumer = KafkaConsumer(
bootstrap_servers=['Upstash Kafka cluster URL'],
sasl_mechanism='SCRAM-SHA-256',
security_protocol='SASL_SSL',
sasl_plain_username='replace with yours',
sasl_plain_password='replace with yours',
group_id='$GROUP_NAME',
auto_offset_reset='earliest',
)consumer.subscribe(['orders'])
records = consumer.poll(timeout_ms=10000)
print(records)
consumer.close()
The Upstash REST API, CLI tools (kcat and kaf), and upstash-kafka also support consuming messages from the Kafka cluster we just created.
Integration with serverless FaaS platforms
The serverless nature of Upstash Kafka makes it an ideal choice for producing and consuming events from serverless Function-As-A-Service (FaaS) platforms such as AWS Lambda and Cloudflare Workers. That way, you can build event-driven application architectures that are entirely serverless.
Typically, the serverless runtimes based on WebAssembly (Cloudflare Workers, Fastly Compute, Auth0 Actions, etc.) do not allow TCP connections. So you need an HTTP-based connection to access your Kafka cluster.
Fortunately, Upstash offers REST API, enabling you to:
- Produce messages to Upstash from within a serverless function.
- Trigger a serverless function whenever Upstash Kafka receives messages.
You can check out the following guides to learn more about this.
Summary
Upstash Kafka stands out from the crowd among many Kafka-as-a-service offerings by being a true serverless Kafka solution. Its per-request pricing and simplicity attract a great majority of developers to build entirely serverless streaming architectures.
Upstash Kafka is compatible with Kafka APIs, allowing you to bring in your favorite language-specific Kafka client to produce and consume messages. Also, its REST API is instrumental in seamlessly integrating with serverless FaaS platforms such as AWS Lambda and Cloudflare Workers.
Head over to Upstash Kafka docs and start building your serverless Kafka application today.
