☁️Huawei Cloud DMS for Kafka
Introduction
Hello to everyone! 👋 In this article, we will talk about Huawei Cloud Distributed Message Service (DMS) for Kafka. Apache Kafka is distributed message middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming, online/offline system analytics, and real-time monitoring. Distributed Message Service (DMS) for Kafka is a fully-managed service that delivers high reliability, concurrency, and scalability in real-time data transmission, stream data processing, system decoupling, and traffic balancing use cases.
First of all, we will briefly discuss the features of the DMS for Kafka service. We will then create a DMS for Kafka cluster and complete our demo with producers and consumers for an example topic in the demo. If you are ready, let`s move.
Huawei Cloud DMS for Kafka
Distributed Message Service (DMS) for Kafka is a message queuing service based on the open-source Apache Kafka. It provides Kafka premium instances with isolated computing, storage, and bandwidth resources. The computing, storage, and bandwidth resources used by an instance are exclusively occupied by the user. You can apply for instances as required. The instances can be used right out of the box, taking off the deployment and O&M pressure for you so that you can focus on developing your services.
Huawei Cloud DMS for Kafka provides easy-to-use message queuing based on Apache Kafka. Services can be quickly migrated to the cloud without any change, reducing maintenance and usage costs.
· Rapid Deployment: Simply set instance information on the DMS for Kafka console, submit your order, and a complete Kafka instance will be automatically created and deployed.
· Service migration without modifications: DMS for Kafka is compatible with open-source Kafka APIs and supports all message processing functions of open-source Kafka.
· Security: Operations on Kafka instances are recorded and can be audited. Messages can be encrypted before storage. In addition to Simple Authentication and Security Layer (SASL) authentication, Virtual Private Clouds (VPCs) and security groups also provide security controls on network access.
· Data reliability: Kafka instances support data persistence and replication. Messages can be synchronously or asynchronously replicated between replicas and flushed to disk.
· High availability: Kafka runs in clusters, enabling failover and fault tolerance so that services can run smoothly. Kafka instance brokers can be deployed across AZs to enhance service availability. Data is synchronized between different AZs based on Kafka’s in-sync replica (ISR) mechanism. A topic must have multiple data copies and distribute them across ISRs. When ISR replication is normal, the recovery point objective (RPO) is close to 0.
· Simple O&M: Huawei Cloud provides a whole set of monitoring and alarm services, eliminating the need for 24/7 attendance. Kafka instance metrics are monitored and reported, including the number of partitions, topics, and accumulated messages. You can configure alarm rules and receive SMS or email notifications on how your services are running in real time.
· Massive accumulation and scaling: Kafka features high scalability because it runs in a distributed system, or cluster. Users can configure up to 200 partitions for a topic. The storage space, broker quantity and flavor can be also expanded. This means that billions of messages can be accumulated, suitable for scenarios requiring high concurrency, high performance, and large-scale access.
· Flexible specifications: You can customize the bandwidth and storage space for the instance and the number of partitions and replicas for topics in the instance.
The final note for DMS for Kafka Service is Kafka instances are compatible with Apache Kafka v1.1.0, v2.7, and v3.x., for competency purposes, users need to keep the client and server versions the same. Next, we`ll go through the step-by-step instructions for installing DMS for Kafka service, creating a sample topic, and utilizing producer and consumer for the topic.
Produce and Consume Messages with DMS for Kafka
Prepare the Environment
- First, we need to create the DMS for Kafka cluster and the VPC for Kafka client services. For the demo, we can create the VPC with the following parameters.
Region: Singapore
Name: vpc-tl-kafkacluster (You can give a name whatever you want)
Ipv4 CIDR Block: Keep it as is. (1912.168.0.0/16)
Subnet Name: subnet-tl-kafkacluster (You can give a name whatever you want)
Subnet Ipv4 CIDR Block: Keep it as is. (192.168.0.0/24)
AZ: Keep it as is.
Keep the other parameters as default and create the VPC. As a result, the VPC we created for the demo was as follows.
2. Let’s build our Kafka cluster now. Select DMS for Kafka from the Huawei Cloud Console, click buy instance, and enter the following parameters.
Billing Mode: Pay-per-use
Region: AP-Singapore
Project: Keep it as is
AZ: AZ1, AZ2, AZ3
Instance Name: kafka-cluster
Enterprise Project: Give your project name.
Specifications: Cluster
Version: 3x
Broker Flavor: kafka.2u4g.cluster
Brokers: 3
Stora space per broker: Ultra-high I/O 100 GB
Capacity Threshold Policy: Automatically delete
VPC: Use the VPC and subnet that we`ve provisioned in the first step.
Security Group: Create an SG with the below inbound rules. I`ve created it and named it sg-kafkacluster-tl.
Plaintext Access: Disable
Ciphertext Access: Enable
Security Protocol: SASL_SSL
Private IP Adress: Auto
Public Network Access: Disable
SASL/PLAIN: Enable
Finally, just click to buy and wait for 3–5 minutes for the provisioning.
Prepare the Kafka Client
- For the demo I need a Kafka Client and I used ECS for it. Let`s provision it. Go to ECS page and click Buy ECS and use the below values for the given parameters.
Region: Singapore
Billing Mode: Pay-per-use
AZ: Random
CPU: x86
Specifications: General computing-plus c6.large.2 (2vCPUs, 4GB RAM)
Public Image: Ubuntu 20.04
System Disk: Keep it as is (General Purpose 40 GB)
VPC: Choose the VPC I`ve created
Subnet: Choose the subnet I`ve created
Security Group: Create a new one with the below inbound rules and named it as sg-ecskafkaclient-tl.
EIP: Auto Assign
Billed By: Traffic
Bandwidth Size: 50
ECS name: Give a name
Login Mode: Password and provide a strong password with best practices as the pop-up says.
CBR: Not required
Finally, buy it. It will be available in a few seconds. The next step is preparing the Kafka Client.
2. Kafka Client Installation. Connect to your ECS with a Remote Login or a remote terminal like MobaXterm. I`ve used Moba for the demo. After connecting the instance, use the below commands to update the OS of the ECS and install Java.
apt update -y
apt install openjdk-8-jdkTo view the Java installation path, use the below command
update-alternatives --config javaAnd mine is as follows;
/usr/lib/jvm/java-8-openjdk-amd64Now, we need to add these paths as a variable to the .bashrc profile.
vi ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATHAnd reload it.
source ~/.bashrc3. Download the Kafka client from the below link to your ECS and decompress it.
wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.12-2.8.0.tgz
tar -xzvf kafka_2.12-2.8.0.tgz4. The next step is getting connection info from Kafka cluster. In the kafka cluster connection info section get the broker IP addresses. Then, put them into /etc/hosts file in your ECS Kafka client. To do this;
vi /etc/hosts
192.168.0.226 server01
192.168.0.110 server02
192.168.0.14 server035. Download the certificate from the Kafka Cluster. To do this Cluster>Basic Information>SSL Certificate in the Connection area.
Decompress it in your local machine.
6. Create a directory in your /root directory at the ECS Kafka client.
mkdir -p /root/kafka_certificates7. Upload the certificate files into the directory that you`ve created via the MobaXterm file browser.
8. Add the following commands in both the consumer.properties and producer.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="kafka-master" \
password="XXXXXXXXX"; // (Use your kafka cluster pw)
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
ssl.truststore.location=/root/kafka-certificates/client.jks
ssl.truststore.password=dms@kafka
ssl.endpoint.identification.algorithm=
Produce and Consume Messages
- Now, We can produce messages from the Kafka client. Before producing messages, create a topic in the Kafka cluster from the DMS for Kafka console. I`ve named it topic-01 with the below parameters.
Topic Name: topic-01
Partitions: 3
Replicas: 3
Aging Time: 72
Synchronous Replication: Disable
Synchronous Flushing: Disable
Message Timestamp: Select CreateTime
Max. Message Size (bytes): Retain the default value.
Jump into your Kafka client.
cd ../bin
./kafka-console-producer.sh --broker-list 192.168.0.99:9093,192.168.0.191:9093,192.168.0.62:9093 --topic topic-01 --producer.config ../config/producer.properties
>Hello
>DMS
>Kafka!
ctrl+c
2. Let`s consume these messages.
./kafka-console-consumer.sh --bootstrap-server 192.168.0.99:9093,192.168.0.191:9093,192.168.0.62:9093 --topic topic-01 --group test-consumer-group --from-beginning --consumer.config ../config/consumer.properties
>Hello
>DMS
>Kafka!
ctrl+c
As you can see in the above output our cluster is up and running. It also gets and delivers the producer and the consumer messages.
Conclusion
Kafka is a popular message-oriented middleware that features highly reliable, asynchronous message delivery. It is widely used for transmitting data between different systems in many industries, including enterprise applications, payment, telecommunications, e-commerce, social networking, instant messaging, video, Internet of Things, and Internet of Vehicle. In this demo we`ve simulated the above definition and in a way, we have also provided service decoupling and we have also provided the micro-service paradigm with the Huawei Cloud Managed Kafka service.
