Problems We Face for Sending Events to the Secure Kafka in Python
Hello everyone, 🙋♂️ in this article, I will share information about how we use Kafka technology in our tests and the problems we face when using it in Trendyol Order Management System Team.
The article focuses on sending events on the secure and non-secure Kafka in Python language in our tests. I will also give some information about which of our test projects we use Kafka in the team.
First of all, sending an event is a necessary form of the end two end testing in our test orders. We create an order which does come from the Checkout team by simulating an event. In this way, we can independently complete the orders we want, and forward the relevant API calls in the order flow through this order. We write our backend tests which send events to Kafka Topics in three projects.
In this article, I will focus on the Scripts project. Kafka producer helps for sending events to the Kafka Topics. We have been using non-secure Kafka for this job for a long time. For that process, we create the message which we want to send to the Kafka topic with the KafkaProducer. Let’s look at how we send events to the non-secure Kafka in our projects.
After a while, we needed secure Kafka. As non-secure clusters only work on one environment so if they have any possible problems or interruptions, they affect the entire Kafka cluster. As a result of the problem, we have written another method for secure Kafka which shows below.
The method needs a file of Certificate.pem file. Kafka producer uses a certificate file to send events to secure Kafka. The sample template of the file which shows below.
-----BEGIN CERTIFICATE-----
...
xxx
...
-----END CERTIFICATE-----We need to use this file for ssl_cafile. The main problem we face here is nested classes’ file paths. The keyword argument of ssl_cafile only wants the file path, so you cannot send this certificate content there directly.
For instance, If you want to add the .pem file to the root as in the example project we use and call it from the mp package, you must give the path as “../../certificate. pem”. If you are going to call it from the created order package, it should be “../certificate.pem”. Otherwise, ssl_cafile will not see the file path.
Additionally, security protocol and sasl mechanism information are mandatory fields taken from the broker configuration section. The method gives errors if we do not add them.
Another crucial piece of information is related to the API version. Unless we explicitly specify api_version to be (0, 10, 1) the client library’s attempt to discover the version will cause a NoBrokersAvailable error.
Also, when we send an event to the secure Kafka, KafkaProducer needs sasl_plain_username and sasl_plain_password information for connecting the broker.
We write load test projects in the same way as the second method. However, the load test needs other libraries for the project. I will give information about our load test in a different article.
What is more, creating a retry mechanism, which a crucial part of test scenarios. When we send the event to secure Kafka, it has errors sometimes, so we add it because it contributes to retrying and retesting if sending a message to Kafka fails for any reason.
Also, adding a dependency of tenacity to requirements.txt does necessary for the code block. So the program will continue to run the script until it reaches the number of retries given by the developer.
Furthermore, we run tests on stage which is an environment before we push the codes to the live. Sometimes we have errors when we get the response in it due to dependent teams. The retry mechanism mentioned below allows getting the result of a request with the correct status code, so it does healthier to produce successful scenarios, especially in getting functions.
First of all, we need to add a dependency of busy pie to requirements.txt for the code block. Then, we can write the code block which shows below.
Since we have given one hundred twenty in this example, the program will continue to try the status code of the process every second until it reaches that second. Using this snippet, we can purify our code from the overused thread sleep feature.
Thanks for the readings. 🙋♂️
