Kafka Search
Every article I wrote mentioned that Kafka is everywhere in Trendyol. Trendyol has many tech stacks. Every backend team has used many services but the most important one is Kafka.
Every time, Searching has been an issue. Nowadays, there are some solutions to searching in Kafka. First, let's look into them.
Every Event transfer to ElasticSearch:
- You can transfer events via the Kafka connector.
- It looks like the better option at first view but, you have to maintain a new big service. Also, you allocate some resources of Cluster and you have to make sure every Event upserted into ElasticSearch
- When you can’t find an event in Elastic. You will ask this question yourself in doubt, "what if it wasn’t delivered?"
Offset Explorer:
- You can install your local machine and use it if you don’t have any access issues to the production environment. Also, it’s too slow.
- In my experience, it takes about 4 minutes to get search results from it, and freezes at every step.
Kowl or something:
- You can use it for searching but, when you start searching for a Topic with huge messages, they cannot respond to users many times.
- In my experience, it takes under a minute to get search results from it. During the get result, they crashed 1 time.
What is the problem?
If the topic has a high message count or more than one user uses the same time, the apps I mentioned above or method run slowly. Sometimes they can't respond to UI and crash because searching for any word in the Topic which has 1 billion messages is a long-term process and it's CPU bound process. It consumes every CPU resource.
Another problem is where is the Topic? Trendyol uses more than one cluster. When we search the word in the Topic, we have to know which Cluster has the Topic otherwise, we can't search. When you use the apps I mentioned above have to search Cluster by Cluster.
What do we try to achieve here?
I developed a macro service for Event searching in Kafka. You can search any word in key or value at the specified topic without any freeze, crash, or slowness.
What is the difference here?
Here, we separate the searching process and the serving process. Searching is a really massive consuming CPU resources. Two applications (API/searching consumer) are working with Kafka and Redis.
When we request to API to start searching, we pass the process to the searching consumer. We don't use resources of API during the search. Therefore, the search process doesn't affect the API.
How it works:
I sent this request http://localhost:8080/search?topicName=product.spda.enricher.invalidateListing.1&value=843078897 and refreshed the page to get results in seconds.
Api and consumer share search data via Redis. When the consumer starts searching for the value in the Topic and finds data then saves the data to the Redis. Api gets data from Redis and presents it to the user.
Design of API
First, let's talk about API; There are 2 scenarios of searching.
One is "Start searching"
API provides that if the first request of searching, send an event to Kafka to start searching in consumer.
Another is "Getting the result"
API provides that if the search started before, then get the result from Redis and return the user.
API UML Sequence Diagram:
Design Of Consumer
Second, let’s talk about the consumer; There is a scenario of searching.
This is “Start searching”
Start searching in the Topic that the user specified as a parallel in every Partition. If any event contains the keyword that the user specified, then insert it into Redis.
Consumer UML Sequence Diagram:
ABOUT PERFORMANCE
I did a benchmark with 1 billion events. If you wonder about the results, you can reach this article here.
"Talk is cheap. Show me the code." - Linus Torvalds
API link: https://github.com/ferditatlisu/kafka-search-api-python
Consumer link: https://github.com/ferditatlisu/kafka-search-consumer-go
Happy searching in Kafka. Thank you for reading until here. See you next in my article.
If you’re interested in joining our team, you can apply for the role of backend developer or any of our current open positions.
