Kafka Cron using wix/greyhound
I think one of the best ways to learn about a tool is to use it by solving an interesting problem. The problem can be simple or complex based on how deep your curiosity is.
For me, the tool was wix/greyhound and the problem I wanted to solve was scheduling tasks using Kafka by providing schedules as Cron strings. As an additional constrain I did not want to use any other external storage other than Kafka.
Dealing with State
One of the first problems we face is persisting the state of the app between restarts while keeping Kafka topic request payload small. We can achieve this by using events to describe the change and building the state in memory.
Interactive with Kafka
As mentioned before we will use wix/greyhound as a mechanism to interact with Kafka, so we need a simple interface to describe the interaction.
There are 2 parts that we care about: recording and subscribing to recordings.
To start building Topic query we first need some information about Kafka configuration. We will have 2 topics cron for CronEvents and action for scheduled actions:
We also need to describe the configuration of Kafka broker and Key that we will use to differentiate our records:
It’s important to note that Key and Value that are stored in Kafka need to be serialized and deserialized. For that wix/greyhound provides a concept calledSerdes (Serializer/Deserializer). Because we will use circe for JSON encoding/decoding we can express that as:
Now we can plug this into wix/greyhound like this:
Our topics will look like:
Expressing CRUD API for CronJob
We will keep the CRUD API simple and express it as:
More information about using this with http4s API layer can be found in provided example project or in my other post https://medium.com/wix-engineering/http4s-service-with-authentication-using-tsec-9c5b97004c19.
In case of our Cron CRUD implementation, we will also add a way to stream CronJobs:
Now the only thing left is to connect cronJob topic with action a topic by running a subscription stream as part of the app:
Summary
Modeling time using streams is an exuisit endeavour that requires patience and commitment, but as I hope you can see it’s quite fun when you can finally see your action stream being printed out in the console.
Try running the example project and review the test cases for more insights. The example project has tasks endpoint that can be reached curl — no-buffer http://$FP_HOST:$FP_PORT/tasks that streams the tasks that are scheduled via API. Don’t forget to run docker-compose in devEnv/kafka-docker before running any tests or the app.
Possible improvements
- Adding an authentication layer to be able to deploy this as a public API
- Providing an
inMemorytopic, so you can run this service without requiring Kafka - If you have any other suggestions for improvements please leave them in the comments or try them out on the example project
References
- Example project: https://github.com/Algiras/Cronula
- https://kafka.apache.org/
- https://circe.github.io/circe/ — encoding/decoding data to/from JSON
- https://github.com/wix/greyhound for interactive with Kafka
- https://github.com/alonsodomin/cron4s for Cron expressions
- https://http4s.org/ for the HTTP API
- https://zio.dev/ and https://typelevel.org/cats/ for effect system
