KaufmannEx — Elixir Microservices with Kafka and Avro
KaufmannEx is a library for building highly parallel, efficient, Kafka backed services in Elixir. This article describes KaufmannEx, the motivations for its design, and details of basic service implementation with KaufmannEx.
I’m Grant McLendon, principal back-end developer at 7Mind. 7Mind makes an excellent app for guided meditation. I’ve spent the past few years working with Rails monoliths and Ruby microservices.
At 7Mind, we’re in the process of migrating from a Ruby on Rails monolith to a microservice oriented architecture. Rails has been fantastic for rapid iteration and expanding business needs. As we continue to grow, our goals are scalable, cost-effective, fault tolerant, and asynchronous services. In the future, we want to support heterogeneous services in multiple languages.
The Tools
After evaluating multiple languages and messaging systems, we chose to use Elixir as our primary language, and Kafka as a messaging bus.
Elixir is a fantastic language (we’ll probably write another article gushing about it) well suited to building microservices. Elixir provides outstanding performance, stability, and BEAM + OTP are a foundation we can build services on with confidence. We could build our services solely within a single Elixir and OTP application, however we want to support development in other languages, and potential future changes in architecture.
Apache Kafka is many things. We are using Kafka as a distributed, scalable, high-throughput messaging bus and broker. Language support for Kafka is excellent and allows us a lot of freedom in our future development. Kafka also provides lots of functionality we don’t need now, but have an eye on for the future.
Message Schemas
One trouble spot we’ve encountered in micro-services is guaranteeing standard message schemas across different services. It’s reasonably common for a developer to make a small change to a message that unexpectedly breaks other services.
To avoid this issue outright, we’ve chosen to use Apache Avro. Avro is a binary data format coupled with built-in schema validation. Avro allows us to encode messages with a given schema and guarantee any application that receives an encoded message will decode it with the correct schema.
We also want to guarantee all applications have access to the latest correct schemas. We found the Confluent Schema Registry perfectly suited to this task. Schema Registry provides an HTTP interface for retrieving and managing schemas, as well as validating schemas for forwards or backwards compatibly.
In a KaufmannEx project message schemas are defined either in app/priv/schemas or another directory configured in the Application environment. KaufmannEx provides a schema migration script to populate the schema registry in production deployments. I encourage you to review this script before use. It may not suit all use cases and has an expectation of a metadata schema you may not wish to use.
For more information about Avro Schemas, check out the Avro Spec Documentation.
KaufmannEx
Implementing one Kafka-backed microservice is pretty straightforward, implementing 10 involves a painful amount of code duplication and undifferentiated busy work. Furthermore, consuming messages from Kafka can be akin to sipping from a firehose. We wanted to restrict message consumption to available consumer capacity, using Kafka Consumer Groups to distribute messages across multiple consumers according to capacity. This lead rather directly to implementing a library to handle the boring parts of writing a new service in a consistent, parallel manner. To control consumption rates, we have included a back-pressure mechanism inspired by Mathew Gardner’s Elixir with Kafka talk.
KaufmannEx is our library for implementing simple, parallel, Kafka-backed services. We’ve elected to open source it, as it appears there are a lot of other people trying to solve similar problems. The name, Kaufmann, comes from Kafka’s Der Kaufmann, the seventh story in Kafka’s Meditations. The name was chosen haphazardly, don’t read too much into it.
KaufmannEx is built around two core components: Event Handlers and Event Publishers.
Event Handlers
Event Handlers handle events consumed from a Kafka topic. KaufmannEx starts a KafkaEx Consumer Group, and dispatches consumed events in parallel to an EventHandler module.
An Event Handler is expected to define a given_event/1 method which will be called with incoming messages.
So for instance, if we were handling an event to create a new user, we might write this event handler:
Elixir's pattern matching makes handling additional events really simple, just define another permutation of given_event/1. It's also recommended to have a catchall given_event/1 to handle unexpected messages (unless you want to crash on unexpected messages, which is a legitimate strategy)
Event Publishers
Publishing events is even simpler. Publishers are even strictly required, an event handler or other module could easily handle formatting events to be published. All that is truly required to publish a message is to call KaufmannEx.Publisher.publish/4. KaufmannEx will attempt to encode the provided message in the named event schema.
Sample Application
We’ve provided a sample application in the project repository which provides a good starting point for writing an application with KaufmanEx.
Testing
KaufmannEx also provides a testing module KaufmannEx.TestSupport.MockBus to test Event Consumers and producers. MockBus can trigger events and test for their effects.
Consider this example from the Sample Project, where a simple test specifies one event and then observes the resulting event.
KaufmannEx’s internal parallelism
KaufmannEx tries to be very parallel.
We start by using the KafkaEx GenConsumer to consume from the configured topic(s) and partitions to parallel instances of a service. GenConsumer is clever about allocating topics across multiple instances of a service and can be relied on to balance topic partitions to members of a consumer group.
Internally KaufmannEx uses GenStage with a back-pressure mechanism to guarantee messages are only consumed from the bus as there is capacity to process said event. Each message produced from this GenStage is sent to a GenStage ConsumerSupervisor which de-serializes each event and calls EventHandler.given_event/1 in parallel.
The parallelism can be configured in the application config.exs as :kaufmann_ex, :event_handler_demand, with a default of 50
If that explanation wasn’t quite confusing enough, this diagram shows how KaufmannEx gets a message from Kafka to an EventHandler.
Caveats
KaufmannEx was built for internal use at 7Mind and then generalized, causing some behaviors and patterns to occur as a result. The default message naming scheme, for instance, is what we use internally and may not generalize terribly well to all use cases. We’re working on improving this, any and all contributions are welcome.
This is also our first significant Elixir project. We’ve almost certainly overlooked some aspects of Elixir or Kafka. This project has a lot of room for improvement. Please check out our Github, and open an issue or pull request.
