TL;DR — new JS OpenTelemetry plugin for kafkajs, is available here.
Open Telemetry is a CNCF project, which, among other things, enables the collection of distributed traces.
Aspecto is a developer-centric observability solution and we use OpenTelemetry at the backbone of our product. While implementing it on our backend, we found a few plugins that are missing, especially when dealing with asynchronous communication. One of them was kafkajs. We took this opportunity to give back to the community and develop it ourselves. With this plugin, you can track all Kafka interactions in your collected traces and get a more comprehensive view of your application behavior when using Kafka as a message broker. This article briefly describes a new plugin for the kafkajs package.
The kafkajs plugin captures producer and consumer operations and creates spans according to the semantic conventions for Messaging Systems.
- Each message being produced or consumed is represented by a span with attributes such as
messaging.destination(topic name). - Context is propagated from producers to consumers, which means that when a message is sent to Kafka, the trace will reveal which services consume it and what other cascading operations happen down the pipe.
- Batch operations can aggregate multiple messages into a single batch and receive\process them together. This is handled in the plugin according to the specification.
- The plugin can be extended with hooks, which enable users to run custom logic for adding span attributes depending on the Kafka message.
The above screenshot shows an example of a producer application named kafka-producer, that exposes an Http endpoint (first line), route it in express (second line), and produce two messages to a Kafka topic namedtest, which are then consumed by another application calledkafka-consumer .
As mentioned above, kafkajs was one of the missing plugins we found and as you are reading this we are working to add more plugins. For developers who develop in microservices architecture, Aspecto is a developer-centric observability solution that abstracts the native complexity of microservices. As so, we are very much invested in OpenTelemetry so if you have any questions, feel free to reach out to me or the team.
