Getting Started with Kafka and .Net core 2.2

Are you looking for a simple way to step into the Event-driven programming leveraging the latest tools and the familiarity of Dotnet core? Look no more! Here is a (Very) simple way to build an event-driven system in Dotnet core using Kafka as a message broker.

Before starting:

Make sure you have the following installed on your machine:

• .net core SDK 2.2 (download)
• Kafka 2.11 (download)

Step 0 - Run the Kafka server

Before starting this step, I want to mention I’m using Linux subsystem for Windows with the Ubuntu Shell. Also I’m using Zookeeper for Kafka Server configuration, as suggested on the Apache Kafka quickstart guide.

Follow the instruction on the Apache Kafka quickstart guide on where to get the latest version and how to open it. Then, move to the directory where you saved Kafka and start the server as follows:

a) Start the Zookeeper instance:

The zookeeper scripts and configuration files comes with the Kafka tarball

b) Start the Kafka instance:

Inside the properties file you can find the configuration and connections

Step 1 - Producer console application

a) Let’s create the Dotnet console application:

A store selling bananas

b) Let’s add the Nuget Kafka package to the application:

Confluent.Kafka is a wrapper around C library librdkafka

c) Let’s start writing some basic logic for the producer app. We have to, in this order, connect to the Kafka server, register as a producer and produce a message (🍌):

• Let’s add some namespaces:

Program.cs

• Let’s add some logic to connect, register as a producer and send a 🍌:

producer console application

Step 2 - Consumer console application

a) As of step 1 we can repeat a) and b) by creating a Dotnet console application and adding the Nuget Kafka package:

consumer console application

b)Now let’s add write the logic to connect to the Kafka server, register as a consumer and poll messages:

consumer console application

Step 3- Test the applications

First we have to build our console applications and then we can run both of them and we should see the messages from the BananaShop in the BananaConsumer CLI.

a) Build the BananaShop console application:

Build

b) Run the BananaShop console application:

Run

Do the same for the BananaEater application.

Aaand, it’s done!

Apache Kafka

This is just a brief introduction and a step-by-step guide to get started with Kafka and .net core. In the future I’ll follow up with more in-depth articles about the advantages of Kafka in stream and messaging processing and event-driven architectures.

Github repository: repo

Apache kafka: link

Twitter: @0xtheo

Linkedin: profile

🍌🍌🍌🍌🍌

Edits:

19/06/2019 — Updated Confluent Kafka Nuget package to 1.0.1.1 and changed consumer code to use ProduceAsync