“eCupcakesFactoryOnContainers” — A Reference app for event driven containerized micro services architecture using .Netcore and Kafka
INTRODUCTION:
This is a continuation of my previous blog posts on “Building Real-time Streaming Apps using .Net core and Kafka”.
Libraries like librdkafka and confluent-kafka-dotnet have opened up Kafka to .NET world, however, there don’t seem to be many examples available. As part of our digital transformation training suite, we started implementing this sample .Net Core reference application — “eCupcakesFactoryOnContainers” based on a simplified “Microservices” architecture using docker containers. Core intent of this reference application is to provide reference implementation for .Net developers to easily get started with containerized microservices using .Net Core and Kafka.
The Domain:
What is eCupcakesFactory ??
It’s a fictional cupcake factory use case mentioned in my colleague William Scott’s blog post on event streaming, please read his blogpost before you start looking into this sample codebase.
To simplify things, our focus will be only on just ordering “Cupcakes” with very few flavors, modifying the order and cancelling the order scenarios are out of scope for this blog. When users visit our eCupcakeFactory application, they could order one or more cupcakes, and the kitchen staff would immediately get notified about these new orders. Below are the main steps involved in preparing user order:
- Mixing: Involves preparing the cupcake batter with right ingredients.
- Baking: once the mixed batter is available, we pour it in evenly into liners and bake cupcakes in a preheated oven.
- Decorating: once cupcakes are completely cooled, we can frost and decorate them as simply or creatively.
- Packaging/Boxing: packaging the right size box and packaging the order.
- Shipping: delivering the order.
Architecture:
It’s an event driven architecture implementation with multiple micro-services(some are background services) each having its own responsibility and configured to respond on certain events like “NewOrderRecieved”,”OrderBaked”,”OrderMixed” etc.
- Microservices implemented with best coding patterns/practices like CQRS(using MediatR Framework),multi route WebApi controllers etc.
- Used Kafka as message broker for communication b/w these micro services, also leveraged Kafka as persistent datastore.
- Used confluent’s “kafka connect” sink and CDC plugins to import/export data to/from “MongoDB”.
- React-Redux architecture for client apps, used http as the communication protocol between the client react apps and the microservices.
- Used SignalR hubs for notifying react client applications about the data updates.
Technology Stack:
- .Net Core
- Confluent Kafka Cloud
- Confluent Kafka Connect
- SingalR
- React-Redux
- Docker
Prerequisites:
- VSCODE or some .net code editor
- .NET Core 2.2
- Docker&Docker-Compose
- Kafka Installation and Topics setup
- Kafkacat command line tool
Git Repository:
Before we go into detail…
Why Kafka for event streaming?
We could use other pub-sub/messaging systems like RabbitMQ , but we preferred Kafka because of its powerful core features like durability, high throughput, high availability , mutli-consumer model etc.
Why .NET Core?
we have used .NET Core as it comes with original powerful .net framework and the cross-platform ability to run anywhere like Linux or Windows containers.
Kafka Confluent’s team is actively working on Kafka .NET Core client, releasing every capability in librdkafka(core c library) into this client. Current version v1.0.0 looks very stable with well written patterns.
Why SingnalR?
SignalR will give you a much higher level API than raw sockets which is its big advantage, allowing you to do things like “RPC” from server to clients in a broadcast (or targeted) fashion.
**we couldn’t find any browser based KAFKA client that can stream Kafka messages directly into our react application using “Consumer Group Semantics”. so we have implemented a .Net core SignalR based Proxy similar to microsoft’s nodejs version.**
Setting up and Deployment:
In Local Machine using Docker:
To make things simple, we have created a docker compose file with all required apps&services, so running “docker-compose up”command at root directory is enough to start the whole system ( it builds all required docker images and containers in localbox).
on successful run, you can access your UI/Services at the below endpoints:
Note: Above setup assumes that you have a running Kafka version in local machine on standard ports.
User Scenarios & Flow of events:
How do we make a cup cake?
- Customer will place an order using our web client (React).
- Mixer(a cook who is also using our web client interface) in the kitchen gets notified about the new order, he/she prepares the mix and updates the order status as “mixed”.
- As the order is mixed, Its baker that gets notified about this , he/she pours batter, bakes in oven and updates the order status as “baked”.
- Now, decorator gets notified about the baked order , he/she decorates them creatively and updates the order status as “decorated”.
- Now, packaging team gets notified about the baked order , they pack the order in the right size box and updates the order status as “packaged”.
Apps/Services:
eCupcakesFactory Web Client — (React-Redux ui):
A react-redux based web application with customers and cooks as its users.
- provides interface to place new orders.
- provides interface for kitchen staff(mixer/baker/decorator/packaging-team) to view newly arrived orders and update the current order status after they complete their process.
API Layer (REST):
Provides the below APIs to place new order and update the current orders.
- api/v1/order → (to place a new order)
- api/v1/order/mix → (to update the current order as mixed)
- api/v1/order/bake → (to update the current order as baked)
- api/v1/order/decorate →(to update the current order as decorated)
- api/v1/order/box →(to update the current order as packaged)
Background Services(with SignalR server):
.NetCore hosted services continuously monitors KAFKA topics and push notifications to all connected clients(i.e react-ui here) about the newly received orders/updates to existing orders.
- Each of these hosted services has SignalR server with a defined endpoint and actions, UI subscribes to these actions to get notified about the updates to the order.
Example: When a new order arrives in our system, an event gets posted to “orderrequests” topic, and our mixer background service which is continuously monitoring this topic gets this message and it uses SignalR server to inform subscribed/connected clients about this.
Note: why do we need SignalR here? Though Kafka is pub/sub system, there is no direct browser client available. So we need something like SignalR to push these notifications to UI clients.
our react ui subscribes to the following events of our SignalR endpoints:
- “InformNewOrderToMix”
- “InformNewOrderToBake”
- “InformNewOrderToDecorate”
- “InformNewOrderToPackage”
KAFKA Topics:
- “orderrequests” → all received orders
- “readytobake” → orders that were mixed and ready to be baked.
- “readytodecorate” → orders that were baked and ready to be decorated.
- “readytopackage” → orders ready to be packed
Conclusion:
The “eCupcakesFactoryOnContainers” is our open-source reference implementation with some patterns (Connect,Streaming topics,CQRS) that we’ve used while implementing .NET Core micro-services using Kafka . You can refer our github repository README files for detailed explanation of each Api/Service/Ui components.
