Event Streaming

Introduction: Microservice architecture has become a popular way to build scalable and robust applications. In this tutorial, we will explore how to build a microservice architecture using .NET 7 and Kafka. We will also cover topics such as API Gateway, Orchestrator, Audit Trail, and Data Consistency. …

This article discusses how to implement an Event Stream Registry. If your first question is “What is an Event Stream Registry?”, you can read the first part of this series of articles here. Considerations for an Event Stream Registry At its core, the Event Stream Registry is a small dataset that can be queried and updated…

What data to publish in an event? How to consider consumers’ needs? How many types of events are worth including in one stream? How does relational data affect event streams? — these essential questions and more are addressed in this article, along with primary considerations and best practices, relevant to any event-driven architecture — Event streams are growingly adopted because of their capability to process large volumes of data in real-time, while allowing systems to interact continuously without coupling. However, designing effective event streams can be challenging, especially when considering the diverse needs of consumers and the variances of data that can be transmitted…

Apache Kafka is an open-source distributed streaming platform that has become a popular choice for building real-time data pipelines and streaming applications. However, getting started with Kafka can be challenging, especially if you are new to its unique terminology. In this article, we will demystify some key Kafka terms and…

Apache Kafka popularized the concept of stream-processing of messages or events, and it’s now widely used to implement event-driven architectures to support the communication between bounded contexts in a micro-services topology. But what if you do not want to use Kafka, maybe because you already use Amazon ElastiCache and Redis…

What data to publish in an event? How to consider consumers' needs? How many types of events are worth including in one stream? How does relational data affect event streams? — these essential questions and more are addressed in this article, along with primary considerations and best practices, relevant to any event-driven architecture — Event streams are becoming more commonly used for their ability to handle large volumes of data in real-time. However, designing effective event streams can be challenging, especially when considering the diverse needs of consumers and the variances of data that can be transmitted. In this article, we will focus on…

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. …

Real-time is an excellent marketing term describing how businesses can add value by processing data as quickly as possible. This post explores what real-time means and how apache kafka accomplishes the mission of providing real-time data processing. Let’s first understand What real-time is? In the computing world, Real-time programs must guarantee a response within specified time…

In this practical article we will work with three of AWS services which are SQS, Lambda and DynamoDB. These three services are used frequently between AWS users. The purpose of this article is to learn how to use AWS native solutions to implement an event or message based behavior. Here each…