Stream Processing

by Damian Święcki @SwieckiDamian With the recent Confluent announcement of the Stream Designer, the company entered a vast field of low-code visual tools aimed at professionals without programming experience. It may feel like a step forward to the democratisation of data streaming application development and toward the wide adoption of…

1. Log shipping Log shipping in Kafka refers to the process of copying data (or “logs”) from one Kafka cluster to another. This can be used for a variety of purposes, such as replicating data for backup or disaster recovery, or to distribute data to multiple regions for improved performance and availability. Log…

— coauthored with Antonio and Prema Kuppuswamy At Intuit, we have been using Apache Beam with FlinkRunner (or Beam/Flink for short) as a real-time data processing platform for our data pipelines. A typical pipeline consumes data from one or more Kafka topics, runs several transformations, and outputs to another Kafka…

Stateful stream processing is a type of computing that involves processing a continuous stream of data in real-time, while also maintaining a current state or context based on the data that has been processed so far. …

This is the first post of the series that shows building operational use cases with Apache Kafka and Apache Flink. — Apache Kafka is a distributed streaming data platform. It is designed for high throughput, low-latency streaming workloads, where scalable real-time data ingestion and fault-tolerant storage are critical. Apache Flink is an open-source, unified stream-processing and batch-processing framework capable of executing arbitrary dataflow programs on data streams.

What is Stream Processing ? In reality, a lot of data is unbounded because it gradually arrives over time. Hence, there is a need to process these unbounded datasets, as and when they arrive. This is the idea behind stream processing. In general, a “stream” refers to data that is…

Data is an essential asset for modern businesses as it helps them to monitor all aspects of the business. Every second we are processing, analysing and transforming a large amount of data. So the need for handling the dynamically generating data is important. As the number, variety, and velocity of…

— by Manisha Malla It is a capital mistake to theorize before one has data - Sherlock Holmes in ‘A Study in Scarlet’ by Arthur Conan Doyle‍ When one of the most famous fictional detectives in history who is known for his analysis & deduction prowess makes this statement, it…

This blog post contain steps and instructions on testing RDBMS store with SI analytics. The common problem while testing SI is complexity of calling stream without tooling UI. Here, we used siddhi triggers on the siddhi script it self to trigger RDBMS event process. Download SI product pack from WSO2…