Making Sense of Unbounded Data
A reference architecture for a real-time event processing system
Unbounded data refers to continuous, never-ending data streams with no beginning or end. They are made available over time. Anyone who wishes to act upon them can do without downloading them first.
As Martin Kleppmann stated in his famous book, unbounded data will never “complete” in any meaningful way.
In reality, a lot of data is unbounded because it arrives gradually over time: your users produced data yesterday and today, and they will continue to produce more data tomorrow. Unless you go out of business, this process never ends, and so the dataset is never “complete” in any meaningful way.
— Martin Kleppmann, Designing Data-Intensive Applications
Processing unbounded data requires an entirely different approach than its counterpart, batch processing. This article summarises the value of unbounded data and how you can build systems to harness the power of real-time data.
Bounded and unbounded data
Unlike its “bounded” counterpart — batch data, unbounded data has no boundaries defined in terms of time. The data may have been arriving from the past, continuing today, and expected to arrive in the future.
Events, messages, and streams
Streaming data, real-time data, and event streams are some similar terms for unbounded data. A stream of events consists of a sequence of immutable records that carry information about state changes that happened in a system. These records are called events, rather than messages as they merely transmit what just happened.
Conversely, messages are a way of transferring the control or telling someone what to do next. I’ve written about this difference sometime back.
A continuous flow of related events makes an event stream. Each event in a stream has a timestamp. Event sources continuously generate streams of events and transmit them over a network. Hardware sensors, servers, mobile devices, applications, web browsers, and Microservices are examples of event sources.
Benefits of real-time data and practical examples
Today’s organisations are abandoning batch processing systems and moving towards real-time processing systems. Their main goal is to gain business insights on time and act upon them without delays.
Batch processing is better in terms of providing the most accurate analysis of data. But it takes a long time to deliver and requires careful planning and complex architecture. In a world full of competition, businesses tend to trade accuracy for quick but reliable information. Real-time processing systems starting to shine there.
Following are some practical examples that demonstrate the power of real-time event processing.
- Real-time customer 360 view implementations
- Recommendation engines
- Fraud/anomaly detection
- Predictive maintenance using IoT
- Streaming ETL systems
Making sense of streaming data
Moving data alone doesn’t make sense to any organization. It has to be ingested and processed to gain more value out of it.
The best analogy I can provide is a small stream ( a real stream where water flows). If you consider the stream in isolation, it is not that useful except for supporting the life forms that depend on it. But when multiple streams are combined, they carry enough power. When stored in a reservoir, they can even generate electricity to power an entire village.
Likewise, event streams flowing into an organization have to undergo several stages before they become useful. Usually, an architecture that processes streams employ two layers to facilitate that; the ingestion layer and the processing layer.
Event ingestion and processing is challenging
The ingestion layer needs to support fast, inexpensive reads and writes of large streams of data. The ordering of incoming data and strong consistency is a concern as well. The processing layer is responsible for consuming data from the ingestion layer, running computations on that data, and then notifying the storage layer to delete data no longer needed. Also, the processing should be done on time, allowing organizations to act on data quickly.
An organization planning to build a real-time data processing system should plan for scalability, data durability, and fault tolerance in both the ingestion and processing layers.
A reference architecture for stream processing
A real-time processing architecture should have the following logical components to address the challenges mentioned above.
1. Event Sources
Data sources can be almost any source of data: system or weblog data, social network data, financial trading information, geospatial data, mobile app data, or telemetry from connected IoT devices. They can be structured or unstructured in format.
2. Ingestion system
The ingestion system captures and stores real-time messages until they are processed. A distributed file system can be a good fit here. But a messaging system would do a better job as it acts as a buffer for the messages and supports reliable delivery of messages to consumers.
The stored messages can be processed by many consumers multiple times. Hence, the ingestion system must support the message replay and non-destructive message consumption. That goes beyond the capabilities of traditional messaging systems and opens the door for a distributed log.
Distributed and append-only logs provide the foundation for ingestion systems architecture. Such a system appends received messages into a distributed log while preserving their order of arrival. This log is partitioned across multiple machines to support scalable message consumption and fault-tolerance.
The architecture and behavior of an ingestion system go beyond the scope of this post. Hence, I’ll save it for a separate article.
Technology Choices
Apache Kafka, AWS Kinesis Streams, Azure Event Hubs and IoT Hubs.
3. Stream processing system
After ingestion, the messages go through one or more stream processors that can route the data or perform analytics and other processing.
The stream processor can run perpetual queries against an unbounded stream of data. These queries consume streams of data from the ingestion system, analyze them in real-time to detect anomalies, recognize patterns over rolling time windows, or trigger alerts when a specific condition occurs in the stream.
After processing, the stream processor writes the result into event sinks such as storage, databases, or directly to real-time dashboards.
Technology choices
Apache Flink, Apache Storm, Apache Spark Streaming, Apache Samza, Kafka Streams, Siddhi, AWS Kinesis Analytics, Azure Stream Analytics.
4. Cold storage
In some cases, the stream processor writes the ingested message into cold storage for archiving or batch analysis. This data serves as the source of truth for the organization and doesn’t expect frequent reads.
Later, batch analytic systems can process them to produce new reports and views required by the organization.
Cold storage systems can be in the form of object storage or much sophisticated Data Lakes.
Technology choices
Amazon S3, Azure Blob Containers, Azure Data Lake Store, HDFS
5. Analytical datastore
Processed real-time data is transformed into a structured format and stored in a relational database to enable queried by analytical tools. This data store is often called the serving layer and feeds multiple applications that require analytics.
The serving layer can be a relational database, a NoSQL database, or distributed over a file system. Data warehouses are a common choice among many organizations to be used as such a store.
However, the serving layer requires strong support for random reads with low latency. Sometimes, the batch processing systems use this as their final destination. Hence, the serving layer should also support the random writes as the loading of processed data can introduce delays.
Technology choices
Amazon Redshift, Azure Synapse Analytics, Apache Hive, Apache Spark, Apache HBase
6. Monitoring and notification
The stream processor can trigger alerts on different systems when it detects anomalies and some conditions. Publishing an event into a pub/sub topic, triggering an event-driven workflow, or invoking a serverless function are few examples.
The ultimate goal is to notify the relevant party that has the authority to act upon the alert.
Technology choices
Pub/sub messaging systems, AWS Lambda
7. Reporting and visualization
Reporting applications and dashboards use processed data in the analytical store for historical reporting and visualizations. Additionally, the stream processor can directly update real-time reports and dashboards if a low-latency answer is needed.
Technology choices
Microsoft Power BI, Amazon Quicksight, Kibana, Reactive web, and mobile applications
8. Machine learning
Real-time data can be passed through machine learning systems to train and score ML models in real-time.
Conclusion
Real-time processing can be beneficial to organizations in terms of getting actionable insights on time. A real-time processing architecture should cater to high volumes of streaming data, scale-out processing, and fault-tolerance.
But real-time processing can fall short when it comes to accuracy. Often, real-time systems trade accuracy for low latency. For a better level of accuracy, both real-time and batch processing need to be combined. Architectures such as Lambda and Kappa provide guidelines for that.
I will continue this article series to discuss each logical component in deep. Stay tuned.
