Modernizing a Messaging Strategy
Author: George Trujillo
Moving from simple yet different messaging approaches, such as pub/sub and message queuing, to a more dynamic event stream where data is transformed into decisions will accelerate your business outcomes. Learn more about how these approaches compare and gain insights into modernizing your messaging system to event streaming.
Organizations are winning customers and markets one real-time decision and customer experience at a time. Winning in real-time is moving event stream processing to the center of data management strategy to accelerate business outcomes. According to Gartner, “it’s Event Stream Processing technology which gives organizations the ability to integrate or analyze data as it arrives, enabling situational awareness and near-real-time responses to new challenges and opportunities.” Every data strategy needs to be addressing the complexity at data integration points for data in motion.
In Gartner’s 2022 Hype Cycle for Data Management report, Gartner analysts W. Roy Schulte and Pieter den Hamer rate “Event Stream Processing as transformational for data management in its ability to impact business across industries.” This has data leaders and enterprise architects are looking for ways to streamline data ecosystems and reduce the complexity of siloed message queues and event-streaming technology stacks that source data at integration points. In this article, we will highlight the differences between message queues and event streams, and show why modern architectures require an event streaming approach. We will also address the biggest barrier to organizations maximizing business outcomes from the data in their message queues and event streams.
Data (messages) that flow from a source to a destination in real-time or near real-time are usually part of a message queue or streaming architecture. A messaging system is responsible for transferring data from a source(s) to a destination(s). The source and destinations can range from applications, devices, social media, databases, memory caches, etc.
The strategic challenge facing data leadership is addressing the technical debt and impact of technologies that have laid the foundation for messaging but were not designed for the scale, speed, and business requirements of today. Let’s start with a quick review of the different types of messaging such as message queues, event streaming, and publish-subscribe. We’ll then look at how business demands are driving the next generation of messaging and data integration.
Message queueing is a first-in, first-out asynchronous method to send a message (data) from a source to a destination. Message queues have been a popular technique for delivering messages between different services or applications. They’re useful for decoupling the producer (message sender) from the consumer (message processor) without having to worry about timing or availability. Message queues are a well-known pattern for asynchronous communication in distributed systems.
Message queues have some notable limitations, that have organizations re-evaluating their message queueing strategy.
- First, they’re one-to-one in nature, meaning they only transmit data from one sender to one recipient. They aren’t the most conducive to having a connected microservices architecture and don’t support real-time messaging.
- Second, message queues are operationally complex, as each queue needs to be created, configured, and monitored. And while there are tools available to help manage this, they don’t fully remove this complexity.
- Finally, message queues don’t handle scaling very well — especially when there are unexpected surge loads.
A major benefit of message queues is a serialized/ordered delivery mechanism. However, the drawbacks of performance and scalability cannot be ignored.
Continuous flows of data that have to be processed in real time or near real-time require additional characteristics and capabilities. Event streaming has become a popular method for continuously processing data in real time and handling each data point independently. Event streams operate by collecting data across varying sources, processing this data in the prevailing context, and acting on important business triggers — all while iterating and learning continuously.
Event streams often react immediately to new information as it arrives from other parts of a system. As such, they are often used for real-time analytics or viewing changes in real time. For example, consider event changes from intelligent IoT devices, new posts on social media sites or activity on your website that is reflected on your dashboard immediately.
An event stream can transmit any kind of data — not just events. However, the most common use cases are logging events, monitoring metrics, transmitting application state changes across distributed systems, processing data from intelligent devices as well as supporting real-time customer experiences with mobile applications, social media and augmented reality.
A Winning Messaging Strategy
The key to developing a winning messaging strategy, is to NOT ask should my next project or initiative use message queues, events streams or pub-sub. Organizations win with data by addressing two key challenges. The first is the ability to innovate with correlated data from multiple sources at data integration points. Change management is critical for new insights and evolving business models. These data integration points often have data from message queues, event streams, pub-sub and database change data capture tools. Each with a different technology. Each development team from these technologies speaks a different technology language with different capabilities and often a different mindset. The second challenge is addressing legacy messaging systems that have increasing failure rates and performance issues as they were not designed or architected to support today’s data volume and velocity.
The first step in accelerating business outcomes is to reduce the number of technologies that create complexity at data integration points. Here is where the innovation of open source provides a streamlined solution. Apache Pulsar is a framework that supports message queues, pub-sub, event streaming and an event-driven architecture. It’s an open-source toolkit for building real-time streaming applications, supporting server-to-server messaging, and queuing built on pub-sub messaging. Apache Cassandra has a change data capture solution that feeds Apache Pulsar which reduces another technology for change data management and downstream processing of data. Because Apache Pulsar exchanges data in real time, Pulsar enables developers to build systems that scale easily without the burden of unnecessary complexities, costs, and time.
Architecting a streamlined real-time ecosystem that reduces the complexity of data integration and downstream processing is a critical success component of a modern data management strategy.
The limitations in scale, performance and agility of message queues are being addressed more and more in data modernization strategies. However, that is only a first step in an updated messaging strategy. The biggest barrier to transformational impact, highlighted by Gartner, is focusing on event stream processing. Part of this transformation includes a messaging strategy that aligns message queues, pub-sub, event streaming and CDC into a real-time data ecosystem that drives the next generation of insights from messaging systems.