How to use a Stream Processor as a Notification Manager?
Streaming applications process events, identify relevant information or anomalies in them and create alerts. Although most of the stream processing systems support filtering, aggregation, joining, pattern matching, and alert generation, they don’t concentrate on how those alerts are managed. Managing alerts is not just a matter of detecting conditions and responding to them. The systems need to identify the relevant authorities to alert, notify them based on user’s preferred frequency and message formats, provide guaranteed delivery for mission-critical alerts, retry if the end user is not reachable, and finally escalate to other stakeholders when the allocated person has not attended to the alert on time. In most scenarios, the preference of the notification receiver is the key. The system should have the capability to let the end users manage the alert condition, when, and how they should get notified. Further, the system should also allow the users to change the configurations using alert templates and forms easily.
As a result, some organizations rely on an alternative system such as notification management systems for alerts. Stream processors can support this use case and often do a better job. In this post, let’s look at how we can use WSO2 Stream Processor to process streams while providing autonomy to the users for managing their notifications and alerts.
WSO2 Stream Processor (a.k.a. WSO2 SP) is an open source, cloud-native, and lightweight stream processing platform that understands streaming SQL queries in order to capture, analyze, process and act on events in real time. WSO2 SP facilitates real-time streaming analytics and streaming data integration. It has many tools and components to support different kinds of users from developers to business users to serve their requirements. For example, it includes built-in developer IDE to build stream processing applications, a status dashboard to provide observability, a business user portal to visualize stream processing results and aggregations, and business rules manager to provide domain-specific stream processing configuration management. These components and tools are grouped together under different profiles such as Worker, Manager, and Dashboard such that the can be deployed and scaled in an efficient way. Here, Worker profile executes stream processing logic, Manager profile deploys and manages stream processing apps on Worker profiles, and the Dashboard profile consists of the status dashboard, the business user dashboard and the business rules manager. As we will discuss later, these profiles provide fine grain control over the deployment architecture.
Scenario
As an example, let’s consider a production pipeline of a manufacturing plant such as electric bikes and processor chips. We need to monitor the quality of the products, the rate of production, and notify necessary authorities when there are important production incidents such as defects in products, declined throughput of production output, and machine failures.
This use case has two parts. The system first needs to detect quality and production issues. Then it has to notify them to appropriate authorities. We can collect the status of the production machines by inspecting the products produced at various stages of the pipeline and publishing them to WSO2 Stream Processor (WSO2 SP). The streaming application running in WSO2 SP will identify anomalies and generate new notifications streams. Finally, we need to manage notifications, deliver them to relevant authorities, provide event format transformations, rate limiting, redelivery, and escalation as needed. Templated Siddhi streaming applications and business rules management component of WSO2 Stream Processor can handle this. The following section shows a reference architecture to achieve this.
Reference Architecture
Following reference, architecture shows how to use WSO2 SP to implement a notification management system.
The reference Architecture separates the management of alerts based on user preference from the general stream processing layer that is responsible for detecting alerts because any issue while pushing out the notifications should not hinder the general event processing flow. For example, unavailability of notification endpoints should not produce any back pressure on the processing environment and halt alert detection altogether.
WSO2 SP can consume events from various streaming data sources such as Kafka, JMS, MQTT, HTTP, Databases and via other protocols in data formats such as JSON, XML, text, key-value, CSV and others. The stream processing layer will be the main processing section that processes all incoming events based on general requirements to identify interesting event occurrences that are worth notifying to the end users. These matched events comprising quality and production incidents are then passed to the alert management layer to filter out, rate limit, and to notify the relevant users’ based on their preferences. For example, these notifications will be delivered on a defined rate limits such as and when an incident has happened or one every hour, and they can also be based on users intended notification types and formats, such as e-mails and text messages with custom text content, API calls to endpoints with XML, JSON formats, or many others.
When the stream processing layer detects interesting events based on the defined Siddhi applications, it generates new notification streams containing the identified events. These notification streams need to be further processed, and the alerts should be only delivered to the intended audience based on their required frequency and message format. At times, if end users did not attend alerts on time, these alerts need to be escalated to other stakeholders.
Generating and managing such alerts can be done using the business rules component of WSO2 Stream Processor. This allows business users to create, delete, edit, enable and disable the alerts they receive in a self-service manner without issuing change requests and waiting for the requests to be serviced.
Let’s look at the business rules component in detail.
As the alert management UI of business rules speaks the domain language, it provides full authority for the business users to generate and manage their alerts by themselves.
The business rules component plays a key role in this use case. It let the developers define a common stream processing logic with information such as the messaging protocols, the frequency of alert delivery, and escalation threshold. Then it let them template the logic and map them to a customized domain-specific input form as shown above image. As these mapped forms speak the domain of the end users it will be much easier for the end user to manage their alerts. Furthermore, WSO2 Stream Processor also allows the developers to add validations to the forms such that they can validate the inputs and provide suggestions to the end users. It also allows the developers to combine the collected inputs using JavaScript to generate the necessary content for the stream processing template. Through this, it provides better usability for both end users and developers, where the end users are only asked relevant information in a way they understand, and the developers are allowed to manipulate the input data in any form they prefer. In addition to these, these domain-specific business rules can also be accessed via rest APIs. Hence, they can be easily integrated with existing configuration system in the organization.
Consequently, the notification management system built using WSO2 Stream Processor allows end users to configure the alerts without worrying about the stream processing complexities while shortening the development lifecycle. For the details about how to use business rules and build domain-specific forms, go through the following screencast and for more information refer my post and the user documentation.
Deployment
figure shows how to deploy the discussed reference architecture.
Based on the diagram the stream processing layer will contain worker and manager profiles for stream processing, and the alert management layer will additionally contain the business rules component for managing notifications. Based on the scale of the system WSO2 Stream Processor can be deployed either in High Available (HA) or in the distributed deployment model. Further, the stream processing layer and alert management layer can have two separate WSO2 Stream Processor clusters or share the same cluster based on the scalability and high availability needs. This is because the stream processing layer, and the alert management layer depicted in the above deployment diagram do not mandate a physical separation between them, but rather they only represent a logical separation in the overall stream processing flow.
Conclusion
In this post, we looked at a reference architecture to build a notification management system using WSO2 Stream Processor and its business rules component. We discuss how WSO2 SP can go beyond simple alert generator and act as a sophisticated alert manager supporting domain-specific notification rules building and management, validating business rules for correct user input while providing guaranteed message delivery for mission-critical data. To implement the above solution, download and try the latest stable version of WSO2 Stream Processor, and to get more details about the product by visiting WSO2 SP website.
