Making Stream Processing Accessible to Business Users
Current Big Data environment is complex with multiple options for processing data, such as batch, realtime, and machine learning. Nowadays organizations have understood the importance of time and started to focus more on realtime and machine learning technologies to achieve immediate diagnosis and future predictions. Stream Processing is increasingly gaining a lot of popularity among developers as the way to do realtime processing in a faster, distributed and scalable manner. But business users are still reluctant to have a first hand experience in using this technology.
When compared to other data analytics approaches, Stream Processing is much more straightforward where it consumes incoming streaming data such as events coming from a Kafka Topic, and executes predefined rules on them. Some Stream Processing systems also have the capability of accessing historical data and doing predictions in the realtime. Further, it also does not need an intervention of a Data Scientist who need to get involved in continuous monitoring and configurations, because if we setup once, then it can continue to run and check for anomalies by itself.
Yet, as we all know neither the market or the business does not stay still, and their requirements always change. With this constant changes, without an agile development environment, the organizations will fail to meet their deadlines in keeping up their realtime systems uptodate. Also, more often these change requests are simple tweaks such as changing the KPIs or threshold values.
Overtime, Stream processing vendors have realized this requirement, and now there is an increased momentum on building drag-and-drop and building form based query builders to get business and non technical folks to use Stream Processing systems. Companies such as Hortonworks, and Tibco Streambase have followed this path. Though this give some advantages to a novice technical user, based on my experience with the customers it does not give a lot of advantage to non-technical, business and even to highly technical user. This is because a highly skilled users find these as inefficient way to implementing streaming applications, where they usually prefer typing the rules compared to meddling with complicated UIs. For this reason, some systems like Spark, Flink, & Storm have continued to stick with SQL-Like or general purpose programing language to implement stream processing applications. Also, when it comes to business and non-technical users, they find this very complex both in understanding streaming terminologies and concepts such as windows, partitions, patterns, etc. Therefore, these users need more domain specific configurations to manage such systems.
I’m my opinion, the best way forward is to build a domain specific configuration UIs such that it allows business users to directly interact with the system. But, generally building such UI require immense effort, where it takes a lot of development time and it can also be error prone. At the same time we can’t also provide prebuilt UIs, because it might work for one specific use case but not for the other, and further it also should confess the users domain, be customizable as per the organization, and support validations.
With modern technology and innovation, building domain specific UIs have become much more feasible. New systems like WSO2 Stream Processor 4.2 takes an innovative approach of templating configurations and enabling developers to build a domain specific UIs for business users in just few minutes. This reduces the complexity of the processing logic and give only the required probes to the business users to manage the system.
The following steps explains the process of creating and managing templated domain UIs in WSO2 Stream Processor 4.2, including steps done by developers and business users:
The developer:
Step 1: Builds a standard stream processing application.
In this case it’s a SiddhiApp file, that can be generated via the Siddhi Editor.
Step 2: Template the application with business information and necessary validation such that it confess the domain of the end user.
Though this is quite straightforward to manually generate templates, WSO2 Stream Processor provides a template builder tool that makes building templates effortless.
Step 3: Deploy templates in production as business rules, such that the business users can manage those rules.
The business user:
Step 1: Access and configure the domain specific business rules from the management UI.
Here with a single click the system will automatically deploy the rules as Siddhi App with appropriate configurations in to relevant nodes. This also supports deploying the application in a distributed and scalable manner.
Step 2: As and when the business requirement changes, the business user modifies, disable or enable the rule via the management UI.
As always there can be exceptions, if the requirement has drastically changed, then the business users will not be able to handle this by themselves. At this point, such request has to go through the traditional change request process. Nevertheless, the chances of this circumstance is minimum. Even if it happens quite often, we are not losing anything, because the effort for building templated rules are insignificant, and hence templated rule can be regenerated by the developer.
Following screencast demonstrate how we can use WSO2 Stream Processor to build and manage Business User friendly templated rule management that we have discussed so far.
The major advantage of this approach is, its flexibility and autonomy that it brings to the business user, and letting them control the system they manage with almost no addition effort from the developers.