All about the WSO2 SP Business Rules Manager
As the first project of my internship @ WSO2, I got the opportunity of working on the Business Rules Manager for WSO2 Stream Processor 4.
In stream processing, we come across scenarios where data is received from a source, sent through a stream, processed, optionally stored, and finally sent to a sink in order to be published to an endpoint.
For an example, we will consume readings of a temperature sensor from an external Kafka source, process them to identify whether the temperature is above 20℃, and send email alerts for such events.
When Stream Processing is done with the WSO2 Stream Processor, we use Siddhi for processing these events. We write Siddhi Applications and deploy them, to perform their respective stream processing tasks.
What is the Business Rules Manager?
For a business use case, there can be one, or sometimes many related Siddhi apps involved in processing data from streams. When requirements for these business use cases change, such changes can be accommodated by modifying necessary Siddhi apps appropriately. This can be done by anyone who has some knowledge in Siddhi (typically, a developer or a data scientist).
There are instances where such business requirement changes happen frequently, and these frequent changes are minor tweaks.
As a very basic example, let’s consider monitoring a company’s stock exchange values. If the share volume of the company exceeds X, you will be getting an email. Here, the value of X will be adjusted over time.
Business use cases can be very complex, where such adjustments might have to be made in so many places across so many Siddhi apps.
Since these are frequent adjustments, always expecting a developer is going to hold you back in time. Business users should be able to do these changes by themselves, without touching lines of Siddhi code.
In order to serve these kinds of purposes, a developer can group necessary Siddhi apps for a rapidly changing business use case, in the form of skeletons. So that, actual working Siddhi apps can be derived by business users — using those skeletons, and accommodating necessary business requirement changes into them.
The Business Rules Manager for WSO2 Stream Processor provides the ability to maintain such skeletons, and derive deployable Siddhi apps from them.
A Business Rule is simply a mapping between such skeleton, and the actual Siddhi app (or group of Siddhi apps) produced by the skeleton.
Thinking of Templates
It is necessary to recognize the parameters, that could change with respect to a business use case. The values for those parameters are going to change based on business requirements, from time to time.
A Developer who has some knowledge about developing Siddhi apps, is going to recognize these parameters and create the skeletons.
A Business User who has no/less knowledge about developing Siddhi apps, is going to use those skeletons, in order to derive Siddhi apps — to serve an updated business requirement.
An Example Scenario
Let’s think of an example scenario, and understand how this can be solved with the Business Rules Manager.
David wants to analyze ATM transactions of Unicorn Bank. Unicorn Bank has ATMs around the country. He wants to get the transaction details by email, when any withdrawal of an amount of more than 5000 is made from the ‘city’ branch.
The following SiddhiApp can be used by David, to work according to the above mentioned scenario.
Now, what if David wants deposits, where amount is above 6000? What if he is interested in another branch than ‘city’? And what if David is not David, but Thomas?!
When requirements for this kind of a scenario change, these values are going to be different, but everything else remains the same. So instead of storing with actual values, we are going to identify such elements, and template them like ${this}.
define stream UnicornATMStream(debitCardNo string, branch string, transaction string, amount double);@sink(type='email',
username ='unicornAdmin',
address ='unicornAdmin@gmail.com',
password= 'unicornAdmin',
subject='Notification of withdrawal: cardNo: {{debitCardNo}}',
to='${emailAddress}',
port = '465',
host = 'smtp.gmail.com',
ssl.enable = 'true',
auth = 'true',
@map(type='text'))
define stream NotificationStream(debitCardNo string, branch string, transaction string, amount double);@info(name='query1')
from UnicornATMStream[(branch == '${branch}') AND (transaction == '${transactionType}') AND (amount > ${thresholdAmount})]
select *
insert into NotificationStream;
Each ${templatedElement} can now change if we replace them.
David is going to take the role of developer and going to create the above skeleton, since he knows Siddhi.
Anyone else can take the role of a business user and create SiddhiApps using that skeleton. They are not expected to have much knowledge about Siddhi.
How are these templates going to be stored, and how are they going to produce actual SiddhiApps?
Let’s begin to understand that, by getting to know about the entities used in the Business Rules Manager.
Let’s dive in!
Template
This is a skeleton for a SiddhiApp. Each templated element will be denoted like ${this}. type must be siddhiApp and the templated SiddhiApp will be contained in the content.
{ "type": "siddhiApp", "content": "<templated SiddhiApp body>" }Let’s include the SiddhiApp’s skeleton from our scenario.
{
"type": "siddhiApp",
"content":
"@App:name('atmNotificationsApp')
define stream UnicornATMStream(debitCardNo string,
branch string, transaction string, amount double);
@sink(type='email',
username ='unicornAdmin',
address ='unicornAdmin@gmail.com',
password= 'unicornAdmin',
subject='Notification of withdrawal: cardNo:
{{debitCardNo}}',
to='${emailAddress}',
port = '465',
host = 'smtp.gmail.com',
ssl.enable = 'true',
auth = 'true',
@map(type='text'))
define stream NotificationStream(debitCardNo string,
branch string, transaction string, amount double);
@info(name='query1')
from UnicornATMStream[(branch == '${branch}') AND
(transaction == '${transactionType}') AND
(amount > ${thresholdAmount})]
select *
insert into NotificationStream;"
}Property
Being self descriptive is necessary for each ${templatedElement}. To describe an element, a property should be defined with the same name of it. Each one is going to be replaced by the user, just before deriving a SiddhiApp. An input field will be getting the input for an element.
So that, each ${templatedElement} must be described as a property in the following format.
"propertyName": {
"fieldName": "Display name for the field when getting input",
"description": "Short description about the property",
"defaultValue": "Default value for the property",
"options": ["option1", "option2"]
}Notice the options array. This is an optional element. Before understanding the purpose of it, let’s try to include a property from our scenario without this array.
"branch": {
"fieldName": "Interested Branch",
"description":
"Only transactions from this branch will be considered for
notification",
"defaultValue": "city"
}There will be only a limited number of available branches for the bank — isn’t it? Therefore, we should be able to select a branch only from a list of available branches.
This is the purpose of the options array. We are going to include each branch as a member of the options array.
When the value for a property is not limited in such way, we are not going to include the options array for that property.
fieldName, description & defaultValue are anyhow mandatory.
- Note that, the
defaultValueshould be one among the options included, when there is anoptionsarray present.
Let’s now add the options for the branch property .
"branch" : {
"fieldName" : "Interested Branch",
"description" :
"Only transactions from this branch will be considered for
notification",
"defaultValue" : "city",
"options": [
"city",
"main",
"hillSide",
"countryTown"
]
}Simply, each prompt for a value from the user, should be a property.
Rule Template
A template along with properties that describe them, will be contained in a Rule Template.
{
"name": "Name of the rule template",
"uuid": "Unique ID for the rule template",
"description": "A short description",
"instanceCount": "Usage constraint of the rule template",
"type": "Type of the rule template",
"script": "To do some magic on given values",
"templates": [
{template1},
{template2}
],
"properties": {
"property1": {property1},
"property2": {property2}
}
}A Rule template will have a uuid, name, description to identify & describe itself. The instanceCount can be either one or many, which specifies that how many times we can use this rule template to create a new business rule.
Note: With the instanceCount being mentioned here, rule templates and usable nodes are mentioned in WSO2 Stream Processor’s deployment configurations. Refer the documentation for the instructions.
A Rule Template can have a type of input, output or template.
1.‘template’ Rule Templates
These can have more than one templates under them. Each such template's content will be a full SiddhiApp, that is templated with elements.
This is used when full SiddhiApp(s) is/are about to be derived by filling templated elements.
Let’s create a rule template of type template for our scenario.
{
"name": "ATM Analysis Template",
"uuid": "atm-analysis-template",
"type": "template",
"instanceCount": "one",
"description":
"Notify high amount of transactions from an ATM",
"templates": [
{ "type": "siddhiApp",
"content":
"define stream UnicornATMStream(
debitCardNo string, branch string,
transaction string, amount double);
@sink(type='email',
username ='unicornAdmin',
address ='unicornAdmin@gmail.com',
password= 'unicornAdmin',
subject='Large amount of ${transactionType}:
cardNo: {{debitCardNo}}',
to='${emailAddress}',
port = '465',
host = 'smtp.gmail.com',
ssl.enable = 'true',
auth = 'true',
@map(type='text'))
define stream NotificationStream(
debitCardNo string, branch string,
transaction string, amount double);
@info(name='query1')
from UnicornATMStream[(branch == '${branch}') AND
(transaction == '${transactionType}') AND
(amount > ${thresholdAmount})]
select *
insert into NotificationStream;"
}
],
"properties": {
"emailAddress": {
"fieldName": "Email Address",
"description":
"Email address to which notifications will be sent",
"defaultValue": "example@gmail.com"
},
"branch": {
"fieldName": "Interested Branch",
"description":
"Only transactions from this branch will be
considered for notification",
"defaultValue": "city",
"options": [
"city",
"main",
"hillSide",
"countryTown"
]
},
"transactionType": {
"fieldName": "Type of transaction",
"description":
"Type of the transaction made through the ATM",
"defaultValue": "withdrawal",
"options": [
"withdrawal",
"deposit"
]
},
"thresholdAmount": {
"fieldName": "Threshold limit",
"description":
"Notification is sent when transaction amount
exceeds this value",
"defaultValue": "5000"
}
}
}2.‘input’ Rule Templates
These will have only one template under them. The content will be just the input part of a SiddhiApp. Input part of a SiddhiApp, is a defined source along with a stream definition. The stream definition should be clearly defined as exposedStreamDefinition in the one and only available template, in order to identify that from which stream, do the events arrive.
This is used, when only a SiddhiApp’s input part is about to be derived by filling templated elements.
{
"name": "Name of the input rule template",
"uuid": "Uniquely identifiable ID for the input rule template",
"description": "A short description",
"templates": [
{"type": "", "content": "", "exposedStreamDefinition": ""}
],
"properties": {
"property1": {property1 object},
"property2": {property2 object}
}
}Let’s create an input rule template with our scenario.
{
"name": "ATM Analysis Input",
"uuid": "atm-analysis-input",
"type": "input",
"instanceCount": "many",
"description":
"Configured http source to recieve ATM transaction data",
"templates": [
{ "type": "siddhiApp",
"content":
"@App:name('atmAnalysisInput')
@Source(type = 'http', receiver.url='${receiverUrl}',
basic.auth.enabled='false',
@map(type='text'))
define stream UnicornATMStream(debitCardNo string,
branch string, transaction string, amount double);",
"exposedStreamDefinition" :
"define stream UnicornATMStream(debitCardNo string,
branch string, transaction string, amount double);"
}
],
"properties": {
"receiverUrl": {
"fieldName": "Receiver URL",
"description": "Enter the URL of the http receiver",
"defaultValue": "https://localhost:8005/ATMInputStream"
}
}
}3.‘output’ Rule Templates
These will have only one template under them: which is just the output part of a SiddhiApp. Output part of a SiddhiApp, is a defined sink along with a stream definition. An exposedStreamDefinition should be present in the template, similar to an input rule template. This states that to which stream, are the events sent.
This is used when, only a SiddhiApp’s output part is about to be derived by filling templated elements.
Let’s create an output rule template for our scenario.
{
"name": "ATM Analysis Output",
"uuid": "atm-analysis-output",
"type": "output",
"instanceCount": "many",
"description":
"Configured email sink to send ATM transaction
notification",
"templates": [
{ "type": "siddhiApp",
"content":
"@App:name('atmAnalysisOutput')
@sink(type='email',
username ='unicornAdmin',
address ='unicornAdmin@gmail.com',
password= 'unicornAdmin',
subject='Large amount of withdrawal: cardNo:
{{debitCardNo}}',
to='${emailAddress}',
port = '465',
host = 'smtp.gmail.com',
ssl.enable = 'true',
auth = 'true',
@map(type='text'))
define stream NotificationStream(debitCardNo string,
branch string, transaction string, amount double);",
"exposedStreamDefinition":
"define stream NotificationStream(debitCardNo string,
branch string, transaction string, amount double);"
}
],
"properties": {
"emailAddress": {
"fieldName": "Email address",
"description":
"Enter the email address to send notifications to",
"defaultValue": "example@gmail.com"
}
}
}Rule Template Script
As we already know, a user will enter values, and they will replace a ${templatedElement} directly.
A script contains JavaScript, which can be present whenever the user input should be processed in some manner (doing validations, calculating average of two inputs, etc.), before replacing the ${templatedElement}.
For example, rather than getting an average value directly from the user, we can get the minimum value and maximum value, and write a JavaScript function within the script to calculate and return the average. (Refer the above image)
If a ${templatedElement} is going to be processed in such manner, a variable with the same name of the respective ${templatedElement} should be defined as a global variable of the javascript, only with the var keyword.
____ SiddhiApp template content ________________
| |
| ... from myStream[value > ${average}] ... |
| |
|________________________________________________| ____ Script ____________________________________
| |
| var average; |
| |
|________________________________________________|
These variables can be assigned with values returned from functions. Values of these variables will go and replace respective ${templatedElement}s in template(s).
Each such element won’t be defined as aproperty, since it is not going to come from the user directly. Instead of that, the script will contain additional${templatedElement}s, for each input to be acquired from the user.
var average = getAverage(${minValue}, ${maxValue});function getAverage(min, max) {
return (max - min)/2 ;
}
${average} will be in a template, and it will be replaced with whatever the value that is going to be had in the average variable.
average variable, is going to have the value returned from getAverage function, with minValue and maxValue — provided by the user passed in as its arguments.
minValue and maxValue are going to be defined as properties, since user is supposed to provide values for them.
Note: Instead of using functions, you can have simple mechanisms such as concatenations.
For example,
var notificationSubject = 'Alert from ' + ${factoryName};
Let’s apply a function to our example.
In our ‘template’ type of ruleTemplate, instead of getting the email address directly from the user, we are now going to to get the username for his/her gmail address. A function is going to add the ‘@gmail.com’ part to it.
{
... other elements of rule template ... "script":
"var emailAddress = getEmail('${gmailUsername}');
function getEmail(username) {
return username + '@gmail.com';
}",
"templates": [
{ "type": "siddhiApp",
"content":
"subject='Large amount of ${transactionType}:
cardNo: {{debitCardNo}}',
to='${emailAddress}',
... rest of the content ... "
}
],
"properties": {
"gmailUsername": {
"fieldName": "Gmail Username",
"description":
"Username of your Gmail address to which
notifications will be sent",
"defaultValue": "example"
},
... other properties ...
}
}
Template Group
This is the domain, which categorizes business use cases or scenarios. ruleTemplates that have templates usable for similar kinds of scenarios, will be contained by a templateGroup.
A templateGroup will have a name, uuid and a description to identify and describe itself.
Here is the entire templateGroup for our example. (Considering readability, I haven’t added the entire content of each ruleTemplate)
{
"templateGroup": {
"name": "ATM Analysis",
"uuid": "atm-analysis",
"description": "Analyze ATM Transactions",
"ruleTemplates": [
{ <INPUT RULE TEMPLATE> },
{ <OUTPUT RULE TEMPLATE> },
{ <TEMPLATE RULE TEMPLATE> }
]
}
Note: Rather than manually creating a
templateGrouplike this, you can use the Business Rules Template Editor, which makes this process hassle free. I will be writing about the Business Rules Template Editor in my next blog post.
Business Rule
As we already know, this is the mapping in between a skeleton of a SiddhiApp (or group of SiddhiApps), and the actual working SiddhiApp(s). A Business Rule will just hold the value of each templated element, and the uuids of the respective ruleTemplate and templateGroup, that they should be mapped to.
‘Creating a Business Rule’ actually means creating this map, which will ultimately derive SiddhiApp(s).
These maps are stored in a database and mapped with the respective skeletons, when requested by the business user.
There are two modes for creating a Business Rule.
1. Business Rule from template
A business rule that was created from template, can produce one or many SiddhiApps, depending on the used ruleTemplate. Complex SiddhiApps can be templated in this way.
A ruleTemplate of type template will be used to create a Business Rule from template.
2. Business Rule from scratch
A business rule that was created from scratch, will produce a single SiddhiApp.
Within this context, we consider a simple SiddhiApp in four high level portions, Input, Output, Logic and Mapping(Input-to-Output).
An input rule template, an output rule template, and custom filters will be used in creating a business rule from scratch.
The Input part says from where data comes from, Logic part says what to do and how to process that data, and the Output part says where to send that processed data. Mapping part says that, data from which field of the input stream should go to which field of the output stream.
The logic is created by the user, while deriving the business rule — which is going to be the filtering condition of the SiddhiApp.
After completing the input & logic parts, stream fields of input part are mapped to the stream fields of output part. This will create a complete flow of data through:
input(source) ⟶ [filter] — < mapping > ⟶ output(sink).
- Note: Even though we say ‘from scratch’, we are not creating a SiddhiApp completely from scratch. We are just partially templating input & output parts, and allowing the user to select different combinations of input — output parts, along with own filter logic.
Choosing the correct type of Business Rule
Business rule from template is suitable when at least one of the following is true
- Only the values in a SiddhiApp are supposed to change; not the input (source), output (sink) and filters.
- More than one SiddhiApps are going to work for the same scenario, and some values might be used in all of them, again and again.
- SiddhiApp(s) that are going to be derived at the end are complex in a way that they can’t be simply considered in four portions: input, output, filters and mapping.
Business rule from scratch is suitable when
- There is a possible list of input sources and output sinks. For each deployment, they can be different.
- Filters are subject to change slightly.
If we put everything in a diagram, this is how the entire concept looks like:
Summarizing the concepts
- Developer will provide the
templateGroups. - Business User will use them and create working SiddhiApps.
- The mapping in between
templateGroupand produced SiddhiApp(s) is a Business Rule - Each
templateGroupcan have manyruleTemplatesfrom any of the 3 types : ‘input’, ‘output’ or ‘template’. - ‘template’ type of
ruleTemplatecan have manytemplates. ‘input’ or ‘output’ type ofruleTemplatescan have only onetemplate. - Any type of
ruleTemplatecan optionally have ascriptto do some magic with and for, the replacements. ruleTemplates are configured with their usable nodes, after theinstanceCountbeing mentioned.- Each
templateis a skeleton for a SiddhiApp/part of a SiddhiApp. - Business Rules are derived from
ruleTemplate(s). - Business Rule from template is entirely derived from a single
ruleTemplateof type ‘template’. - Business Rule from scratch is derived from an ‘input’ type of
ruleTemplate, an ‘output’ type ofruleTemplate, and custom logic from the user
The developer creates templateGroups and stores each as a JSON file in the <SP_HOME>/wso2/dashboard/resources/businessRules/templates folder of the WSO2 Stream Processor. The Business Rules Template Editor tool can be used to create template groups.
He/she also specifies the node URLs and the ruleTemplate uuids that can be used in those nodes, in the deployment.yaml file. [documentation]
Then a business user will access these templateGroups, pick one, and choose ruleTemplates within that templateGroup and create business rules out of them using the Business Rules Manager for WSO2 Stream Processor. [documentation]
A Quick Demo
Creating a business rule from template
- Download the WSO2-Stream Processor and extract it.
- Save the prepared
templateGroupin the following folder :<SP_HOME>/wso2/dashboard/resources/businessRules/templates - Open the
deployment.yamlfile in the following folder:<SP_HOME>/conf/dashboard - Find
wso2.business.rules.manager, and you will get to the following part of thedeployment.yamlfile:
wso2.business.rules.manager:
datasource: BUSINESS_RULES_DB
deployment_configs:
-
#ip:port of the node
localhost:9090:
# uuids of rule templates which are needed to be deployed on the node given above
- stock-data-analysis
- stock-exchange-input
- stock-exchange-output
- identifying-continuous-production-decreasesimply add the uuids of each rule template we have created, after the last line and save deployment.yaml
- atm-analysis-template
- atm-analysis-input
- atm-analysis-output5. Go to <SP-HOME>/bin from your terminal.
6. Start both worker and dashboard run times by issuing ./worker.sh and ./dashboard.sh . ( On Windows, use worker.bat --run and dashboard.bat --run )
7. Access the Business Rules Manager from the URL: https://<SP_HOST>:<HTTPS_PORT>/business-rules (eg: https://0.0.0.0:9443/business-rules/ )
8. Click on ‘Create’, and choose ‘From Template’.
9. Click on the template group ‘ATM Analysis’, which we have saved just now. You will be taken to the below screen. Only ruleTemplates of type template will be shown here. Select the one and only available ruleTemplate, that we created.
10. The form shown after selecting that, is where we are going to provide the replacements for ${templatedElement}s. Fill in with some values.
I have modified the username, branch and the transactionType. Notice that how each member of the options array — provided under anyproperty is shown in a drop down.
11. You will get a success message, and will be redirected to the landing page .
Now we are done with creating a business rule, that is a collection of SiddhiApps. For this case, only one SiddhiApp is there within this business rule.
Let’s find the deployed SiddhiApp, and check the content of it. Go to the directory <SP_HOME>/wso2/worker/deployment/siddhi-files. There will be a file named high-deposit-values-for-main-branch_0.siddhi. That is the SiddhiApp, we have derived from the Business Rules Manager.
Let’s simulate some stream data and see how our SiddhiApp works.
Note: This is not a part of Business Rules deployment. We are just making sure that the SiddhiApp is in a working condition.
12. Start the Stream Processor Studio by going to <SP_HOME>/bin in the terminal and issuing the command ./editor.sh ( In Windows, editor.bat). Access it from the URL http://localhost:<EDITOR_PORT>/editor.
13. Open the saved SiddhiApp by going to <SP_HOME>/wso2/worker/deployment/siddhi-files/high-deposit-values-for-main-branch_0.siddhi.
14. Replace the username , address and password within its @sink() part, with valid gmail address privileges. That’s because, there’s no email address called unicornAdmin@gmail.com, and you need one working email to send the data from :) . Save the SiddhiApp. You will see the following in the console :
Siddhi App high-deposit-values-for-main-branch_0 successfully deployed
15. Open the Event Simulator from its button located on the left hand side (or press ctrl + shift + I)
Select the SiddhiApp Name and the Stream Name as we have saved now. The Attributes will be automatically loaded. Enter some data within them, which is similar to making a transaction via a debit card in an ATM of the Unicorn Bank. You have the freedom of choosing the branch, your debit card number and etc!
Login to your email (that you entered in to of the SiddhiApp) and get ready to receive emails!
With the values shown in the left side, if you press the ‘Run’ button, you won’t be getting any emails — since the transaction is not from main branch, nor it’s a deposit, nor its amount is above 5000.
If you simulate a data that could be filtered by our SiddhiApp— you will get a mail.
See that happening in the following screenshot:
Creating a Business Rule from scratch
- Choose the option ‘Create a Business Rule from scratch’. All the steps before that are as same as creating a business rule from template.
- Select the
templateGroupwe have just saved. - Select an input type of
ruleTemplateand fill in the values for the input part. I’m going to keep the default value. Take a look at howexposedStreamFieldsare shown on the right side.
4. Select an output ruleTemplate and fill in values for the output part.
In the Mappings section, you will see each exposedStreamField of the output ruleTemplate under the output column. Each output field should get data from an input field, so that, we have to select a field from the input exposedStreamFields.
5. You can add your own filters and combine them under rule logic. Since we are creating a scenario to filter ‘low deposits from hillSide branch’ (refer the business rule’s name in step 3), I’m going to enter the filters as following
1 AND 2 AND 3 with the above filters, will result in the following filtering condition:
(transaction == 'deposit') AND
(amount < 5000) AND (branch == 'hillSide')6. Save & Deploy the business rule. Then if you check the directory <SP_HOME>/wso2/worker/deployment/siddhi-files , there will be the deployed SiddhiApp with the name low-deposits-from-hillside-branch.siddhi.
That’s how we create business rules using the WSO2 SP Business Rules Manager! You might also be interested in checking out the documentation for the Business Rules Manager.
In my next blog post, I will be writing about the Business Rules Template Editor for WSO2 SP, in which you can create Business Rules Templates.
