Ramanujan Engine, Airtel’s answer to dealing with business decisions on Millions of records quickly! — Part 1

This article describes how at Airtel, Ramanujan Rule Engine product idea was conceived and how it got matured with various design ideas. We will also discuss the architecture and its complete Production processes including Monitoring CI/CD and likewise.

as we will be discussing it in a detailed way, so we have decided to put this blog in multiple parts

1) Business Requirements, key challenges and list of approaches

2) Final Solution and Architectural Learnings

In this part of the blog, we will discuss the business requirement and key challenges involved. In addition, we will also list the approaches that we have tried to implement the requirement.

Business Requirement:

At Airtel & similar organisations, as digital transformation has progressed, we have a lot of systems coming up. At times, there are business cases that require taking decisions & actions basis the milestones published by these systems. To stitch this, Rule Engine was conceptualised which can ingest data from multiple systems (Real-time, Near real-time, Batch mode). When the rules keep on changing frequently Rule engine helps in maintaining speed and agility.

A few of the requirements for which rule engine is used are as below :

1. Incentive Compensation

2. Gamification

3. Sales SR

4. Common Order Monitor

5. Lead Prioritization

6. Dispatcher

7. Risk Monitor

8. Next best action

Many more in the future ….

All the above requirement rules are computed against transactional data which is online as well as offline. When such data are generated across systems business logic shall be applied with pre-defined criteria and appropriate rules are triggered. Once the rules are evaluated relevant actions are taken.

Rule-Based Systems:

Rule-based systems (also known as production systems or expert systems) are the simplest form of artificial intelligence. A rule-based system uses rules as the knowledge representation for knowledge coded into the system. The definitions of rule-based systems depend almost entirely on expert systems, which are systems that mimic the reasoning of human experts in solving a knowledge-intensive problem. Instead of representing knowledge in a declarative, static way as a set of things that are true, a rule-based system represents knowledge in terms of a set of rules that tells what to do or what to conclude in different situations.

Market Research:

As part of this exercise, we went through a list of systems like match rules with huge datasets and with the ability to perform such operations online as well as offline.

We found a paper that matches the criteria of what we wanted and below is the synopsis of what the paper talked about.

Production Matching for Large Learning Systems

http://reports-archive.adm.cs.cmu.edu/anon/1995/CMU-CS-95-113.pdf

Abstract:

Rule-based systems execute actions based on predefined rules. These systems are widely used in business, government, and various organizations. Traditional rule-based systems perform poorly in big data applications such as IoT and others. To meet this challenge, we exploit the inherent nature of the problem to come up with a distributed processing system. We propose a scalable real-time rule-based engine model to logically distribute the events/data streams generated and apply inference on each logically separated event stream.

Following is a list of algorithms that are available for designing and implementing Rule-based engines :

1) Forward-Chaining algorithm

2) Backward-Chaining Algorithm

3) Rete Algorithm

Forward-Chaining Algorithm:

Forward-chaining is also known as a forward deduction or forward reasoning method when using an inference engine. Forward-chaining is a form of reasoning which starts with atomic sentences in the knowledge base and applies inference rules (Modus Ponens) in the forward direction to extract more data until a goal is reached.

The Forward-chaining algorithm starts from known facts, triggers all rules whose premises are satisfied, and add their conclusion to the known facts. This process repeats until the problem is solved.

1) It is a down-up approach, as it moves from bottom to top.

2) It is a process of making a conclusion based on known facts or data, by starting from the initial state and reaching the goal state.

3) Forward-chaining approach is also called the “data-driven approach” as we reach the goal using available data.

4) Forward-chaining approach is commonly used in expert systems, such as CLIPS, business, and production rule systems.

Backward Chaining Algorithm:

Backward-chaining is also known as a backward deduction or backward reasoning method when using an inference engine. A Backward-chaining algorithm is a form of reasoning, which starts with the goal and works backwards, chaining through rules to find known facts that support the goal.

1) It is known as a top-down approach.

2) Backward-chaining is based on the modus ponens inference rule.

3) In Backward-chaining, the goal is broken into sub-goal or sub-goals to prove the facts true.

4) It is called a goal-driven approach, as a list of goals decides which rules are selected and used.

5) Backward-chaining algorithm is used in game theory, automated theorem proving tools, inference engines, proof assistants, and various AI applications.

6) The Backward-chaining method mostly used a depth-first search strategy for proof.

Challenges on above two approaches:

Forward-Chaining challenges:

1) The process of Forward-chaining may be time-consuming.

2) It may take a lot of time to eliminate and synchronize available data.

3) Unlike Backward-chaining, the explanation of facts or observations for this type of chaining is not very clear.

4) The former uses a goal-driven method that arrives at conclusions efficiently.

Backward-Chaining challenges:

1) The process of reasoning can only start if the endpoint is known.

2) It doesn’t deduce multiple solutions or answers.

3) It only derives data that is needed, which makes it less flexible than Forward-chaining.

Rete Algorithm approach:

The Rete algorithm is a pattern matching algorithm for implementing rule-based systems. The algorithm was developed to efficiently apply many rules or patterns to many objects, or facts, in a knowledge base. It is used to determine which of the system’s rules should fire based on its data store, its facts. The Rete algorithm was designed by Charles L. Forgy of Carnegie Mellon University, first published in a working paper in 1974, and later elaborated in his 1979 PhD thesis and a paper titled “Production Matching for Large Learning systems”.

The below figure gives an idea of Rete as a black box with inputs and outputs

Example

Features of Rete Algorithm

1) Uses State-saving approach to save the matching process.

2) Sharing of nodes between productions and similar conditions

Existing Rule Engine systems available in market

Drool Engine:

· Originally developed in 2005

· Apache license, written in Java

· Has multiple projects:

· Drools Workbench: Web UI for Authoring

· Drools Expert: Business Rules Engine

· Drools Fusion: Complex Event Processing

· jBPM: Integrate Rules into workflows

Provides guided rules, technical rule DRL syntax, and support for Domain Specific Language (DSL)

Uses a forward and backward chaining inference engine that leverages the PHREAK algorithm

Supports in-process and standalone deployments

Example of rule written on Drool

As we can see the writing rules are not such user friendly. As the system uses the above two approaches we have decided to use the Rete approach to address the challenge. To be continued in part 2.

In the next part of the blog, we shall show the technicalities of the solution and some architectural considerations.