How Feedzai ARMS automates rule management in large scale systems
By and
Rule-based systems are still widely used by financial institutions (FIs) to develop fraud detection and anti-money laundering (AML) systems because of their ease of implementation, interpretability, and for compliance reasons. Ideally, these systems will have only the minimum number of rules necessary to achieve the desired detection performance. However, finding and maintaining an optimal set of rules is a time-consuming and complex task. Besides creating and testing each individual rule, it involves evaluating possible different combinations of rules and their priorities, including dropping low-performing or redundant ones.
This post will dive into a new feature we developed at Feedzai — ARMS — that, given a set of possibly hundreds of rules, automatically finds the optimal combination of rules and priorities set.
Why use rules?
Even with the rapid adoption of Machine Learning by FIs and the wider industry, driven by companies like Feedzai, rules are still necessary in most scenarios. Rules are easy to implement by both technical and business users and their triggers are easily interpretable. Ultimately, rules place users in full control of the system.
Additionally, FIs use rules to leverage acquired domain expertise and directly incorporate this knowledge into the system, such as well-known fraud or AML patterns (that a Machine Learning model would need to learn from scratch from illustrative training data).
Finally, rules are often necessary to implement strict compliance regulations. For instance, in the AML use case, organizations have to incorporate a predetermined set of rules, which cannot be replaced by the predictions of even the most sophisticated Machine Learning model.
Rules in practice
Let’s briefly describe a rule-based system for fraud detection and AML. In general, analysts identify patterns in the data and manually design a set of rules to capture as many fraudulent transactions as possible, while simultaneously trying to keep the number of alerts and false positives as low as possible.
Examples of rules might be:
- if the user made too many transactions in the last hour, send the transaction to review;
- if the user is on a blocklist, decline the transaction;
- if the user is a known bank employee, accept the transaction.
Importantly, even when applicable regulations are very specific and prescriptive, there is usually some leeway on how to tune the rule set.
As the examples above demonstrate, rules may have different outcomes, e.g., accepting, declining, or reviewing a financial transaction, or even a financial entity (e.g., customer, account). Rules also require priorities. Consider the case in which a financial transaction triggers all the three rules above: what outcome should prevail? The final decision depends on user-assigned rule priorities.
Why automate rule management?
As the number and complexity of rules increase, rule-based systems become harder to manage and optimize. As a result, their performance may degrade. Additionally, as the complexity of the system increases, so do its operating costs.
Consider the following example of why a rule set may need revision and optimization over time.
Imagine a system with a single rule: Rule A. This rule, when triggered, sends a transaction to review, i.e., it is a review rule. During the review, an analyst decides whether the transaction is legitimate (i.e., to accept) or fraudulent (i.e., to reject). We can see above that Rule A properly identifies a region where legitimate and fraudulent transactions coexist, thus needing expert judgment.
After some time, there is a decision to extend the system by including a new rule: Rule B. Rule B, when applicable, automatically accepts a transaction, i.e., it is an accept rule. As illustrated above, Rule B performs brilliantly in approving legitimate transactions.
Over time, analysts identify a pattern in the reviewed transactions. Hence, they decide to create a new rule: Rule C. Rule C declines transactions with specific characteristics, i.e., it is a decline rule. There is a priority adjustment to ensure that Rule C prevails over Rule A when triggered simultaneously to minimize unnecessary reviews. It effectively declines fraudulent transactions and prevents unnecessary reviews.
At this point, is the system optimized? In other words, is the current combination of three active rules and their priorities the simplest satisfying performance requirements? Not quite. Rule C isolates the fraudulent transactions previously triggered by Rule A, removing ambiguity. Once the system includes Rule C, Rule A becomes redundant as it only triggers legitimate transactions. Rule A undermines the system’s performance by triggering legitimate transactions and generating unnecessary manual reviews.
Finding the optimal rule set
The example above naturally begs the question: how to automatically find the optimal combination of rules and priorities from the original rule set? With this objective in mind, we have developed Feedzai ARMS, an Automated Rule Management System, which supports three optimization methods: random search, greedy search, and genetic programming.
The random search algorithm represents the simplest optimization approach and serves as a baseline for comparison with more sophisticated techniques, setting the bar for performance, computation complexity and convergence speed. As the name suggests, it randomly turns rules on and off and attributes random priorities for as many iterations as specified.
There are two key advantages to this approach. First, it is very simple to implement and test. Second, it is guaranteed to find the optimal solution (if there is one) with enough iterations. The main disadvantage is that it may take a prohibitive amount of time to find the optimal (or at least close to optimal) solution(s).
The second optimization method that ARMS supports is the greedy search algorithm. The greedy search starts from an empty rule set and adds the best rule at each iteration, until the specified maximum number of iterations (see how the “best rule” is selected in the next section). The main advantage of this approach is that it is faster than the random algorithm, as it does not go through all possible configurations. The main disadvantage is that, due to this, it is not guaranteed to find the optimal solution. To further improve the algorithm, we implemented some novel techniques for the initialization and “backtracking”: the ability to walk back and remove previously added rules.
Notably, we have also thought that this optimization problem would be a perfect fit for a genetic algorithm. In essence, it mimics natural selection and biological evolution. Here’s how we formulate the problems: we start with a population consisting of a fixed number of different rule sets, i.e., the individuals. A set of variables characterize each rule, indicating whether it is on or off and its priority, i.e., the genes. A chromosome comprises all genes (whether each rule is turned on or off and its priority) for each individual (a candidate rule set).
At each iteration, the best rule sets are selected to be part of the next generation and to generate new individuals (children) through crossover and mutations. These two processes for individual generation result in increased variability, and work by mixing two individuals into a new one (crossover), and by introducing gene alteration (mutations).
Ideally, the new generations of individuals will be better than the previous ones. The process continues for a specified number of generations (or epochs) and returns the best rule set configuration (or chromosome) in the final population.
Empirically, we found that the genetic algorithms implemented in ARMS were the fastest to converge. If you need to find a good, but not necessarily the optimal, solution, the greedy method proved to be very efficient. The random method can find the optimal solution but at the cost of requiring many more iterations, perhaps better spent tuning the parameters of the genetic algorithms.
What is the best rule set among several candidates?
In the approaches described above, at each iteration we select the “best rule set”. How do we know which one is the best? For that, we compute the loss function, which is a linear combination of performance metrics of interest, subject to business constraints as required. For example, a loss function 𝝀 for a given rule set R in ARMS may be configured as:
where 𝛼 and 𝛽 are parameters that can be chosen so that the search converges to rule sets with high recall* and low FPR*, and FPRmax causing solutions with an FPR above a certain value to be penalized further.
Moving to the juicy part — how does ARMS perform?
Can Feedzai ARMS bring the promised benefits in actual use-cases using these optimization algorithms? To verify this, we performed experiments in three types of data: synthetic, real merchant, and banking data. In all of these cases, ARMS significantly reduced the number of active rules while maintaining or improving the system’s detection performance. More details can be found in our paper available here. In this post, we present the results for the experiments on a banking dataset.
In summary, the original rule set for this banking use case had more than 200 active rules. ARMS was able to find a subset of 40 rules that maintained the decline rate* of the original set and increased transaction recall by 5 percentage points (p.p.) and money recall by 3 p.p. at the same time.
Due to ARMS fully customizable loss function, it was possible to tune the old rule system to decrease the decline rate to half of the original and still increase transaction recall by 2 p.p. 😲. This decrease in the decline rate directly translates to billions of dollars of legitimate customers transactions being approved instead of blocked.
Conclusion and Next Steps
Using efficient optimization approaches such as greedy and genetic algorithms, Feedzai ARMS significantly reduces the number of active rules, and handles different rule priorities, all while achieving the detection performance required by our clients.
What are the next steps for ARMS? Finding an optimal ordered set of active rules from the original set of rules carefully designed by risk analysts already brings a significant impact — especially for systems with hundreds of rules. But we believe more can be done to improve these rules systems’ detection performance and decrease the manual, labor-intensive work required of risk analysts. In addition to selecting best rules, automatically adjusting the parameters in the rules logic and automatically suggesting new rules based on patterns observed in the data are the next steps in evolving Feedzai’s already first-class rule capabilities and the world’s first RiskOps platform.
Footnotes
decline rate — number of transactions declined divided by total number of transactions
transaction recall — number of correctly declined transactions divided by total number of declined transactions
money recall — sum of money of correctly declined transactions divided by sum of money of total number of declined transactions
FPR (false positive rate) — number of transactions wrongly declined divided by total number of transactions legitimate
Thanks to David Aparício, João Bravo, João Ascensão and Pedro Bizarro.
