How to build a Fraud Detection System

Now that you have decided to build your own fraud system, here are some tips on how to build it and architect a fraud monitoring system. You will learn how to iteratively build out a system, the key KPIs to monitor, what data to collect, and how to secure your software and system.

Essential Components of a fraud monitoring system

Your goal is to identify, flag, report and stop fraudulent activity. To successfully deter fraud, you need more than just code that flags suspicious activity. Here are the three essential components you should consider:

1. A core service/engine that is constantly running and detecting fraud. You can make it event based and send alerts out real time. You definitely want to store results in a database (or any form of datastore). This is a service that should run real time and periodically.
2. A report, interaction service that can send reports to the risk team as well as mechanisms to display and act on those alerts. This can include a UI dashboard. It can pull information from the database where the first service/component stores information.
3. Develop a mechanism for other software systems/services to get guidance from your fraud detection systems in-order to act on specific threats. For example, oAuth and other systems need to be able to terminate access keys to fraudulent actors trying to access your APIs. This goes for transaction processing systems as well. Getting guidance from a fraud service will help to prevent certain users from making transactions or creating accounts.

Types of data to collect

You must collect as much information as possible about all aspects of your business in order to get a holistic picture. Fraud detection is a composite problem, so you need to look at the sum of multiple features and factors.

Once you have a good data collection and storage mechanism, you can start looking for signals, patterns, and anomalies that can help detect threats at a composite level. Data can include everything from meta information about users to system information. Here are some primary data points you’ll need to collect.

Location information; How to Store:

• Ideally this information should be stored as latitude and longitude.
• IP addresses; While useful, IP addresses when converted to longitude/latitude can include range uncertainty. Furthermore, IP addresses can be masked through VPN services, so they do not reflect the actual location

Location information; What to collect :

• Location of customers at the time of purchases or when using products
• If you are facilitating a market place, then the location of your merchants
• Home addresses of customers
• Address associated with the credit card or payment instrument

Activity Peaks and Rate of Transaction Volume
Not only should you look at individual transaction numbers (high purchase numbers etc), but also the aggregate rate and activity over time.

Failed transactions
In addition to monitoring successful transactions and activity, you also need to monitor failures. For example, are there credit card failures on certain purchases or transactions?

Chargebacks
If you are in credit card processing, it’s crucial to monitor chargebacks. These types of transactions can offer critical insights that flag fraudulent activity.

Product Usage Data
Product usage data includes looking at how to users interact with your interface. For example, consider features such as button clicks, the type of data entered in forms, time spent on screen, and user screen to screen transition.

What KPIS should I keep track of?

There are four important KPIs outside of core engineering and system metrics — True Positives (TP), False Positives (FP), True Negatives (TN), False Negatives(FN).

• True Positive (TP): Real Fraudulent activity detected.
  Your system has correctly flagged a fraudulent transaction or activity; This can be validated and vetted by your risk monitoring team. This is also why you need a way to get feedback on the output of your system
• False Positives (FP): A Legitimate transaction incorrectly detected as a fraudulent.
  The vast majority of what your system will be flagging are false positives. In most cases, business activity will not be fraudulent. So this is okay. Keep in mind that looking for fraud is finding a needle in a haystack. However you want to keep this to a minimum so you do not create fatigue in your fraud monitoring team.
• True Negatives (TN): Correctly identified non-fraudulent activities.
  This represents the number of transactions that were correctly tagged as not fraud.
• False Negatives(FN): Fraudulent transactions not tagged as fraud.
  The FN metric is critical and needs aggressive monitoring. You need to actively work towards bringing this down to zero. Every time there is a false negative, perform an RCA and implement steps to improve your fraud detection algorithms as soon as possible.

In addition to this, you need to keep track of various engineering and system level KPIs, for example; system efficiency metrics, how long does it take for each algorithm to calculate? What is the delta between an event occurring and when it was first flagged? How long does it take for each run? Keep track of crash metrics, bugs reported as well.

Who Needs System Access?? Why keep it separate?

Keep the repo secure, the services that run the core engine and also data store must be isolated from regular business systems. Only give access to the team working on this system. Starting this early is good as it will help when you go through compliance — especially PCI Level 1 compliance.

Generic systems architecture of a fraud monitoring system

If you’re an engineering leader trying to figure out what programming language on which to build your fraud system, Python is the most effective. It provides easy access to machine learning projects, and most data scientists and data engineers work out of Python.

Secondly, start with a simple rule based system and then scale to a machine learning based system. For example, consider implementing the following:

• A core service that creates features.This can then be fed into the actual fraud service. Which will most likely be a rule-based decision making system or utilizes machine learning/AI.
• A service that can display alerts to risk management team
• A service that can provide guidance to other services, like payment services, merchant onboarding.

Remember that fraud detection is a composite problem. You will need to bring multiple features together to get a holistic picture of potential activity that might be anomalous.

What is V1?

Build simple threshold rule based algorithms as a start. If something is above a certain value/threshold. Then alert, send a flag. You can calculate these thresholds using basic statistics. This gives you basic protection while you work on more complex solutions. With this you can also start monitoring TPs, FPs, FNs and other key metrics.

What is V2?

Once you have a basic rule based system, start building systems that can analyze these features and thresholds periodically. This makes your system more adaptive. That way your thresholds are not static and you are starting to set the stage for feature analysis and more adaptive algorithms

What is V3?
You have a dynamic system and have started mechanisms to analyze various features (such as distance based metrics etc.) Start using more advanced algorithms for threat detection. You can utilize a combination of machine learning, cluster algorithms or a neural network, or a polynomial algorithm that can calculate a threat score.

UI/UX of Dashboards for Risk Management Teams:

In most business cases the vast majority of transaction activity will be False Positives (FPs). Hence you will need to train your risk team to shift through them.

The reports and dashboards examined by your risk management team should include some guidance on why it was flagged. For example, they need to include a general reference to a feature or set of features that might have triggered the alert. This way, they have a general area to focus on. One example might be “unusual user activity” or “unusual purchase activity”. Otherwise, your team will get fatigue-bias and might allow illegitimate activity to pass through.

How to determine if fraud or not

There will always be an element of subjectivity when it comes to determining fraud. Whether you use a rule based system, a neural network, or a machine learning algorithm, you will end up with a system that returns a normalized score. Based on this score, set a threshold which will classify as Fraud or Not Fraud. The goal is to maximize TP, reduce FP and FNs.

Conclusion

Start small, do basic statistical analysis on the last 90 days of activity. Then build a basic algorithm that sends alerts and some guidance for a risk team member to review if those thresholds are violated. Then start focusing on more complex algorithms and machine learning models. Constantly track the rate of TP, FP, FN. Finally constantly iterate.