Building a manual order review process with a focus on keeping an Order Decline Rate under 2%


In this quick How To I’ll break down the high-level crucial steps required to build a manual review team that maintains a low decline rate while supporting a low chargeback rate.

While the e-commerce CNP industry standards report a decline rate of 1% to 5% of orders to maintain an average chargeback rate of 0.7% to 0.8%, my team has been able to maintain an average decline rate of 2% or lower, while also maintaining an average chargeback rate of under 0.4% over the last two years. In addition, each month we only see a handful of unique users slip through with malicious fraud orders.

Don’t hesitate to reach out to me directly with questions or feedback at courtneyabode@gmail.com

Isolate a clear breakdown of the type of fraud you see

No two companies are the same, no two fraud strategies can be the same either.

The most important thing you can do as you start to grow a manual review team is to understand what type of fraud is specific to your company. You need to understand the breakdown between true malicious fraud, friendly fraud, and the ‘Grey Area Customers.’ You truly just cannot build a solid plan of attack until you know the deep characteristics behind your fraud adversaries.

The best way I’ve found to do this is by reviewing lost fraudulent disputes. Once you have this list, all you need is a few hours and an eye for detail to pick up on a few solid trends associated with each group contributing to your fraudulent disputes. That’s really the first step.

You just can’t catch ’em all

When we first started filtering out for suspicious orders, my team would often rejected what I call ‘Grey Area Customers.’ These are customers that have some fraudulent characteristics, but also some good characteristics. Digging into why our decline rate was so high during the 2015 holidays, I realized that it is not just due to a significant increase in fraudulent customers, but rather a natural bias on my Manual Review Team.

It’s here that we really learned that Fraud Analysts are often fundamentally risk averse individuals. These are people who would rather reject a ‘Grey Area Users’ knowing they might prevent fraud, than accept that suspicious order and risk an incorrect assessment.

So how do we pivot our team away from such a natural bias?

From the top down. You need to instill faith that the goal is NOT to catch all fraud, it’s just to catch as much as possible. Instilling this sense of confidence is crucial. My teams knows to look for key indicators of true fraud, and if these are missing or not found to be the majority, the process the order and move on. No harm, no foul if this order gets disputes later as long as the team can clearly walk me through their logic in deciding to process the order. It’s also important to quickly note that targeting to reduce ‘Friendly Fraud’ cannot and should not be done. Instead, you must focus on winning these disputes instead of preventing them.

Key Indicators for the good, the bad, and the Grey Area Customers

Going back to the first point — trend analysis — you cannot avoid it. At the end of the day, you will have to spend time digging into the disputes to isolate those key indicators of good and bad users. Once you have a couple solid indicators, you can easily teach your team to keep an eye on these. The further you develop your list of Key Fraudulent Indicators, the easier it becomes for your team to easily isolate that ever so blurry line between a Grey Area Customer and one that is actually highly likely to be fraudulent. This is up to you and should not be delegated until you have a firm process in place for analysis. Key indicators will not apply to a majority of users, you should look for a collection of smaller traits. My team maintains around 10 key indicators pointing to fraudulent activity, and if a user hits a few of those, we decline it.

Label, Label, and again, LABEL fraud team decision-making

You’re going to need to quickly isolate the top reasons orders are rejected. I promise that you will not find a good balance between healthy decline rate and a healthy dispute rate quite so fast. You’ll have to shave little pieces away week over week for a while, and that’s okay!

The labeling technique we started to use was one of the first ways I was able to quickly isolate that we were rejecting WAY too many grey area users. You’ll save a whole lot of time and headache is you can understand why your team is making their decisions using a uniform labeling system. This system should simply allow your team to quickly pick a couple labels letting you know why they made their decision.

Not only does this give you the ability to assess on the fly why orders are rejected, it also holds you accountable to developing a systematic process of review for your team. Remember those key indicators? Those should essentially become your labels. You’ll very quickly understand whether you’ve created clear key indicators for you team when you assess their labels. If you can’t figure out why orders are getting declined in a few minutes, you probably need to keep fine-tuning first your Key Indicators, then your manual review process.

My team now provides a weekly report with a full breakdown on all the users process and all the users declined. I review this weekly and make adjustments, digging in where I need. Label analysis needs to be very periodic, very logical, and very inclusive.

Don’t forget to think small

I so frequently speak with others in the industry who want these big, egregious fraud trends to target. That’s the thing… there really aren’t many other than the obvious such as ‘this user tried 20 different credit cards in 2 minutes’ rule.

The more you look for these big trends in your manual review, the higher your decline rate and false negative rate will be. With fraud, there aren’t many one-size fits all strategies. Fraudsters are smart, tricky, and they constantly develop towards the next new idea. Think small, think unique. I know that it sounds against the grain, but it couldn’t be more true. Once I see even a couple disputes come through with the same trait, I flag that immediately as a new trend to look for. On our team, we have these bigger rules such as X many orders placed in X timeframe, while also considering several Key indicators related to user characteristics. These characteristics are usually the smaller indicators. We couple these more unique indicators with the bigger, more obvious ones. Combining both strategies allows us to keep our malicious fraud at a handful of unique users per month.

Putting all this together to craft a unique solution for your fraud needs

To build an effective manual review system with a low decline rate, you have to slowly put the pieces together.

You start by understanding your data to learn what type of fraud you experience. From there, you decide which type of fraud you can prevent and narrow in on the key characteristics for that fraud type. By creating this list of key indicators, you create a logical judgement flow for a manual review team that tells them when to decline and when to accept an order. Track these team decisions using labels, then you can continue to improve their judgment and performance overtime. Lastly, remember to always try to isolate the smaller fraud trends that might only affect a handful of users — this is how you really begin to chip away at malicious fraud.

Ultimately, building my team took many months and, to this day, we still make weekly adjustments. Everything in fraud is ever-changing, and all you can do is adapt. This requires on-going analysis, constant reworking of your rules & key indicators based on new disputes. Lastly, your goal is usually to target malicious fraud. Anything else is likely to result in false negatives and a high decline rate without decreasing your dispute rate. Truth be told, a lot of malicious fraud is pretty obvious! You just need to spend the time learning the malicious fraud you see on your unique platform. My biggest recommendation is to chip away, targeting 1–5% of your malicious fraud at a time.

Find this and more posts on opstales.io.

Show your support

Clapping shows how much you appreciated Courtney Bode’s story.