Global Ratings Fail, By Design, But Some Countries Benefit

AML rating system fundamentally flawed, harms some countries, and lets others ‘dial up’ better scores on demand

Dr Ron Pol
Mar 11, 2020 · 6 min read
RF Pol,

This week saw a new milestone. The Paris-based Financial Action Task Force (FATF) just released the 100th country-level report (Mali) assessing the effectiveness of anti-money laundering regimes.

This article explains why the global rating system fails to evaluate effectiveness in any meaningful way. But low ratings nonetheless harm some countries, whilst others have learned some of the secrets for using the system’s design flaws to ‘dial-up’ higher scores.

Effectiveness disconnect

Despite extensive, intrusive, and costly, evaluation processes, a trio of leading professors lamented the reality:

“there has been minimal effort at [anti-money laundering] evaluation…in the sense in which evaluation is generally understood [in] public policy and social science,…namely how well an intervention does in achieving its goals.”

Mostly measuring effort, activity, and processes (wrongly labeled ‘outcomes’), the rating system fails to assess outcomes as generally understood as the impact and effect of anti-money laundering activities.

Outputs mislabeled ‘outcomes’

“…measure ‘outputs’ rather than ‘outcomes’, what the organisation does, rather than what, if anything, it achieves”.

Seeking to overcome such limitations, the emergent post-NPM environment is often characterized as refocusing from efficiency to effectiveness, and from outputs to outcomes.

However, NPM’s target mentality remains deeply entrenched in many policy areas and countries (eg Health outcomes are what matter, not easy targets), with metrics often re-labeled ‘outcomes’.

Likewise, FATF’s “effectiveness” framework based on “outcomes”. Using the language of outcomes largely absent its scientific grounding means that official anti-money laundering evaluations remain functionally rooted in the past, with superficial NPM-era metrics re-badged with ‘outcomes’ labels.

Defining and measuring outcomes can, however, be hard, and differences between ‘outputs’ and ‘outcomes’ difficult to distinguish, so it’s useful to ground discussion with definitional clarity.

Outcomes define policy effectiveness

However, unlike the sequential flow in green (controlled by managers), social and economic outcomes intended by leaders don’t necessarily flow directly.

Results chain

Worse, if ‘outputs’ are wrongly labelled ‘outcomes’, they might be achieved, but focusing on the green part of the diagram while thinking we’re looking at the blue means the real outcomes may never be met.

Despite its ‘effectiveness’ and ‘outcomes’ terminology, FATF’s ratings inadvertently extend the checklist-mentality of its decades-long output-oriented technical compliance framework into the new ‘outcome’ measures.

Unrealizable objectives are not outcomes

Processes and activities are not outcomes either

In a typical results-chain they are not even outputs, let alone outcomes as they purport. They focus on processes, not the impact, effect or consequences of anti-money laundering activity.

Overall, assessors rate countries according to easy-to-measure metrics which may have an impact on laundering, crime, or the harms from crime. Or not. No-one knows. The system doesn’t measure any of the things intended when the G7 nations established FATF in 1989/1990.

This can be demonstrated in any ‘evaluation’. For example, “immediate outcome” 7 (IO7) in Australia’s assessment.

What do prosecutions reveal?

Australia’s FATF effectiveness ratings

A hypothetical example illustrates problems with that idea, and why it matters.

Say Australia prosecutes five percent of all serious profit-motivated crime, but only some attracts concurrent money laundering charges. The next year, police detect the same proportion and charge all offenders with money laundering as well as predicate offenses. Australia’s FATF effectiveness rating would increase, as assessors anticipate, likely to the highest rating.

Now consider an alternative. New initiatives advancing Australia’s main objective (“to disrupt and deter predicate crime”) allow authorities to intercept ten percent of profit-motivated crime but, with prosecutors overwhelmed, no concurrent money laundering charges are laid.

Australia’s ‘effectiveness’ rating on this count might fall to the lowest score. Bizarrely, detecting and prosecuting twice as much serious crime might result in failure on this so-called ‘outcome’ measure of ‘effectiveness’.

Before concluding, it’s worth noting an unintended, albeit obvious, consequence of such a rating system: it can be used to advantage.

Using the system for better ratings

Evidence of ‘gaming’ the system first appeared in re-evaluation ‘follow-ups’, mostly prompted, apparently, by political leaders turning to more effective strategies after low ratings when officials slavishly followed in-group doctrine. These strategies now appear in some ‘mutual evaluations’.

But, whatever the strategy mix between doctrinal and pragmatic, the statistical disparity is stark. The proportion of upgraded technical compliance ratings ranges between a dispiriting 0% and a stunning 78% increase. Some ratings are also notably more amenable to upgrade. My latest data, updated daily, reveals 15 rating measures up to 6–10 times more likely to be upgraded, and another nine at least three times as likely as the least improved measure.

Despite the dawning realization of ‘ratings on order’, FATF’s new ‘effectiveness’ framework remains, nonetheless, largely incapable of assessing effectiveness in any meaningful way.

But that’s not to say that the new rating regime hasn’t had some impact.

Whither FATF’s shiny new ‘effectiveness’ cloak?

“…may not have worked in eliminating money laundering, but it has worked as a strategy for globalizing an extremely expensive new regulatory order.”

Likewise, adding a new set of ‘effectiveness’ compliance measures to a global anti-money laundering experiment “almost completely ineffective” may not have reduced terrorism or serious profit-motivated crime, but it has worked as a strategy for globalizing more complexity and cost into an already extremely expensive regulatory order.

To better connect with the business of criminal justice goals, rather than the ‘busy-ness’ of activities with no clear line of sight to meaningful objectives, it may be time to reexamine the ruler against which effectiveness is measured.

— — —

What do you think?

Please engage, comment, share, connect, or follow to be notified about upcoming articles.

— — —

Primary source: Anti-money laundering effectiveness: Assessing outcomes or ticking boxes? Journal of Money Laundering Control

Read more in this series

  1. Saying It as You See it: How to Lose Friends and Infuriate People
  2. Anti-money Laundering “Almost Completely Ineffective.” Why it Harms us All
  3. Global Ratings Designed to Fail, But Some Countries Benefit

The Capital

A publishing platform for professionals now available on

Sign up for Try The Capital Platform at

By The Capital

Head over to, sign up and publish your first article today! Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Dr Ron Pol

Written by

Outcomes science, applied to life. Current main focus AML/CFT. Outcomes architect, loves engaging with open minds. See:

The Capital

A publishing platform for professionals in business, finance, and tech

Dr Ron Pol

Written by

Outcomes science, applied to life. Current main focus AML/CFT. Outcomes architect, loves engaging with open minds. See:

The Capital

A publishing platform for professionals in business, finance, and tech

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store