Global Ratings Fail, By Design, But Some Countries Benefit
AML rating system fundamentally flawed, harms some countries, and lets others ‘dial up’ better scores on demand
Last week, I explained why the global anti-money laundering system (costing us all trillions of dollars) is “almost completely ineffective.” Technology and other ‘solutions’ framed in a regulatory straight-jacket limit the success rate to a puny 0.1 percent impact on criminal finances.
This week saw a new milestone. The Paris-based Financial Action Task Force (FATF) just released the 100th country-level report (Mali) assessing the effectiveness of anti-money laundering regimes.
This article explains why the global rating system fails to evaluate effectiveness in any meaningful way. But low ratings nonetheless harm some countries, whilst others have learned some of the secrets for using the system’s design flaws to ‘dial-up’ higher scores.
Effectiveness disconnect
About half-way through the current round of assessing countries’ compliance with FATF standards, 100 full evaluations and 82 follow-up reports released between 5 December 2014 and 10 March 2020 generated 21,709 pages of analysis, 5,092 ratings, and 570 re-ratings.
Despite extensive, intrusive, and costly, evaluation processes, a trio of leading professors lamented the reality:
“there has been minimal effort at [anti-money laundering] evaluation…in the sense in which evaluation is generally understood [in] public policy and social science,…namely how well an intervention does in achieving its goals.”
Mostly measuring effort, activity, and processes (wrongly labeled ‘outcomes’), the rating system fails to assess outcomes as generally understood as the impact and effect of anti-money laundering activities.
Outputs mislabeled ‘outcomes’
Possibly contributing to design faults, an ethos called “new public management” dominated public administration in many countries from the 1980s. NPM focused on efficiency and performance targets. Good things, you might think. However, as described by Professor Garland, such targets tended to:
“…measure ‘outputs’ rather than ‘outcomes’, what the organisation does, rather than what, if anything, it achieves”.
Seeking to overcome such limitations, the emergent post-NPM environment is often characterized as refocusing from efficiency to effectiveness, and from outputs to outcomes.
However, NPM’s target mentality remains deeply entrenched in many policy areas and countries (eg Health outcomes are what matter, not easy targets), with metrics often re-labeled ‘outcomes’.
Likewise, FATF’s “effectiveness” framework based on “outcomes”. Using the language of outcomes largely absent its scientific grounding means that official anti-money laundering evaluations remain functionally rooted in the past, with superficial NPM-era metrics re-badged with ‘outcomes’ labels.
Defining and measuring outcomes can, however, be hard, and differences between ‘outputs’ and ‘outcomes’ difficult to distinguish, so it’s useful to ground discussion with definitional clarity.
Outcomes define policy effectiveness
In simple terms, outcomes are the result of inputs, activities, outputs and external influences.
However, unlike the sequential flow in green (controlled by managers), social and economic outcomes intended by leaders don’t necessarily flow directly.
Worse, if ‘outputs’ are wrongly labelled ‘outcomes’, they might be achieved, but focusing on the green part of the diagram while thinking we’re looking at the blue means the real outcomes may never be met.
Despite its ‘effectiveness’ and ‘outcomes’ terminology, FATF’s ratings inadvertently extend the checklist-mentality of its decades-long output-oriented technical compliance framework into the new ‘outcome’ measures.
Unrealizable objectives are not outcomes
For example, “immediate outcomes” 10 and 11 (IO10 and IO11) and the first part of IO5 are lofty goals. For terrorists to be prevented from raising, moving or using funds, and shell companies prevented from misuse, are laudable aspirations. It is, however, impossible to assess their achievement. Outcome measures require specificity. Loose, aspirational terms with no clear statement of impact and effect are virtually meaningless.
Processes and activities are not outcomes either
The remaining “immediate outcomes” are not outcomes either. For money laundering risks to be understood (IO1), and international cooperation to deliver appropriate information (IO2) describe mere activities. Likewise, for supervisors to supervise, monitor and regulate (IO3) and firms to apply anti-money laundering measures (IO4). For beneficial ownership information to be available (IO5) and authorities to use financial information (IO6) also describe processes and activities.
In a typical results-chain they are not even outputs, let alone outcomes as they purport. They focus on processes, not the impact, effect or consequences of anti-money laundering activity.
Overall, assessors rate countries according to easy-to-measure metrics which may have an impact on laundering, crime, or the harms from crime. Or not. No-one knows. The system doesn’t measure any of the things intended when the G7 nations established FATF in 1989/1990.
This can be demonstrated in any ‘evaluation’. For example, “immediate outcome” 7 (IO7) in Australia’s assessment.
What do prosecutions reveal?
FATF assessors said it “should be relatively easy” for Australia to achieve a ‘high’ rating for IO7 (money laundering investigations/prosecutions). With “double-charging challenges” resolved (which had limited the ability to sustain charges for money laundering as well as the underlying ‘predicate’ offense), it’s now easier to add such charges. More money laundering prosecutions would clearly boost Australia’s lowly “moderate” score.
A hypothetical example illustrates problems with that idea, and why it matters.
Say Australia prosecutes five percent of all serious profit-motivated crime, but only some attracts concurrent money laundering charges. The next year, police detect the same proportion and charge all offenders with money laundering as well as predicate offenses. Australia’s FATF effectiveness rating would increase, as assessors anticipate, likely to the highest rating.
Now consider an alternative. New initiatives advancing Australia’s main objective (“to disrupt and deter predicate crime”) allow authorities to intercept ten percent of profit-motivated crime but, with prosecutors overwhelmed, no concurrent money laundering charges are laid.
Australia’s ‘effectiveness’ rating on this count might fall to the lowest score. Bizarrely, detecting and prosecuting twice as much serious crime might result in failure on this so-called ‘outcome’ measure of ‘effectiveness’.
Before concluding, it’s worth noting an unintended, albeit obvious, consequence of such a rating system: it can be used to advantage.
Using the system for better ratings
Savvy officials in some countries began learning how to ‘dial up’ better ratings across all 51 metrics, largely irrespective the ‘real’ risk or impact of money laundering controls; which, ironically, the system doesn’t measure.
Evidence of ‘gaming’ the system first appeared in re-evaluation ‘follow-ups’, mostly prompted, apparently, by political leaders turning to more effective strategies after low ratings when officials slavishly followed in-group doctrine. These strategies now appear in some ‘mutual evaluations’.
But, whatever the strategy mix between doctrinal and pragmatic, the statistical disparity is stark. The proportion of upgraded technical compliance ratings ranges between a dispiriting 0% and a stunning 78% increase. Some ratings are also notably more amenable to upgrade. My latest data, updated daily, reveals 15 rating measures up to 6–10 times more likely to be upgraded, and another nine at least three times as likely as the least improved measure.
Despite the dawning realization of ‘ratings on order’, FATF’s new ‘effectiveness’ framework remains, nonetheless, largely incapable of assessing effectiveness in any meaningful way.
But that’s not to say that the new rating regime hasn’t had some impact.
Whither FATF’s shiny new ‘effectiveness’ cloak?
In Global Business Regulation, Braithwaite and Drahos observed that the global strategy of establishing a “community of money-laundering compliance”…
“…may not have worked in eliminating money laundering, but it has worked as a strategy for globalizing an extremely expensive new regulatory order.”
Likewise, adding a new set of ‘effectiveness’ compliance measures to a global anti-money laundering experiment “almost completely ineffective” may not have reduced terrorism or serious profit-motivated crime, but it has worked as a strategy for globalizing more complexity and cost into an already extremely expensive regulatory order.
To better connect with the business of criminal justice goals, rather than the ‘busy-ness’ of activities with no clear line of sight to meaningful objectives, it may be time to reexamine the ruler against which effectiveness is measured.
— — —
What do you think?
Please engage, comment, share, connect, or follow to be notified about upcoming articles.
— — —
Primary source: Anti-money laundering effectiveness: Assessing outcomes or ticking boxes? Journal of Money Laundering Control
Read more in this series…
