Machine Learning and Mobile Data Improves Aid Delivery in Togo

This week, CEGA and IPA researchers released a working paper with results from a groundbreaking project in Togo that distributed emergency humanitarian aid using machine learning and mobile phone data. The working paper finds that the machine learning approach reduced the number of people who didn’t get benefits, but should have, by 4–21% relative to the alternative geographic targeting approaches being considered by the government at the time.

This post is written by Senior Program Associate Anya Marchenko.

Game Akoko, a cash transfer recipient in Togo. Credit: GiveDirectly

Why targeting, why Togo?

When COVID-19 lockdowns began, the Togolese government launched an emergency social protection program, “Novissi,” to distribute cash to its poorest citizens. In the absence of a traditional social registry to determine eligibility for the program, the government needed an accurate and fast way to target the cash to those who needed it most.

A team of researchers from CEGA and IPA brought their expertise to help. Since April 2020, Joshua Blumenstock (CEGA Faculty Co-Director), Emily Aiken (UC Berkeley), Suzanne Bellue (University of Mannheim), Dean Karlan (Northwestern, IPA) and Chris Udry (Northwestern) have been helping the Togolese government target cash transfers using a machine learning (ML) and mobile data approach (the approach is described in more detail here). Though the program was successfully implemented — over 140,000 Togolese have received $10 million in cash, with our partner GiveDirectly distributing payments — the question remains: Did the ML approach effectively put cash in the hands of those who needed it most? And how did it perform relative to other, more common approaches to targeting cash?

This figure compares different approaches to targeting cash in Togo. The machine learning approach used in Togo is represented by the green bars. Red bars represent other targeting approaches available to the government at the time, while blue bars represent infeasible alternatives. Darker bars indicate recall and precision (left axis); lighter bars indicate Area Under Curve (right axis). Recall measures the fraction of individuals the algorithm identified as eligible that are actually eligible. Precision measures the percentage of eligible people that the algorithm identified. Area Under Curve measures the relationship of true positives to false positives (higher numbers are better, an algorithm whose predictions are 100% correct has an AUC of 1.0).

Evaluating ML-based targeting of social assistance in Togo

One way to evaluate a targeting approach is to compare the people selected to those who would have been selected under a different approach. A targeting approach performs well if a high percentage of people it selects are actually poor (low errors of inclusion, aka high precision) and if a high percentage of the eligible poor are selected (low errors of exclusion, aka high recall).

The researchers compare their ML approach to the other targeting approaches that were available to the government at the time, as well as to several common ways of targeting that were not feasible for Togo.

So, how well did the ML approach perform relative to:

…the government’s outside option at the time?

Without ML, the government would most likely have targeted cash geographically, by sending cash to all people living within the poorest cantons. Relative to this, the machine learning approach reduced errors of exclusion by 4–21%. This means that the ML approach reduced the number of people who would have been incorrectly excluded from receiving benefits by 4–21%.

…a hypothetical method that requires a comprehensive social registry?

While no such registry exists in Togo, this comparison benchmarks against a common targeting method. Relative to using a hypothetical registry, the ML approach would have increased exclusion errors (the number of people who should’ve gotten benefits but didn’t) by 9–35%. However, this range includes the best possible proxy means test (i.e., using observable characteristics of Togolese individuals to estimate their income or consumption), which likely makes the hypothetical registry appear more accurate than a real-world registry.

…targeting based on mobile phone usage, without applying machine learning?

A straightforward approach the government could’ve used is targeting transfers to people with the lowest mobile phone expenditures. Intuitively, people who spend less on airtime and calls are more likely to be poor. However, this approach performs significantly worse than the ML-based approach; so while easier to implement, it increases targeting errors.

One growing concern in algorithm decision making is its potential to discriminate against vulnerable groups. To address these concerns, the authors look at whether the algorithm systematically excluded different demographic groups in Togo, relative to those groups’ true poverty rates.

They find that the ML approach does not result in women being systematically more likely to be incorrectly excluded from receiving benefits than men relative to alternative targeting methods (see figure below); nor does it result in people of different ethnic groups in Togo to be unfairly excluded. This parity holds across religions, age groups, or types of households.¹ However, the paper also finds that there is no targeting approach that would have achieved perfect demographic parity (i.e., where the proportion of a group that is targeted is the same as the proportion that is actually poor).

This figure shows the fairness of targeting for different demographic subgroups, showing the differences between ranking according to predicted wealth from the ML approach and ranking according to true wealth. Left-skewed bars indicate groups that are consistently under-ranked (less likely to receive benefits); right-skewed bars indicate groups that are consistently over-ranked (more likely to).

A rapid, cost-effective and supplemental tool

The authors conclude that their results — while heartening in the case of Togo — do not imply that ML and phone based targeting should replace traditional approaches reliant on proxy means tests or community-based targeting. Rather, similar to the ideas we outlined here, the authors emphasize that “these new methods provide a rapid and cost-effective supplement that may be most useful in crisis settings or in contexts where traditional data sources are incomplete or out of date.”

[1] An important note is that inclusion / exclusion may occur because of factors unrelated to the algorithm. Program awareness, registration success, and phone ownership are other sources of exclusion from the program. For example, in the authors’ survey, 53% of women and 79% of men report owning a mobile phone, which may cause differential impact by gender.