A critical examination of the State of Aadhaar 2018 report by IDinsight
The State of Aadhaar Report (SOAR; henceforth “Report”) by IDinsight (funded by Omidyar Network) was released on 17th May 2018. Herewith we examine the claims and recommendations in the Report to determine if it has indeed added new insights and nuances to the debate on Aadhaar.
Claim 1: Most felt it was easy to enrol
There are a few data points the survey arrives at:
- In Andhra Pradesh, 9% of those surveyed paid a fee to enrol.
- In West Bengal, 5.7% paid a fee.
- In Rajasthan, 24% paid a fee.
- 1% of surveyed households across all states were turned back because of enrolment errors, of which 30% (or 0.3% of the surveyed households), had biometric capture quality errors.
- 9% in Andhra Pradesh, 9.8% in Rajasthan, and 11.6% in West Bengal found enrolment difficult.
- Many residents reported facing inconveniences such as failed enrolment or the charging of unnecessary fees.
Bribery (unauthorized fees)
Let us just analyze the problem of bribery first. Bribery has two distinct components:
- Frequency of occurrence
- Monetary value of the demand
The monetary value was not asked in the survey. We do not know why such an obvious and important question was missed.
If we assume an average bribe of ₹100 and take the average of the bribery demands from the three states
(9%+24%+5.7%)/3 ≈ 13%, that suggests 13% of 120 crore (1.2 billion residents) paid ₹1,560 crores in bribes, a non-trivial sum.
Further, the survey did not ask the following questions:
- How often were the enrolments of genuine beneficiaries rejected after they were issued an Enrolment ID (EID)?
- What is the average time they spent in the enrolment queue (including repeat visits)?
- What is the average number of work days they lost in the process of enrolling?
These questions are important as it allows us to understand the cost the project imposed on residents. However, no such attempts were made, even though IDinsight’s funders seem to think that the cost is just about ₹9,000 crores, when in reality if all such factors are considered it could be as high as ₹35,000 crores.
The Aadhaar enrolment infrastructure has largely collapsed from 2016 onwards, because of rampant bribery and UIDAI’s subsequent decision to shut down 50,000 private operators and even government entities like CSC. This resulted in enrolment failures accelerating, as shown below (see link for calculations).
Hence the questionnaire should have contained the following questions:
- Have you enrolled for an Aadhaar?
- What was the time period during which you had enrolled?
- How often were you rejected (OR) asked to come back again?
- Did you only attempt to enrol in government offices, banks or post offices?
- Did you attempt to enrol in other places (private shops) after July 2017?
The last question is of supreme importance because of rampant use of cracked ECMP (Enrolment Client Multi-Platform) software:
India's ambitious digital ID project faces new security nightmare
A slew of complaints and alerts to UIDAI reveal that operators can bypass biometric and geo-location safeguards to…
The difficulty faced by residents in AP (9%), Rajasthan (9.8%) and WB (11.6%), when scaled to a national level will be around 10.8–13.92 crore (108–132.9 million) “genuine people”. By only pointing out these lower percentage figures without projecting to a national level, the survey made a crucial omission. Further it also failed to distinguish between enrolment failures after submission to the CIDR (for deduplication and capture quality control) and before submission (rejected by the enrolment software and measured as 0.3%), but still went ahead and said “exclusion from Aadhaar is not only low, but also not systematically biased against the poor or vulnerable”.
It also ignored critical questions on enrolment packets that were never submitted to the CIDR. Given that cracked versions of ECMP allowed standalone generation of an EID, and given the widespread use of physical copies of Aadhaar card or EID slip for availing various services, a retained enrolment packet of a head of family is priceless for committing fraud.
Claim 2: Aadhaar updates are painful
The survey only focuses on demographic errors between the Voter ID database and the Aadhaar database, and produces a top line error rate of 8%, 4.8%, 12.2%. However it conveniently forgets its own data source from UIDAI about how wide these errors are: in the range of 20–35%.
Comparing with Voter ID rolls for demographic errors is an exercise in obfuscation, because linking Aadhaar with Voter IDs is not mandatory (yet), but linking for Aadhaar Act Section 7 benefits is. That means every such demographic error is a denial of service for beneficiaries in all the schemes covered under Section 7.
In effect, the survey chose a specific methodology that is particularly irrelevant for measuring exclusion, but had the advantage of showing a reduced headline error rate (8–12.2%) than the already acknowledged error rate (20–35%).
The Report’s recommendation on establishing grievance mechanisms for updates
The Report fails to mention the widespread use of cracked ECMP software, which was created for the purpose of solving the update problem. It also fails to mention the Gujarat biometric fraud case, where bank officials’ fingerprints were used for making updates.
Here we have a curious recommendation for a problem that is widely known to exist, and for which solutions already exist (albeit illegal ones), but which the UIDAI continues to deny — and probably cannot fix because of the design choices they made. Yet, none of these are discussed or even mentioned in the report.
Claim 4: Photocopies are often used for opening bank accounts
The survey confirms an existing observation (here, here and here) that Aadhaar authentication is costly at scale, and use of photocopies is a means to avoid the authentication cost. Photocopies of Aadhaar cards, accepted for opening bank accounts, are also a primary misuse vector for bank fraud.
Aadhaar Fraud is Not Only Real, But is Worth More Closely Examining - The Wire
An online examination of publicly reported incidents shows that contrary to its proponents' claims, Aadhaar has indeed…
The Report curiously does not acknowledge or even discuss the possibilities of misuse, while the evidence existed as far back as October 2015. Besides, it again skips important questions on Aadhaar linking with banks, such as:
- Was your account frozen for operations because of failure to link Aadhaar?
- Was your account frozen for operations because of Aadhaar linking errors (demographic or biometric mismatch)?
Or for the survey designer:
- What percentage of those surveyed opened bank accounts using other identity mechanisms?
- Were they given a choice to use alternate IDs, or were they asked for Aadhaar under threat of service denial without it?
- Were they given sufficient information that Aadhaar for eKYC is very different from services like Aadhaar-enabled withdrawal, or subsidy reception through Aadhaar-linked bank accounts?
- How many of them were impacted by the common problem of receiving subsidies in accounts whose existence they were not even aware of? (For example, the Airtel Payments Bank fraud with 31 lakh (310k) unaware account holders.)
- How many of them paid an Aadhaar operator to get a printout of their Aadhaar details? (The ECMP software had major vulnerabilities in this area.)
- How many of them paid more than the approved fees for getting an Aadhaar printout?
- How many of them had bank accounts before Aadhaar? (This question is essential to examine if Direct Benefit Transfer helped increase financial inclusion.)
The Report’s recommendation on push towards mobile-based financial services
While the survey asks about account usage and DBT (Direct Benefit Transfer), and notes the poor penetration of Micro ATMs and Banking Correspondents network, it avoids getting into detail on the role of Aadhaar in DBT.
However, it recommends newer delivery channels (mobile financial services) without considering what works or doesn’t in the current scheme of things.
The benefits of APBS in disbursing DBTs compared with other electronic payment mechanisms, such as National Electronics Funds Transfer System (NEFT), are unclear. (Page 21 of the State of Aadhaar Report)
- It enables consolidated routing of all subsidy payments to a single bank account, increasing the chances of higher bank balance.
- It tags all subsidy payments to an individual, allowing data systems to enable credit scoring and other such profiling of the under served.
The claims of Aadhaar for financial inclusion in DBT may be tested with two data sources that are unfortunately not available to the public:
- PMJDY overdraft data (no public data after Dec 2016)
- Credit enabled through data collected from APBS for DBT
Switching those who use Micro-ATMs to mobile banking is a non-starter, since the primary purpose of Micro-ATMs is withdrawal with an Aadhaar number and fingerprint. A phone can’t dispense cash.
The growing incidence of cybercrimes is an indicator that even an educated population will fall prey. It is unclear if the Report’s recommendation is suggesting that Micro-ATMs have failed. It is also unclear if the recommendation has considered cybercrime risks, and whether benefits exceed harms.
The sustainability of Banking Correspondents and Micro ATMs, the cost of cash (to banks, agents and beneficiaries), access to cash, access to payment systems, and payment infrastructure penetration, should all be understood deeply both at the systemic level and in the reflections of people using them, before proposing a new delivery channel. It is unfortunate that the Report despite being based on a survey makes such an unsubstantiated recommendation.
Here are some questions that could have been asked to respondents to understand perceptions around digital financial services:
- Do you have a (RuPay) debit card?
- Have you asked your bank for a debit card?
- Are you aware of the overdraft facility of PMJDY? If so, have you used it?
- Are you aware of, or have used mobile-based wallets, or payment modes such as USSD? If so, any feedback on the experience, cost, and convenience over AePS/cash?
Claim 5: PDS and biometric authentication
The Report claims that biometric authentication mostly works, using the following dataset:
Surprisingly, it fails to ask the following questions:
- What is an “attempt” to get ration? Is it multiple rounds of placing a finger on the scanner, multiple visits on the same day, or visits on multiple days?
- What is the cost of such increased transaction time to a beneficiary? Did they lose a day’s wage?
It also fails to explore an even more interesting question: What is the number of people who are impacted because of Aadhaar if the figures are scaled up nationally?
Claim 6: Aadhaar and Public Distribution System
The Report makes the following claims w.r.t. Aadhaar and issues in PDS.
Connectivity issues with a POS (Point of Sale) machine and the failure to authenticate any member of a household (since rations are issued to households, not individuals) are side effects of the dependence on online biometric authentication. Surprisingly, they have not been considered in the headline figure of 2.2% exclusion, as it increases the total number of individuals impacted from 12 lakh(Page 9, SOA report) to 19 lakh (120k to 190k).
Further, the Report also fails to point out that the primary claim of Aadhaar is that dealers can’t route rations to fake/ghost beneficiaries while claiming non-availability of rations. Since dealers can continue to maintain that rations are not available, to justify dispensing reduced quantities, it means the raison d’être of Aadhaar in PDS has been severely weakened.
In effect, Aadhaar has not solved any of the existing problems in PDS, but has added new forms of exclusion. Surprisingly, in spite of the data showing that this is indeed the case, the Report spins it as:
Exclusion from food ration (PDS) due to Aadhaar-related factors is significant, but lower than non-Aadhaar factors
Claim 7: Majority of Indians approve linking of Aadhaar to services
Policy decisions are not made via opinion polls with a sample size of 3000. Surveys are not a referendum either. Further, the Report may have confused stated preferences with revealed preferences, a novice error in econometrics. For instance, consider the following questionnaire:
- Do you support Aadhaar linking with PDS?
- (If yes to the above), Given the fact that some people die because of exclusion, do you support Aadhaar linking with PDS?
- (If no to the above), What are the primary reasons for not supporting it?
The answers will be very different based on the questions asked. Participants answers are only valuable if they have skin in the game, and the only way to know would be if they, or someone close to them, has faced issues because of mandatory Aadhaar linking with services.
It is difficult to ascertain if the survey does take this into account at all. The only way to know it for sure, would be to share the raw questionnaire, results and also the reasons, why those particular questions were chosen.
Claim 8: Aadhaar, fraud, mobile linking approval
Approximately 87 percent of people approved of the government’s mandatory use of Aadhaar, while 76.9 percent approved Aadhaar’s mandatory use by the private sector.
While the report claims approval for mobile linking, little has been done to understand from people on the ground about awareness of identity theft, frauds and other risks enabled by Aadhaar, all of which gets exponentially higher when linked with mobile (especially when awareness of OTP as authentication factor with the surveyed population is very low).
The Report makes no mention of surveying people about knowledge of frauds and the precautions they generally take, while making a sweeping statement on building strong data protection measures and responsible consent mechanisms regarding the gathering or sharing individuals’ data.
If the participants were made aware of the fact that the largest DBT subsidy theft, amounting to ₹190 crore, was the result of enforced mobile linking, perhaps their answers would have been very different. Here is a partial list of Aadhaar-enabled fraud collated from Google search results.
The Missing document library
State of Aadhaar’s document library on their website had always been a good reference on various aspects of the Aadhaar project such as notifications, acts, press releases etc. So it is curious it has been taken down. See archived version here.
It makes one wonder if SOA is following the footsteps of UIDAI, which has a history of deleting documentary evidence from its website when faced with criticism, without understanding that the Internet never forgets. Perhaps the true state of Aadhaar is in the hiding of inconvenient evidence?
Let us recap the flaws in the State of Aadhaar Report (SOAR) so far:
- It skipped mentioning existing evidence on enrolment failures, bribery and cracked ECMP software, which compromised Aadhaar data quality for years, but instead presented a survey result without context.
- It ignored UIDAI’s own study, available in the Report’s repository, on how much worse demographic data quality errors are.
- It merely reiterated what is already well known — that Aadhaar photocopies are used at a very wide scale — but failed to mention how UIDAI itself warned physical copies are insufficient for authentication, or how they are linked to banking fraud on the vulnerable.
- It made a recommendation to push towards mobile banking without any basis, whether theoretical or evidence-based.
- It ignored asking crucial questions on biometric authentication failure.
- It failed to point out that the rationale for Aadhaar in PDS — as originally claimed by the government — is nullified by the data obtained through the survey.
- It may have made a novice error of using a flawed survey methodology.
The report also fails to acknowledge existing discussions about the thoroughly debunked savings claim by the government on Aadhaar. It ignores a long running debate with evidence that on LPG, the fiscal impact of using Aadhaar is negative for the exchequer, and instead makes the strange claim that the evidence on Aadhaar’s specific contribution to deletion of non-genuine beneficiaries is not available.
In conclusion, the quality of the report is disappointing. It presents ignorance as insight and errors as nuance. The fact that it has been funded by Omidyar Network only makes the disappointment deeper. Clearly, good quality scholarship and funding are not correlated.