Conducting M&E in our New Reality
Ten years ago, while I was working for DARPA, I landed in Kabul, Afghanistan for the first time. This guy named Neilesh Shelat, from USAID, picked me up from the airport. It was my first exposure to the people of USAID, and the problems they try to solve all over the world. One of the most interesting problems I found was effective Monitoring and Evaluation in places where we have limited access. That appears to be the entire globe at the moment…
There has been no shortage of conventional and uncomfortable truths laid bare by the COVID-19 pandemic — whether it be the efficacy of holding in-person meetings, or how vitally important a local barber or salon really is. The same can be said about project monitoring and evaluation. How do development partners verify projects, assess the information environment, or simply know what is going on “on the ground” when expensive consultants are no longer allowed to travel and face-to-face interviews are no longer safe or feasible to conduct?
For decades, the discipline of M&E has been dependent on an individual’s ability to physically get to a site, rarely venturing past a capital city’s limits, to conduct a few interviews or focus groups, then extrapolate findings. In areas deemed insecure, third party monitoring (TPM) has been touted as an integral feature of project monitoring, purporting to enable real time data collection and course correction. The cost of survey design plus the operational costs (flights, per diem, lodging, etc.), in addition to the time-lag of data collection to analysis, often rendered conclusions outdated; and the finished product served as a contractually-compliant deliverable, rather than a tool to inform future programming.
Like most harsh realities that have been exposed by this pandemic, the issue of how funders assess, verify, and evaluate project implementation will need to be re-evaluated. The spigot of donor funds will not be turned off; and while the virtues of TPM initiatives have been extolled in many well-rehearsed talking points, donors will need to demand an answer to a basic question, ‘how exactly will you monitor your project?’
Fortunately, technologies have existed for decades to provide M&E services that are remote, scalable, cost efficient, sensitive to the realities of vulnerable populations, and more accurate than standard methods. My company, IST Research, recently conducted the following research in April/May 2020 — at the height of the pandemic:
- Over five days, received 300,000 responses from 40 countries in eight languages on a 25-question survey focused on medication use, health education and current health conditions related to COVID-19 (cost: $9,300.00).
- Over four days, surveyed 6,000 Overseas Filipino Workers (OFWs) located in Saudi Arabia, Kuwait, UAE, Qatar, Bahrain, and Oman to assess the impact of COVID-19 on transnational migrants (cost: $5,000.00).
- Over six days, received 10,300 responses from Ecuador, Brazil, Colombia, Peru, and Venezuela to assess the effects of COVID-19 on three vulnerable populations: people experiencing homelessness, those at risk of domestic violence, and migrant workers ($2,500.00).
How much would it normally cost to pay for consultants to cover multiple countries, targeting thousands of hard-to-reach and often stigmatized populations, in less than seven days…and in the middle of a pandemic with a worldwide prohibition on travel?
How do we do this?
The short answer is that we have leaned forward to adapt to technological and social changes that have been in the works for decades. A plethora of technologies exist to provide M&E services that are scalable, cost efficient, and more accurate than standard methods. Our users can easily direct and triangulate data collection from multiple sources to produce reliable findings. They can combine data from social media and internet sources such as stakeholder opinions from Twitter, Facebook, or Telegram; targeted responses from local surveys; and time-stamped photos from low-cost local monitors to provide a complete picture of a project’s status.
For the monitoring effort itself, the solution is currently in your hand…or in your pocket. Apps have been developed to support both text message surveys and remote data collection, with findings verifiable through GPS coordinates, IP addresses, timestamps and geotagged photos. Digital survey design and execution is adaptable, and systems such as ours support branched survey logic, which allows for clearer, more specific questions and responses. We built our engagement technologies to also function offline in areas with limited cell phone service, allowing monitors to collect information regardless of connectivity. In the many areas where literacy is an issue, interactive voice response is well-honed to help. Also, when data sharing (e.g., photos, survey data) becomes costly and difficult in areas of limited connectivity, a monitor can store data in a smartphone application that automatically uploads this content to servers when the smartphone is connected to WiFi or an internet hotspot, circumventing any issues associated with data sharing and connectivity.
For project design and data analysis, we use powerful cloud data processing & analysis capabilities built into our technology stack and employ experienced social and data scientists to examine and produce rigorous findings drawing from all available data. Our staff has a track record of devising innovative methods for obtaining hard-to-measure information and making sense of diverse data aggregated from many original sources. For survey development, we’ve employed previously used techniques like indirect questioning and vignette-based approaches to response calibration to elicit accurate responses concerning extremely sensitive topics, and developed additional “truth anchoring” techniques in-house. In addition, expert data scientists use methods such as topic modeling and cross-lingual sentiment analysis to identify key themes from publicly available online data (e.g., social media posts) and employ a variety of NLP, clustering, and classification techniques in the filtering & analysis of free-text and image-based survey responses. By leveraging multiple methods and data sources, forward-leaning organizations can ensure that M&E programs measure more than just basic statistics and narratives at a lower price point than these traditional approaches.
I’m sure I was sad when I realized that I could no longer rent a movie from my local Blockbuster video store — but I’ve moved on. As a community, we must face an uncomfortable truth that the brick-and-mortar model of hiring enumerators, training them for weeks as part of costly site visits, and relying heavily on face-to-face interviews, is outdated. This pandemic, rather than paralyzing our monitoring efforts, should be recognized as a long-overdue disruption in the way we further innovate to monitor last-mile programming, and how we can be more accountable to our funders.
Ryan Paterson is CEO of IST Research, a veteran-owned small business which provides tech-enabled answers to complex questions about hard to reach populations. Please contact me at info@istresearch.com if you have questions.
