What’s in an Algorithm? Lessons from MeasureDev 2024
In this post, Sean Luna McAdams (Senior Manager for Conflict and Digital Development at CEGA) summarizes insights from our recent Measuring Development (MeasureDev) conference, co-hosted with the World Bank’s Development Impact (DIME) and Data groups and the University of Chicago’s Development Innovation Lab (DIL) in Washington, D.C. MeasureDev 2024 showcased efforts to improve and expand responsible data infrastructure in low- and middle-income countries (LMICs), facilitate the development of a new generation of ethical AI tools, and optimize generative AI (GenAI) tools like LLMs for social impact.
Co-hosted by CEGA, the World Bank, and the University of Chicago the tenth annual Measuring Development (MeasureDev) conference explored the subtext of the data and computations driving novel artificial intelligence (AI) tools in global development. During a keynote address, Dan Björkegren (Columbia University) stressed that “Algorithms necessarily encode values” in the responses they give to user prompts. Another keynote speaker, Uyi Stewart (data.org), added that “[Algorithms] encode culture. They encode beliefs. They encode worldviews.”
MeasureDev 2024: AI, the Next Generation explored how these digital edifices can best serve the varied interests and needs of global communities. Scholars and development practitioners shared frontier studies and applications of generative AI (GenAI) for social impact. As our keynote speakers highlighted, harnessing GenAI for development requires continued investments in social inclusion and globally representative data. Getting these foundations right could align AI in the service of human well-being, giving us a powerful tool to address generational inequity. Without them, AI is poised to deepen the digital divide and exacerbate inequality.
Representation and Representativeness
For low- and middle-income countries (LMICs) to effectively adopt AI to address social and economic development challenges, we need to expand the cadre of AI developers outside the US and China and proactively include women and LMIC citizens. As Uyi Stewart highlighted, local talent is best placed to manage local data and language complexities. Dunstan Matekenya (World Bank) and Daniel Nkemelu (Georgia Institute of Technology) provided examples through their ongoing research to improve the semantic representation of lower-resourced languages in Malawi and Myanmar. Opening sourcing of models is also essential, as doing so creates opportunities for the global research community to scrutinize and build on existing work.
Obstacles persist beyond who does the analysis to include what they analyze. English and Mandarin drive generative AI, excluding 99.99% of the world’s languages — and the many people who do not speak either English or Mandarin. Charles Mberi (African Institute of Mathematical Sciences) went to the heart of this in his remarks: “If you look at only two languages, you’re excluding five billion people. My grandmother and my grandfather would never be in your datasets.” This data gap biases downstream models, resulting in lackluster and harmful AI systems, and prevents AI tools from benefiting poor and underrepresented communities. Tackling this challenge begins with a comprehensive understanding and continuous diagnosis of the problem. Carina Ines Hausladen (ETH Zurich) offered a good example of how we might do this in her study measuring a vision-language model’s bias in its social judgment of human faces.
With careful assessment and shared performance metrics, we can track how integrating new data sources — especially “bringing online” vast troves of analog microdata waiting to be digitized in LMICs — improves AI-informed services relative to its earlier versions and human-only delivery approaches. With a global community of social impact data scientists to interrogate more representative data, we will have solid foundations to build transformative AI innovations to perceive, structure, generate, or predict human information.
In Silico Social Science?
Since the late 1980s, branches of science like biology have used computer simulations (in silico) to supplement experiments. With the appropriate foundations, we can imagine AI tools that move beyond perception and generation to simulate complex social systems through accurate predictive analytics. This in silico social science could improve our estimation of counterfactuals, drastically reduce the cost of experimentation, and thus help the development sector better understand its impact. In his talk, Benjamin Manning (MIT Sloan) urged the research community to address their assumptions when using these models: “As far as the assumptions we’re making…, and the actual features we need to specify when querying them for information and having them ‘pretend’ to be people, these are open questions… there is no overarching framework.” The next generation of computational social scientists are continuing to iterate on how and when we might apply in silico approaches alongside more familiar “traditional” empirical approaches.
However, as Sayash Kapoor (Princeton University) pointed out, these predictive systems must be externally valid to be useful. Kapoor’s metaanalysis of eight consequential examples demonstrated this is not yet the case. Social scientists and decision-makers must develop transparent and reproducible approaches to determine the usefulness of insights from digital simulations for real-world decisions.
As we build this evidence base, it is clear that GenAI models are powerful tools to help synthesize, structure, and interpolate data today. On remote sensing, Vivek Sakhrani (AtlasAI) and Hamed Alemohammad (Clark University) gave multiple examples of how GenAI models are helping to address earth image challenges like cloud cover and low-resolution inputs, improving the data quality available for cutting-edge computer vision models. Others, like Karim Lasri (World Bank) and Jahnavi Meher (IDinsight), shared tools in development that can partially automate the tasks of aggregating evidence and querying data to further democratize their use. These efforts can be transformative for researchers in resource-constrained settings and could further galvanize efforts to address global inequities in the production of knowledge. To manifest their potential, we need to invest in LMIC scholars and public-minded institutions to ensure their access to the data, tools, computing resources, and skills to integrate them.
GenAI-augmented Social Services
Deep learning frameworks unlocked the capability of digital data holders to create highly personalized and adaptive products. GenAI promises to further this by making these highly personalized interventions personable. Several presentations exemplified this potential, including a personal advisor for small business owners in Kenya, a postnatal health expert and confidant for new parents in Perú, and a survey instrument expert for data collection teams. GenAI can amplify engagement with insights people can harness to realize their ambitions and build the life they want. As Arianna Legovini (DIME) remarked, AI tools can improve the timeliness and cost of public service delivery, creating shorter pathways to results.
Development practitioners need more research on who will benefit from AI tools and its implications for inequality. GenAI may fail to outperform the cost-effectiveness of existing approaches and user behavior may actually produce worse outcomes when given access to a GenAI-powered tool. Understanding this variation will allow the development sector to use this technology effectively. Here we could learn from the tech industry’s user experience (UX) professional community, which leverages qualitative approaches like focus groups and agile prototyping coupled with evaluation to generate impactful innovations.
GenAI is here to stay, with its effects persisting for years to come in new frontiers. The task at hand now is to mitigate its harms and put it in the service of community well-being.
Videos of all the talks at the conference are available on the conference website: https://www.worldbank.org/en/research/dime/measuring-development/2024
