Meeting the target and missing the point: Putting society at the center of digital public infrastructure
Written by Jessica Osborn — CEO, Emrys Schoemaker —Senior Director of Advisory & Policy, and Niamh Barry — Senior Director of Measurement & Impact, all at Caribou Digital.
Alongside this year’s World Bank and IMF Annual Meetings, and following the insightful Co-Develop DPI Summit in Cairo earlier in the month, Caribou Digital participated in several conversations on the social and economic impact of digital public infrastructure (DPI).
Together, these events demonstrated a welcome shift in the conversation toward the importance of putting people at the center of DPI’s design and implementation in order to increase adoption and use. Yet, this people-centric approach seems more nascent in discussions of DPI’s measurement and impact, which still often centers on institutional efficiency and access. While these are important goals (and often the initial impetus for DPI implementation), by omitting nuanced consideration of people-level impact we risk — at best — missing an opportunity for DPI to drive more meaningful development outcomes and — at worst — DPI causing people harm. Digital transformation affects the lived experiences of citizens in very real ways, and by bringing into view goals on inclusion, agency, and empowerment, we uncover a whole range of metrics that must be considered to ensure that the impact on people’s lives is positive. The need to build an efficiency-based investment case for DPI should not trump the need to build the human impact case.
DPI’s outcome problem: A “shared means to many ends”
That people are underrepresented in the conversation on DPI measurement is symptomatic of the fact that, while there is growing consensus around the “whole of society” approach to DPI implementation, this is still nascent when it comes to measuring DPI’s impact. DPI is an emergent system that is deeply interconnected, and as such it requires a systems-level theory of change and measurement approach.
The description of DPI as “a shared means to many ends” highlights the numerous possibilities of use and, therefore, the numerous potential outcomes for different actors within a given system — government, civil society, private sector, businesses, households, and individuals. These are connected actors; thus, impact and information flows are also multidirectional.
As a DPI community, we have many reasonable hypotheses (see Caribou’s illustrative examples below) but not a coherent narrative on the multitude of outcomes that DPI — in its diverse forms — could enable. A shared understanding of DPI’s potential outcomes for different system actors could unlock multi-stakeholder collaboration on the “right measures” and mitigate the risk of misalignment and diminished effectiveness. Investing time in defining outcomes is crucial, ensuring they reflect the voices and needs of all stakeholders. Only then can metrics that genuinely serve these outcomes be defined.
Caribou’s illustrative examples of DPI outcomes
Metrics align intention and value
“When a measure becomes a target, it ceases to be a good measure.”
Metrics are useful ways of measuring outcomes — but only when they are aligned with a broader understanding of potential impact. Fundamentally, outcomes are expressions of what is valued; they reflect intention and galvanize collective action around what gets measured. A focus on misaligned outcomes can have lasting and challenging real-world effects; the financial inclusion sector’s fixation on account access, exacerbated by global measurement tools like Findex, is a case in point. The onus is on us as a development community to ensure an inclusive and “whole of society” approach to defining and measuring the changes that can result from DPI to drive genuinely inclusive and meaningful impact.
Who gets to define these outcomes is also a critical question involving power dynamics that influence whose voices and needs are prioritized. DPI is necessarily a state-driven initiative, but it implies a rearticulation of at least a triad of relationships in the social contract: between the state and individuals, between individuals and the market, and between the state and the market. There are power dynamics and deeply held (and sometimes contested) values underpinning all three relationships, pointing to the complexity and necessity of involving all stakeholders in finding common ground and defining outcomes that matter.
A moment in time for DPI measurement
C. V. Madhukar has said that we are at a unique moment in time in digital transformation. This unique moment offers an opportunity for the development community to align key stakeholders on a common set of DPI outcomes and the right metrics to measure those outcomes. These metrics could: 1) provide a clearer understanding of the benefits DPI delivers to different groups; 2) reveal the risks of DPI, so that products and services can course-correct, and 3) enable comparisons between approaches that could help define “Good DPI”, akin to the influential efforts to mobilize consensus around “Good ID.”
This clarity could guide funding decisions and channel resources toward solutions with the greatest potential for impact. Defining such a measurement framework requires a systems-focused theory of change that incorporates individuals, businesses, civil society actors, and governments, and that is underpinned by a critical synthesis of the existing evidence (in this regard, DIAL and Co-Develop’s forthcoming DPI Evidence Compendium is an excellent first step).
Digital development measurement practices can show the way
While the multifaceted nature of DPI presents a measurement challenge, we are not starting from scratch. As a digital development community, we have learned a great deal from measuring digital initiatives, and these form a valuable knowledge base from which to start. Some key learnings:
- Prioritize outcomes over adoption metrics. Measurement systems reflect values and intentions, and we must prioritize outcomes tracking alongside — easily and digitally obtained — adoption tracking to ensure that decision-making extends beyond access and use. Funders and implementers should measure the change they want to see in order to drive inclusive impact.
- Based on their extensive experience supporting DPI implementation, Public Digital’s call to measure value from the perspective of service users is an important reminder to focus on outcomes. Building on this, we could also draw on Amartya Sen’s influential “human capabilities” approach, as well as C. V. Madhukar’s emphasis on societal capabilities to consider outcomes from a multi-stakeholder perspective. To make a compelling case for DPI, it must be clear that DPI makes a real difference in the public’s lives and that there must be a swift response to any harm — something that matters to politicians, policymakers, planners, implementers, and, most importantly, people.
- Adopt a systems-focused, complexity-aware theory of change. DPI warrants a systems-led, complexity-aware theory of change and measurement framework informed through system mapping, evidence synthesis, and deep and wide stakeholder consultation. As DPI is both ever-dynamic and advancing rapidly, theories of change must also evolve continuously. This approach should consider both opportunities and risks for various actors engaging with DPI. Without identifying all sides, we risk a one-sided view of impact, potentially overlooking significant risks to different stakeholders. Developing a nuanced and adaptive theory of change can support DPI to be responsive, equitable, and impactful for all involved.
- Embed iterative measurement within tech systems. Data on metrics can often be captured in real time using digital solutions themselves, enabling feedback loops that drive continuous improvement. Such cost-efficient embedded measurement and adaptive management approaches can ensure that DPI initiatives focus on delivering public value beyond deployment and adoption.
- Utilize a multi-method approach. Iterative measurement (above) may need to be triangulated with other instruments in order to capture all required data. Findex-type survey data may be required to gather some data points. Additionally, literature measurement can act as a “purpose navigator,” ensuring that deployments deliver tangible public benefit.
DPI impact at a societal scale requires collective action
By building consensus on the outcomes that matter and metrics that measure those outcomes — particularly as they reflect the lived experiences of those impacted — DPI can support inclusive growth, empower individuals, and deliver societal-scale transformation.
The knowledge, tools, and momentum to make a real difference exist, but impact requires collective action and a shared vision.
Please reach out to Jess (jess@cariboudigital.net), Emrys (emrys@cariboudigital.net), or Niamh (niamh@cariboudigital.net) if you would like to discuss our thinking further.
