Does the whole add more than the SUM of its parts?

Four lessons for monitoring & Evaluation — Part 1 of 3

Florencia Guerzovich

In the social change sector, the call for bigger, traceable impact is everywhere. At its core, there is the hope that interventions and change agents collectively will add more than the sum of the parts. There are many labels. Different ways to cut and dice the trend. In the Monitoring and Evaluation space, this concern is captured by the new evaluation criterion from the OECD-DAC, coherence.

The challenge is that evidence about additive effects is harder to come by than assumptions about it, including in multi-organization, multi-site programing (e.g. in the Transparency, Accountability and Participation space). Moreover, Monitoring, Evaluation and Learning (MEL) systems often have a coherence blindspot. They rarely are built to tell us whether, how, why & under what conditions the parts jointly (i.e. the cause) produce MORE than they would have produced on their own (i.e. the effect). There are not many written up examples of organizations actually tracking or tracing this kind of change (see e.g. here, here). There is not much guidance telling us explicitly what we may need to do differently when it comes to tracing contributions to additive effects (see e.g. SI Impact Assessment Guide; lessons from the Global Partnership for Social Accountability and USAID’s Collaborating, Learning & Adapting (CLA) Maturity Tool).

In this blog post, I set the scene by presenting the new OECD-DAC’s evaluation criterion: coherence. In the next posts in the series, I’ll share 4 lessons I learned by experimenting with colleagues on how to MEL “the whole adding more than the sum of its parts” in a range of organizations, projects and geographies. Collectively, these lessons provide some practical cues to right fitting MEL systems to assess and learn if 1 + 1 = 3?

What is coherence?

Coherence, according to the OECD-DAC, is about how well an intervention/portfolio fits with others in a country, sector, organization or portfolio. Coherence is particularly significant for complex problems which a stand alone intervention cannot move the needle. It requires us to begin asking whether the ubiquitous hope that our work can contribute more by interacting with other work holds. Take, for example, some USAID project logframes and evaluations that have begun asking how do areas of overlap between efforts under a project and those of other USAID/country office projects present redundancies and/or synergies? — a trend that will likely increase with the implementation of their new Local Capacity Strengthening Policy.Think of work to incentivize that sectors within the organization integrate their work to produce additive, interdependent effects. Oxfam GB’s announced pivot towards portfolio Monitoring and Evaluation of the UNDP’s ongoing quest to focus on portfolios rather than siloed interventions or collections of projects. All of these efforts use different language to express concern with internal coherence.

External coherence focuses on the fit of the intervention/portfolio with other actors’ interventions/portfolio. We are this criteria already make its way into MEL terms of reference, see for example the Inclusive Data Charter’s (IDC) Strategic Review Requests for Proposals: It asks about coherence: “How can the IDC better complement other relevant external initiatives and sit within the larger agendas of IDC Champion organizations?” Now think about efforts to improve donor coordination in a country or field through dialogue or joint funds, work to better integrate interventions led by government, private sector, and civil society (or different parts of each such as journalists, advocates, and think tanks, e.g. here, here), across levels of government, or, more generally, all those ambitious calls for working towards systems change. There are too many of these examples to link in a blog post, but I know few that have an explicit methodology and findings testing their claims.

Coherence is about relational outcomes

Coherence, and more generally portfolio and systems change work, require prioritizing (scarce resources and limited attention spans) relational outcomes over others. Those outcomes that one party could not have achieved without an interaction with another party. I can provide a few examples to illustrate the point. I am aware that relational outcomes are not ubiquitous in MEL. Think of an organizational level outcome with effects downstream: staff are willing to go above and beyond collaborating with each other and integrating programs across silos to work on a multi-sectoral problem because they get personal satisfaction out of collaborating and learning with others. Think of a high level national bureaucrat who realizes that leveraging lessons from the dialogue between a community and local bureaucrats can be adopted and adapted to solve a problem in a service delivery system. Those outcomes are relational because they would not have existed without the often invisible relationships (and associated trust) that enable them to emerge.

The people at the center of these relational outcomes get what this is about. I recently supported a process to coproduce a standard for open contracting in the State of Santa Catarina in Brazil — an organic initiative that grew out of the pain of a local bureaucrat and ended up engaging a broad multi-stakeholder network (material only in Portuguese). During the pause and reflect meeting to close the first phase a local bureaucrat from another town reflected: ”I’m from technology, but (building this standard) was all about relationships”. I have written in the past about relational outcomes and hope to write more on what I have learned since, when I can free up more time ;-). For now, if you only can monitor and evaluate a few outcomes in an initiative with systemic ambitions, consider prioritizing relational ones over others.

Coherence =/= harmonization

According to the OECD-DAC coherence CAN include complementarity, harmonization, and coordination with others. However, I have found it is particularly important to underscore that coherence DOES NOT REQUIRE harmonization of goals, interests and/or strategies, in the sense of making the effort to ensure all are the same or formally part of the same plan of action. Instead, we have to question and reflect on how far one expects (or incentivizes) parts to be fully aligned? Is 70% good enough to get the benefits of broadly going in the same direction AND of the diversity among the group?

Coherence sometimes is conflated with harmonization, but it should not be. Stakeholders can produce positive synergies and add value while avoiding duplication of efforts even when, or in some cases especially when, they divide labor and loosely articulate interventions/portfolios to make the most of their respective comparative advantages. In many cases, an agreement to disagree on specifics is part and parcel of building positive interactions where possible — a point underscored by research and evaluation on co-production and collaborative governance. Such “agreement” is inherently about acknowledging the legitimate diversity of perspectives in a system and the uncertainty that we have about the definition of a complex problem and course for action. In MEL jargon that is the acknowledgement that when context and causal pathways interact we need an approach that allows for testing multiple pathways to the same outcome because we are in the world of equifinality (or multiple conjunctural causation) — a point that I suspect has more similarities with a lot of approaches that focus on emergence than each groups’ conversation, language and branding would suggest?

In development work, as in many other fields, donors’ (and donors’ expert advisors, such as researchers and peer reviewers) perspectives have long had the power/hegemony to speak louder and define what should add up. There is a push from certain organizations to change this radically and trust that communities and beneficiaries become the dominant voice. A course-correction is certainly warrantied. At the same time, eliminating donors and their advisors from the equation seems to obscure the role they play even more rather than make their power in the system more visible. In my experience, sponsors and those they report to, implementers and communities all have different spheres of influence in the system, make different types of decisions, and need MEL that speak to that diversity. Perhaps, instead of a white or black change, in acknowledging the relationships and the system, we should be exploring brokered compromises? The diversity and asymmetries of types of knowledge, resources, power, and perspectives and levels of zoom are part of the system in which all interact, at least for many in the foreseeable future.

This space for enabling and valuing diversity while supporting alignment and synergies is where potential for social learning and action lies. So, the trick is reflecting on those conditions under which harmonization will pay off and those in which other pathways for coherence, such as loose coordination or exchanges of information, will be more productive in terms of positive synergies. MEL professionals need to avoid squeezing out the space for social learning in the quest for elusive harmonization, not least because the quest for harmonization may produce negative interactions we should explore and learn from.

Can you think of all the time spent trying to reach an elusive consensus with partial allies dotting all i’s and crossing all t’s instead of actually trying, learning together, course-correcting and trying again? In my experience of MELing global portfolios, this hope of full consensus has become the first red flag when I assess whether a series of groups will be able to join forces to tap a window of opportunity, broker a new agreement to adapt or they’ll miss it all together. Proactively facilitating good enough, still power-aware agreements is of the essence — and requires a skill set, political economy analysis and realistic expectations that should be valued.

The rest of the series

In the rest of this blog post series I discuss Four Practical Lessons about right fitting MEL for Additive Effects. Post 2 of the series focuses on the first three lessons, while post 3 focuses on the last one, and recaps the insights.

1. Prioritize: Which Parts Do You Think Should be Adding up Now?
2. Articulate Ideas About How & Under What Conditions 1 + 1 = 3 Happens, while operationalizing and legitimizing qualitative thinking
3. Bricolage Heterogeneous Inputs to Make the Most of What We Have
4. Take Time Seriously, including managing projects adaptively ACROSS project cycles

Thanks to Tom Aston, Soren Vester Haldrup and David Jacobstein for helping improve this blog post series. While not directly engaged in this post series, thanks to Sol Gattoni, Dave Algoso, Alix Wadeson and Alina Rocha Menocal for pushing my thinking as we thought through complex MEL challenges together. Also to Lauren Kevill as well as Andrea Azevedo, Mark de la Iglesia, Russell Pickard, Beth Dunlop and a long list of colleagues at their and other organizations for opportunities to explore this issue by doing with them. Ana Palu gave me some tricks and tools to add humanized drawings to my work.