This is why evaluation is a tool for systems-change
“How are you shining your light in your evaluation practice: is it through making sure all stakeholder voices are included, particularly those who are often in the dark? Is it through shining light on the historical imperative associated with certain evaluation methods/practices? Is it by shining the light on the pathway to evaluation for new professionals? Is it by creating new evaluation theories, methods, or practices?” — these interconnected questions shaped the dialogue at this year’s American Evaluation Association Conference 2020 and they set the tone for this piece.
From the devastating, reverberating impacts of the COVID-19 pandemic to the waves of protests happening worldwide in response to the terrible, unjust death of Breonna Taylor, George Flloyd, Daniel Prude, and Rayshard Brooks in the United States, 2020 has been a year marred by crisis. It’s not really that long ago that Australia was engulfed in bushfires and teenager Greta Thunberg was delivering repeated volleys of home truths to governments about the climate emergency. Yet, in the midst of all of this, masses of people are waking up to the urgent, deep need for structural transformation — and their role in it; the need to shine their light — as a prerequisite to building a safe, equitable future for all of life on earth.
Building fair, resilient communities across the globe is a shared responsibility and it is a whole-systems challenge.
The emergencies we face today necessitate that we rethink the role that we all play in these needed transformations. For evaluators, we must do so too. We must move away from the project-dominated approach that has been long ingrained in how we approach systems interventions, and toward a holistic, big-picture approach. Changing our evaluation processes will be crucial to building — and maintaining — the new systems we need to solve the uniquely challenging problems of our time.
These are global phenomenons that affect us all in complex, interconnected ways that do not fit into traditional siloes of analysis. Indeed, the wide-ranging complexities of the climate crisis, for one, demand a change in how practitioners and funders involved in sustainability and international development frame the issues they tackle and assess their impact.
As one example of this complexity, just consider how studies have linked climate change to increased gender inequality in scenarios like the following: extended drought disrupts crop production, which increases the chances of the male member of a family to migrate to a city for work, which in turn increases the chance of a female member in said family to fall into poverty.
Until now, the dominant focus of evaluation has always been a project or program model. Designers, funders and evaluators have been trained to use project thinking, reducing complex dynamic systems to linear logic models aimed at narrow goals. Efforts to bring this approach to a global scale have relied on duplicating narrowly-tailored models, bringing a standard method to bear in dramatically different settings. But this uniform approach is ineffective in dealing with interconnected problems such as climate change, worldwide poverty, or unsustainable food production.
That failure of vision is not for lack of trying. There are well over a hundred different models to evaluate impacts, from formative and summative to realist and theory-based evaluations, from outcome mapping to collaborative and impact evaluation and on and on. These methods have their place. But, for practitioners and funders who want a deeper understanding of their programs’ impacts on the planet, a different approach is needed.
This project-based mentality is deeply embedded in institutional strategies. Yet, it is not appropriate to the reality we are facing. International and governmental agencies, as well as philanthropic foundations, issue thousands of requests for proposals every year. In turn, successful proposal writers excel in generating the appropriate kinds of projects and programs that funders are looking for.
If we have learned anything in 50 years of international development and evaluation, it is that the isolated projects and programs that are often selected for such proposals do not lead to major systems transformation. Systems transformation is dynamic and contextually adaptive. For policymakers, philanthropists, and others responsible for leading social change interventions to truly respond to the systematic threats facing us, we need a similarly systematic approach to evaluation that looks beyond nation-state lines and across sector and issue silos to connect the global and local, the human and ecological, the macro and the micro through evaluative thinking and methods.
Evaluating impacts holistically means watching for, making sense of, and interpreting the implications of things that are interconnected in the global system. A new framework, Blue Marble Evaluation (Patton 2020), keeps this perspective in mind and empowers evaluators to both “zoom out” for a big picture perspective and “zoom in” to understand and incorporate contextual variations, problems, and solutions.
For instance, the Global Alliance for the Future of Food, a strategic alliance of philanthropic foundations and others working to transform global food systems, is using this holistic approach to craft new, better solutions and deepen collaboration among philanthropy, researchers, grassroots movements, the private sector, farmers and food systems workers, Indigenous Peoples, government and policymakers.
Blue Marble Evaluation played an important role in the Global Alliance’s initiation of the Beacons of Hope project and the creation of the resulting research report which highlights case studies from organizations and initiatives working to transform global food systems. The Alliance is using the framework to identify connections between the initiatives identified and to facilitate the creation of a new network of transformative leaders, from Senegal to San Francisco, who are actively engaging in transforming food systems. Crucially, the Beacons of Hope research report helps practitioners and funders understand the potential of food systems transformations to address a wide range of enmeshed challenges across traditionally siloed sectors, from climate change to urbanization to public health.
This collaborative approach means that evaluators must be intrinsically part of program development and program roll-out. Design, implementation, and evaluation are typically treated as separate functions handled by different people. Instead, at the heart of the Blue Marble perspective is the continuous effort to break down these silos and integrate separated functions, to connect people and places and create linkages across time.
It’s become abundantly clear that transformational change is needed for all life on Earth to be sustained, and to be protected and respected; those transformations need to happen sooner rather than later. That urgent need can make it difficult to see both the challenges ahead and positive developments already underway.
But in the face of ongoing global challenges, we have seen positive changes these last few years: major increases in forest restoration, initiatives to reduce pollution, a decrease in extreme poverty, inequality and fossil fuel use, breakthroughs in public health, and more (Hervey, 2017). Equally, we can take heart from the hundreds of thousands of people coming together in solidarity in the Black Lives Matter protests worldwide and in the individual acts of kindness shown by people in response to the impact of COVID-19 in their communities.
These acts add up. And, we can expand and build on them through evaluation with a whole-Earth perspective too — this is the essence of Blue Marble evaluation theory and practice. As the calls for action mount, evaluation can no longer be treated as merely a compliance activity or paperwork exercise. Transforming our society means transforming our systems and our way of thinking — moving away from a project-focused mindset, and toward one that recognizes the fragile, finite, and interconnected nature of the Blue Marble we inhabit.
Michael Quinn Patton is former president of the American Evaluation Association and author of eight major books on program evaluation. He is also a senior evaluation advisor to the Global Alliance for the Future of Food.
Pablo Vidueira is the Blue Marble Evaluator of the Global Alliance of the Future of Food. Evaluation professor and scientist affiliated with the Technical University of Madrid, and president of the Iberian Evaluation Association -APROEVAL-.
