Practicing what we preach — our experience rethinking M&E in complex settings
Lessons from a new UNDP — Bill and Melinda Gates Foundation collaboration
By Andrea Bina and Søren Vester Haldrup
Dealing with complexity is challenging. It is difficult to implement, monitor and report on initiatives that operate in uncertain environments and that seek to effect systems change. In such contexts, we cannot predict how our interventions will play out (i.e. we cannot assume clear linear causal relationships) and it is not (or rarely) possible to identify solutions and results up front. Rather, we must learn to cope with uncertainty by experimenting, learning, iterating, and adapting, and in this process attempt to, over time, contribute to positive change. But how do we design an initiative that operates in this way? How do we monitor and report on progress? How do we know if we are being successful? At UNDP we’ve committed to explore various system and portfolio approaches in an effort to design value propositions for interconnected and fast changing policy challenges. A part of this effort is about rethinking how we design a set of connected interventions that compound and learn from each other to evolve a continuous supply of new options to deal with issues like changing climate or food security. We call this a portfolio approach. Another important element is to engage in and promote effective collaborative action among all system’s stakeholders, an effort which has been pioneered by UNDP within the context of food and agriculture. But to work in this dynamic and participatory way requires rewiring what we call ‘horizontals’ or the ‘how’ of development — from who we hire (competencies) and how we define risk (audit and legal issues) to how we finance (systemic change) and continuously learn and understand change (M&E).
Over the past few years, UNDP convened several co-inquiry cycles with leading expert of systems change and development to discuss how to practically transform internal and external ways of working. To begin to rethink M&E, we launched in early 2022 an M&E Sandbox with the intent to spotlight various experiments in and outside UNDP that seek to evolve ways to do monitoring, evaluating and learning in efforts that seek to tackle complex systems challenges. This included internal work on documenting systems change, as well as efforts to promote continuous learning and adaptation. The Sandbox has attracted people from over 180 different organizations, on one side validating that we all feel the same pain and on the other triggering new partnerships and ventures into this question of understanding change in contexts of radical uncertainty.
A critical new partnership in this regard launched in 2023 with the Bill and Melinda Gates Foundation (BMGF) — we co-designed an initiative called SMLE (System Monitoring, Learning and Evaluation), focused on rethinking how to do monitoring, learning and evaluation (MLE) when working to affect change in complex systems and unpredictable environments — with a particular focus on food systems as our primary space for experimentation.
The SMLE initiative is trying to tackle a complex challenge so we needed a different approach to learning and progress monitoring and reporting (wouldn’t it be odd if we used a traditional linear approach to M&E for an initiative intended to rethink traditional linear M&E?). This blog provides an overview of the approach that we came up with and our experience to date applying it with our partners at BMGF.
Our approach to learning and adaptation
When setting up the SMLE initiative, we drew from what we already knew about systems/complexity aligned M&E to design a different approach to learning, monitoring and progress reporting. We did this in few different steps.
First, we set our guiding star or a strategic intent. We agreed on an overall intent for our initiative, landing on the following: catalyzing the next practice in systems change monitoring, learning and evaluation. Based on this, we articulated a set of learning questions (see Table 1 below) intended to guide our work. Our SMLE interventions and M&E efforts were to be focused on answering these learning questions and in turn help us catalyze the next practice on systems MLE.
Second, we considered how we could best track and report on our progress given we didn’t necessarily know exactly how to achieve our overall objectives. For this, we drew inspiration from Michael Quinn Patton’s ‘principles-focused evaluation’ and proceeded to agree on six key principles for our work. The idea is that checking ourselves (and reporting) on these principles on a regular basis helps us to get a sense of whether we are on the right track (even if we are making changes in our workplan). We hoped this would help us maintain focus on the overall intent, incentivize and allow us to learn and adapt, while remaining accountable to our funder and partners. Our full set of guiding principles is presented in Table 2 below.
Third, we proceeded to design the process for how we will take stock, learn, and adapt: our adaptive management approach. We decided on working through six-monthly learning loops. At the end of each loop we review to what extent we lived up to our guiding principles, we identify patterns and synergies across our portfolio of interventions, we take stock of whether we are addressing our learning questions, and we try to gauge overall progress against our intent. This includes more existential reflection on whether our intent, principles, and learning questions are still relevant and indeed the right ones. At the core of this is an extensive sensemaking process, which we tailored to our initiative based on UNDP’s sensemaking protocol. Our program officer at BMGF is part of this exercise, which not only is immensely helpful as it enriches the conversation considerably, but also because it helps establish a very different relationship between funder and grantee, which in turn encourages more openness and learning and ultimately leads to us collaboratively hold risks and opportunities.
In each sensemaking, we distill key insights (claims) and identify appropriate actions (propositions) to take to improve and increase the likelihood of effectiveness of our initiative. Actions may include initiating new activities, changing or dropping old ones, and other tweaks in strategy and focus. We have also been experimenting with doing more light touch reflection sessions every 2 months, where we focus on reviewing advances in our workplan and discuss how to improve our delivery. We think of these light touch sessions primarily as single loop learning, while six monthly sessions move into the realm of double and triple loop learning (read this for more on single, double, and triple loop learning).
Fourth, in our grant agreement we agreed a different set of milestones with our partners at BMGF. We agreed not to commit to deliverables that “lock us in” to a particular causal pathway (which, as discussed above, we’d likely need to change as we begin to implement and learn). Rather, we structured milestones around completion of learning loops and the delivery of ‘learning products’ (capturing what we have learned in relation to our learning questions). This arrangement gives us flexibility to adapt our workplan and resourcing on a rolling basis in response to learning and changes in the environments that we work in.
What we learned so far
Going into this, we knew that the M&E arrangements described above are experimental and that we’d learn a lot from trying them out. In this section we outline some of the things we have learned, after having gone through our first two learning loops:
Uncertainty is no excuse not to plan: although it is not possible to predefine results nor to design and stick to a comprehensive multi-year workplan when working in complex systems, we still need to decide on what we will do first (i.e. actions that we think allow us to move towards our intent). In our case, we designed a workplan along four workstreams. We have begun to implement these under the assumption that we’ll most likely adapt as we go along. The key is dynamic (portfolio) management — a more frequent clock speed of generating insights and learning, and evolving these into new actions.
Sensemaking is critical and existing tools are useful: sensemaking is something that an increasing number of UNDP team and country offices practice. Indeed, we were confirmed about the usefulness of the practice after having gone through our first sensemaking in this initiative — both in terms of personal and collective learning and growth. However, in-depth sensemaking takes time and resources, so it is useful to supplement intensive reflective processes with more frequent light touch ones.
Don’t forget to link learning to decision making and adaptation: in-depth (and light touch) reflection can be exhausting, and it often takes longer than expected. In this connection it’s easy to forget to ask: ‘so what’ (i.e. linking learning to decision making). Furthermore, we may surface so many different insights that it is hard to figure out what to do with them. This is a challenge we faced and we tried to tackle it by dedicating time to clustering and prioritization of insights where we clearly link learning to proposed actions and designate who is responsible for the implementation of these. However, it is at this point, i.e. when insights need to lead to different action, that we really feel the tension between existing in a projectized world (where our internal systems, established practices and mindsets prioritize stability, accountability and compliance at the expense of learning and adaptation) and a world of complexity (where learning is a currency that we need to trade in). Straddling these two different paradigms is difficult.
Established organizational practices and procedures may present obstacles to adaptation: as we set up our adaptive management process, we were discouraged from our operations and finance colleagues to engage in more than 2 budget revisions per year — due to the high transactional costs that they would have been facing to comply with organizational procedures. Additionally, procurement policies and lengthy processes contribute to limit flexibility in terms of resource allocation and agility. Fortunately, through open collaborative engagement with operations and financing colleagues, we are getting better at working through some of these constraints, though dealing with procurement and programming timeframes remains an obstacle in situations where we want to act in a fast and nimble fashion.
Genuine ongoing reflection requires time and resources: ultimately this is about evolving a different identity — one where learning and continuous adaptation is in high esteem. Doing this while at the same time having one foot in a current system creates tensions that require intentionally ‘budgeting’ space and time. While we have seen encouraging results after going through our first learning loop in terms of generation of insights and consequent adaptive management actions, we must also acknowledge that organizing light-touch and in-depth sensemaking sessions at the frequency that we agreed (and that we find optimal, at least for our initiative) requires additional time and resourcing for planning, project management and coordination, compared to traditional project delivery. This is important to factor into an initiative’s resourcing and budget. At the same time, however, regular reflection and adaptation enables us to increase the efficiency and effectiveness of our interventions, so we feel it’s well worth it (and good value for money!).
Doing M&E differently requires a different relationship with partners: traditional approaches to M&E can be very transactional and characterized by a principal-agent logic, where monitoring and reporting are seeing primary as a way to keep tabs on the behavior of a grantee undertaking work on behalf of a funder. However, a shift towards M&E focused on learning, adaptation, and navigating complexity entails moving away from this logic, towards one focused on co-creation and experimentation, open honest reflection, as well as collaboration among equal partners. This shift doesn’t happen overnight, but requires trust building, unlearning some of our established practices, and a deeper mindset shift in among all parties.
Preliminary conclusions
Based on our experience to date, it’s our (preliminary) conclusion that, when working on complex problems and uncertain contexts, the type of adaptive and learning focused approach that we are piloting is more useful and ‘fit for purpose’ than traditional frameworks and practices for project management and M&E. However, it is still early days, and we will continue to learn and tweak our approach and day-to-day practices. If you would like to know more about our approach and learning so far — and explore how this could be adapted to your organizational/project context — please do not hesitate to reach out to Andrea Bina (andrea.bina@undp.org) and Søren Vester Haldrup (soren.haldrup@undp.org)
