Principles-Focused Evaluation is at the HEART of how we make sense of change

Because HOW we do not just WHAT we do matters

A sketch created by team member Holly Doron of Dudley High Street with CoLab Dudley principles below and the winds of oppression in the sky above.

Principles-Focused Evaluation (PFE) is a form of values based developmental evaluation we have made integral to our work at CoLab Dudley. Unbelievably, there are over a 100 different types of evaluation!! (A really handy summary video here by evaluation expert Michael Quinn Patton guides you through them). Evaluation approaches — or how we make sense of change — range from highly technical, fixed brief, end of project monitoring types, all the way through to more critical learning framework, ‘improve not prove’, ‘evaluation as intervention’ types. PFE falls into the latter category. PFE is a form of evaluation that uses collectively developed principles and counter principles to GUIDE action and understand change. PFE supports continuous learning feedback loops that enable informed practice, experiment, network and infrastructure iteration.

Screenshot from the handy summary video by Michael Quinn Patton showing the range of types of evaluation and their role. Link to video: https://youtu.be/YReAFxv_31s

As a new form of social infrastructure on Dudley High Street PFE seemed like a perfect form of evaluation to weave into our open to all creative learning practice — detectorism. But that was just the beginning of our PFE story.

The way it has evolved and been embodied within our practice has informed so many more facets of our work that we thought it might be helpful to other social change activists and practitioners to share some of the ways PFE brings value. Let us tell you our PFE story …

Our PFE journey so far

Since 2018 we have been testing out and sensing our way with PFE as an approach to learning and doing that felt more relevant to working as part of a complex system, nurturing regenerative cultures on Dudley High Street. And to be honest, PFE spoke as much to our hearts as it did our heads. You can read about that early part of the journey and some lessons, tips and resources we shared back in Feb 2020. In 2022, PFE (developed by Michael Quinn Patton and peers) continues to shape our way of learning together as a lab team and more widely with our ecosystem.

Most projects tend to have some form of evaluation — these tend to be at the end, and based on outcomes you agreed at the beginning of the project (perhaps with a funder). But if your project is dynamic and emergent — so it is changing and responding to shifting opportunities, challenges and contexts — that type of end of project evaluation is less helpful when actually you need that insight to respond to the shifting context.

What if instead you learn as you go? You all learn together? You share that learning as part of an open and transparent commitment to work out loud in the knowledge commons? What if you based that learning on a set of values that you have agreed will help guide your work towards a long term north star goal without boxing you into a corner before you really understand what you are dealing with? Well that is what our principles (and their counter-principles) help us to do. They help guide us both on a day to day and long term thinking basis. Importantly they help guide HOW we do our work, not just WHAT we do. The process or journey is as important as the outcome. All platforms have rules of engagement (also seen as part of their governance) for us that is partly where our principles become activated — yes they’re essential to our evaluation, but are also key to our practice — a form of values led everyday due diligence that shapes our ways of being, learning, convening and organising.

Sharing this experience seems ever more important as it becomes clear that evaluation within social change is evolving [Here is a deeper dive into this shift in approach by the transformational network impact panel]. This shift is partly because what and how we seek to understand, value and show up in the world is necessarily changing.

There are a growing number of social change organisations, like us, testing collaborative, developmental, and holistic evaluation methods like PFE. Evaluation like this brings to the fore a critical questioning of accountability, and the intention, or motivation behind evaluation. We need to be asking WHY are we evaluating? For and with WHOM? To what PURPOSE? For WHOSE BENEFIT? To what RISK? These questions have been front of mind and heart over the last 4 years of testing, sharing and adapting our PFE approach. During that time PFE has become a practical, lived tool that has informed the design of activities, shaped everyday behaviours and decisions, offered strategic guidance, as well as framing our shared capacity to make sense of and evaluate change. The rationale for choosing to use PFE over more separate, outside expert, end of project forms of evaluation helps make clear the participatory and connected ways of working we practice and advocate for. In reflecting upon our PFE journey so far we recognise that it has become a critical tool within the lab as:

1. An aid to navigation, orientation and alignment
2. A form of lived governance
3. Instructive in revisiting our purpose as a lab
4. A values based theory of action
5. A practice of shared learning and sensemaking

Let’s dig into these five qualities or functions in a tiny bit more detail …

An aid to navigation, orientation and alignment

This quality has been particularly true when new members have joined the team and when we have connected with new collaborators. You can read more HERE about an example of PFE’s navigation role at the beginning of our convening of Time Rebels. We regularly use the principles together with other navigation and orientation tools. For example, HERE you can see how we used them to bring structure to the analysis of detectorism in the wild data from Do Fest 2019 and 2021. This in turn informed design questions and provocations for future seasons of learning and doing.

The principles, with just a few short sentences, enable clarity of position and purpose. Lab team practising the principles, (also known as principles adherence) supports the process of alignment and coalescence within the network. Walking the walk of the principles becomes a way of weaving together and attracting a diverse group via common values, purpose, and practice. As part of this alignment we have learnt to be really intentional about revisiting language to ensure we all feel comfortable using the principles. As lab member Kerry explained “I need to feel them”. And lab member David reminded us the principles help us become “less ME more WE ” and to encourage “less monologue more conversation”. In another practical act of aiding alignment we have used the principles in the process of sense-checking and articulating our Manual of Us that supports all sorts of thriving relationships and collaborations across our ecosystem. To get there we cross referenced the principles, sections of the Imagination Sundial, and associated practices and lessons from a shared enquiry with our learning partner Dr Mat Jones (Birmingham City University School of Architecture). The result was: ‘A friendly guide to collaboration with CoLab Dudley’.

All moments in history are of course shaped by uncertainty, but perhaps the last three years have been unusual in the scale and interplay of complex causes of uncertainty. As a team we have felt that the principles have been an important grounding and reassuring presence in the context of local and global turbulence as we continue to navigate decisions and actions required by the COVID-19 pandemic, the raised global consciousness catalysed by the Black Lives Matter movement, and the urgent future focussing science of the climate and ecological emergency shared by the IPCC. There is enormous value in having a lived form of evaluation to GUIDE practice (your HOW) not simply judge fixed outcomes or solutions (the WHAT). By not prescribing project measures and form, PFE instead invites judgement and expands what is possible by opening up room for creative and flexible responses to shifting contexts. PFE has supported us as we flexed between being homeless as a lab, then co-creating in a space on the High Street, to gathering online on the Virtual High Street, to rhythms of blended convening, and all the while sensing and orientating our work to the potential of place on Dudley High Street.

A form of lived governance

The principles act as a decentralising and empowering organising tool for everyone in the team to use in everyday due diligence. They GUIDE decisions concerning partnerships, new experiment designs, funding opportunities, safe and purposeful convening, and even dark matter considerations around legal contracts and lab space/ team finance. The principles and counter-principles offer us confidence in walking away as well as stretching into different scenarios and decisions.

Instructive in revisiting our purpose as a lab

Back in 2018 the development of our holistic lab goals were developed in parallel with a team process of articulating our principles. Again in 2022, the refining and clarifying of our principles has co-existed and been instructive in being able to better articulate and have confidence in our adjusted purpose. The principles weave into and give a strong throughline to the lab purpose. This has been a really clear example of how they can be useful at the strategic level not just the operational day to day activity level. It isn’t that the goals have changed wholesale — more that over the last four years through a process of experimenting in place, and shared learning that is shaped by PFE, our focus has evolved. Our PFE work continues to inform our working with local people in Dudley to spark creative action, collaboration and shared learning so that together we can:

1. Grow collective imagination capacity that helps us dream into being alternative futures defined by regenerative ways of relating to each other and the world.

2. Convene and support a constellation of Time Rebels — local people committing to intergenerational justice and long term thinking — to spark experiments around their What If questions and imaginings.

3. Steward cultural ecologies that support cultural democracy and cultural activism by asking ‘What if … Dudley was a safe, healthy and thriving place for all beings?”

4. Build a network wide social learning capacity — rooted in discovery, shared learning and many ways of knowing — in order to support systems change.

5. Reveal, care for, and design with local place based knowledge and stories. This is part of a regenerative design approach that prioritises whole living system health by discovering and aligning with the unique potential of Dudley.

A values based theory of action

The principles are forward motion inducing. They provoke reflection and design questions that inspire action and iteration. This isn’t thoughtless action, but rather, action that is shaped by what and how we value, how we relate to each other and the more-than-human, and the collective sensemaking that weaves together a constellation of experiments. PFE helps the thinking-doing symbiosis at the heart of the lab and we have tested out weaving the principles into Time Rebel and lab team experiment design canvas’ to support placing creative shared enquiry and discovery at the heart of all experiments.

With the natural focus of PFE upon action and developmental learning, over the years we have prioritised developing and testing out different props to aid deeper meaningful interaction with the principles. Practically this has involved:

• designing visual shorthand symbols for each principle to help weaving them into other types of messaging and visual narratives
• being proactive in their everyday integration into processes such as our experiment design canvases
• refinement to reduce their number and make them more useable
• a regular rhythm of shared reflection — for example via team principles canvas, principles buddy reflection sessions and working out loud via lab notes
• aligning and bringing them into relationship with other navigation tools (for example, we have linked them to the Imagination Sundial, 3 Horizons Framework and Tulloch’s expanded approach to scaling)
• remembering there is always more we can do to keep them alive and useful to us!!

We view a theory of action as being in contrast to, but NOT in opposition to, prediction orientated theory of change that is often evaluated retrospectively. Instead, our PFE approach to evaluation and living our principles animates a practice that we have come to understand is defined by rhythms of collective sensemaking and collaborative design for cultural animation and activism. The regenerative futures we seek are emergent and rooted in cultures, and as such ill suited to linear impact logic models, or prediction-orientated theories of change. Culture change is collective, uneven, messy, ambiguous, contextual and dynamic. Instead of evaluating against fixed outcomes we have to be comfortable with not knowing, with experimenting, and collectively imagining, co-creating and sensemaking our way into those new futures. The outcomes are multiple and will evolve with unexpected ripples of possibilities. In all this ‘not knowing’ we have our principles to GUIDE us as we design, imagine, discover, sense and create together.

In this respect we have embraced PFE as a form of evaluation as intervention.

A practice of shared learning and sense-making

The PFE approach is a critical part of our prioritising of the role of creative shared learning and collective sense-making in seeking systems change. Living the principles and having the PFE framework has encouraged a form of continuous learning which in turn has informed the co-evolution of our pre-existing open learning approach ‘detectorism’. detectorism is a way of being curious in the world, of paying attention to patterns at a range of scales, and experimenting with ways to reveal our collective insights to inform our designs and actions.

Our What is more Possible Now? lab timeline shows the interplay of the principles, experiments and shared enquiries across the network of experiments/action. To visit the timeline and exhibition: https://bit.ly/CoLabDudleyWhatIs2022

The addition of the principles offered a solid ethical framework that gave us confidence in embracing and valuing many more ways of knowing, from many different parts of the ecosystem, and so detectorism (centred largely around lab team learning), evolved to become ‘detectorism in the wild’ (network wide learning through multiple mediums). In combination, PFE and detectorism in the wild, offer us a much more inclusive and more enriched portrait of the many stories of place in Dudley. In this way ‘detectorism in the wild’ helps with decentering our shared learning away from dominant white, anglo-european, academic research traditions and biases. Instead we embrace the diversity of knowledge making and ways of knowing in our team and wider ecosystem, and weave together a range of methods, disciplines, and worldviews. This intentional weaving seeks to make the lab insights more reflective of our complexity and diversity as natural beings.

A zoomed in section of the CoLab Dudley timeline showing a wave of different types of shared learning enquiries in purple between 2022 and 2023. To visit the virtual exhibition: https://bit.ly/CoLabDudleyWhatIs2022

We are in no doubt that in this work the learning, imagining and the doing are interdependent . So to do this we use creative practices from story-telling, communal walking, theatre, dance, community archiving and counter-mapping, design, and visual arts to help reveal experiment insights. These mix with more traditional participatory research practices like ethnography and architecture’s public life studies. Each lab member and time rebel designs their own learning and discovery approach around their creative practice. Then together we use Principles-Focused Evaluation questions to draw out connections, lessons, and stories that feedback into experiments and dreaming across our ecosystem.

As part of the ‘What is more possible now?’ exhibition we identified 22 petals of things that are more possible in Dudley due to the experiments. The circles underneath the petals show the different learning and design tools that Time Rebels used to support their experiment including principles and detectorism related tools. This learning-doing symbiosis is beautifully illustrated in the petals of possibility section at the virtual exhibition: https://bit.ly/CoLabDudleyWhatIs2022

The most recent step in our PFE journey … a collective process of evolving the principles

From September 2021 we embarked upon a process of intentionally revisiting the utility of our principles and how we use them. Lab member Holly described how “this process of refinement makes the team feel even closer to them — especially if we weren’t involved in the original codesign”. The process of shared reflection with new team members around the fire, included consideration of:

1. The continued relevance and resonance of the 8 existing principles based upon adherence data
2. Our experience of their practical overlap and so chance to cluster principles. On this occasion less is more!
3. The shared language in the principles to help deepen how meaningful and translatable into action they are in different contexts by different team members or Time Rebels. We have learnt care is needed when building shared consensus on language
4. The GUIDE qualities of the refined principles — this is critical to them being robust navigation and evaluation aides (see table below from Michael Quinn Patton’s introduction to PFE)
5. New adherence Questions (walking the walk Qs) and a reframing of outcome Qs in terms of ‘What are we noticing ripple from our actions? What movement towards our goals have we noticed?’ This shift from outcomes framing is part of our acknowledgement that cultural change takes time and manifests in multiple indirect ways that is better served by noticing ripples and movement.

Table from Michael Quinn Patton’s book on Principles-Focused Evlaution that explains that all principles should be Guiding Useful Inspiring Developmental and Evaluable

The team process of reflection and interrogation of the principles in order to refine them made explicit our PFE adherence (walking the walk) data is also in fact a portrait of our practices and processes as a lab and ecosystem of Time Rebels. Our collective lived experience of using GUIDEing principles over the last 4 years has built up our practice muscles and made clear this is critical to our process as a social lab. The practice and the process matter. The outcomes will be determined by the relationships that emerge through enacting and animating those principles. That is not to say this is simply a free for all!! There is an overall purpose to this work to nurture the conditions for regenerative futures. What is more, we use cultural strategy practice to orientate activity towards that purpose. (Note this is a PURPOSE not a single vision — lab member Lorna explains more HERE). Using the principles as a theory of action that guides not prescribes within that framing purpose we create space for emergence.

As we focused upon refining our principles to make them more user friendly in everyday action we have been mindful of a balance — while clustering is useful, there is also a risk of loss of nuance, which can make their utility more open to unintended interpretation (of course this might be a good thing!). We have also continued to value clear counter-principles — our experience shows how useful they are in raising red flags early on. That said, we have also learnt to be mindful of not presenting these as binary positions and so alienating those unable, perhaps due to organisational constraints, to fully embrace a principle and related practices. Therefore, we also now pay closer attention to how we can support the conditions for that principle to be activated in time.

We hope this orientation focuses our practice even more upon potential and collaboration rather than separation.

Through the process of refinement and the parallel sharing of our evaluation learning as part of the “What is more possible now?” exhibition, we were reminded to pay attention to our reason for evaluating and to make sure the learning has clear potential pathways into future change work. As a lab we have always had a focus upon learning from experiments, and we try to be really clear about the threads of learning and related decisions, however, there is always room for different people and beings to interact with the learning-doing process in new ways that illuminates interdependency and supports the potential of place.

And finally, these are our new shiny (and a little bit ancient!) GUIDEing principles

An illustration of our revised principles by lab team member Holly Doron — illustrations once again help with communicating the connectedness of our principles and also the anchoring role of our North Star principle: “Be good ancestors”

Through this mindful, collective and engaged process, has finally emerged our new shiny (and a little bit ancient!) GUIDEing principles! We hope they serve us well in our next chapter. As ever we will work out loud in the knowledge commons, and continue to share our journey with this transformative form of evaluation. Do sing out if you want to chat about this journey — we would love to learn from fellow travellers. In the meantime, we will be using these five ‘hope-full’ principles to GUIDE our work:

• Seek Living Systems Health — through collective sensemaking and collaborative design. We connect issues and reveal patterns by joining the dots across and within systems so that we reveal, understand, and design mindfully for this interdependency.
• Invite Curiosity — We invite curiosity and creativity to nurture the conditions for collectively imagined futures.
• Learn By Doing Together — We co-design and build creative experiences and spaces. Shared learning emerges as we co-create and then reflect upon the process, creation and ripples from our doing together.
• Nurture Connections — We create conditions for meaningful connections — between people, between people and place, and between people and the more-than-human — that animate multiple diverse peer networks shaped by reciprocity, creativity, care and interdependence.
• Be Good Ancestors (North Star Principle) — We champion and act with care for our planet and future generations. We make responsible use of existing resources and lifesources, and help to release overlooked potential. We embrace and share regenerative practice; always thinking long-term, strategically and systemically.

Evaluation as intervention

In finishing this part of our PFE story we hope you can see and feel the care we bring to ensure the principles are more than simply the focus of our evaluation as a community. Critically, they inform our governance, our partnerships, our practice, our shared learning, and collectively they articulate our theory of action. Using these revised principles as a theory of action in 2022 and beyond will bring an urgency and energy to the role of Dudley culture and creatives in responding to global climate and ecological emergencies rooted in colonial, mechanistic and extractive ways of relating to the world. We will be rolling out the Dudley Neighbourhood Doughnut, supporting and convening a growing constellation of Time Rebels across a wide range of sectors, and stewarding cultural ecologies that ask:

‘What if … Dudley was a safe, healthy and thriving place for all beings?”