Mirrors and Metre Sticks
Reflection, Evaluation, and Complex System Change
By Terri Block | Illustration by Shelley Chen
The underlying question that drives my work is how to sustain the change in complex human systems that successful service design requires. The glue that has connected all that I learn about sustaining system change is a core tenet that I hold close: if the work we do can demonstrate impact, and if we can communicate that impact, then sustaining change is possible.
Demonstrate Impact
Let’s unpack this idea a bit. My mental model of demonstrating impact follows a particular and incomplete pathway:
- Do the work
- Evaluate it (throughout a project and at the end)
- Share results with others to gain support.
The incomplete part is that I have always had trouble with the idea of “evaluating” service design. Service design creates better experiences for humans and despite the inordinate amount of things we can measure (from sales, to site visits, to shopping habits over time, to interactions — you name it) there is something that is simply unmeasurable, untidy, uncontainable that comes with designing for very human interactions.
Attending the Business Impact session at SDGC15, my ingoing assumption was that I would learn about how others are making and measuring impact and that I would pick up a few concrete tools for evaluating impact. Instead, a brief snippet of Barbara Weber-Kainz and Linda Kaszubski’s workshop on how service design rocks the organization would challenge my mental model of measuring impact and invite me to think about it in a way that I hadn’t thought of before: as Weber-Kainz and Kaszubski explained how they achieved sustaining a design-focused system of problem-solving pods within a large public organization, they shared a simple technique they used in their working sessions with the teams that were implementing the change. They would ask the folks who were testing the work out in the field to reflect on their experience and unpack what worked, what didn’t, and why.
“We do not learn from experience…we learn from reflecting on experience.” — John Dewey
At first I filed this technique in my brain as a straightforward way of allowing people to reflect on the learning they were doing by engaging in the new activities. As a former educator I know that reflection is a critical part of learning and that in order for people to make sense of new information or experiences and apply meaning to new information and experiences, they need to reflect upon it.
My tidy filing of this piece of information into my brain was interrupted by a thought: this group wasn’t evaluating their impact (that is measuring their work against a set of pre-determined dimensions), they were reflecting on it and using that information to map their next move. In this way, reflection was supporting their change process. I had never before thought of reflection as a means to generate and sustain change within an organization in the same way I had thought evaluating impact could. A stream of questions that I hadn’t before considered blew up my neatly-stacked filing system: Was reflection in this circumstance more impactful that evaluation? Could there be a causal relationship between this group’s reflection practice and their ability to sustain change? Might reflection be an activity that bridges new learning to sustaining change? Might reflection be more effective than evaluation? What could evaluation learn from reflection?
To answer some of these questions, let’s engage in a thought experiment: imagine a world in which we only used reflection to measure impact. Here’s what that world looks like:
Teams who are implementing service design come together at critical project milestones and through multiple touchpoints between critical project milestones to reflect individually and as a group on how things are going. Questions they explore include:
- What has been challenging about the process?
- What has been delightfully surprising about the process?
- How has this process been different than other processes you have engaged with? In what ways is this process additive to how you may have worked before?
- How has your behaviour shifted as an employee or a manager? What do you know about yourself now that you didn’t know before? What has been personally exciting to you about this work?
- What are your personal aspirations about this work?
- What has been different about how we view and treat our customers?
In this model, individuals reflect on these questions and then come together as a group to learn about their colleagues’ perspectives. There is lots of sharing of experience and learning. This learning is captured and fed into next steps. There is no predetermined indicator of success. Success reveals itself as reflection takes place. There is no final “evaluation” of the project but rather a set of reflection questions that generates learning and next steps for the next phase or project. In this model, success is communicated through stories and examples of triumph and growth.
In this model, success is communicated through stories and examples of triumph and growth.
Why is this model worthwhile? How might this model benefit employees, the organization, and the customer? Here are some benefits that came to mind:
- Individual reflection that gets captured into group reflection allows the voices of all employees to be heard. Ideas that may not be shared in the first place are shared openly and the group may benefit from these traditionally-unheard perspectives.
- Group reflection enables employee and manager alignment on the ups and downs of complex projects. This enables deeper empathy and connection between employees and managers.
- Where evaluation of predetermined success indicators may have put employees and managers under pressure to “make numbers”, reflection invites people to transparently share, learn, and tweak their behavior, reducing agency issues and temptations to game the system.
- Customers play a more central role in the company’s mind as employees and mangers are encouraged to reflect on how their project work is creating value for the customer and meeting customer needs.
- Because this model of reflection requires that the learnings be fed into next steps, employees and managers feel that they are part of shaping what gets measured and may be more likely to adopt processes that help them achieve those measurements.
- Because reflection reveals success, there are a breadth of success indicators that may not have been considered if defining success indicators happened solely at the start of the project.
This model is about agility and revelation, and what really matters about this model when it comes to sustaining organizational change is that employees and managers collaboratively define success criteria based on their own experience which may contribute to greater support for the change initiative itself.
Now, imagine a world in which only evaluation was used to measure impact.
In this model, only the management team and project leads define what success looks like. Project leads craft the goal of the project and the indicators that would allow the group to know if they were reaching that goal. The indicators that are crafted are based on concrete outcomes and are quantitative in nature. There are systems built for capturing data that will support with evaluation including things like surveys and tracking systems. Making sense out of the data is a key evaluation activity. When the data for the indicators is communicated to others there is complete clarity on what the indicators mean about whether or not the team is on track to meet its project goal. The data helps the executive team and project leads to decide if the project is working or not.
Here are some of the benefits of the evaluation model:
- Because there is a specific group defining what success looks like, there is lots of clarity around what success means.
- Because employees and managers have a clear understanding of what success looks like, they can shape their project activities around that vision of success.
- Data collection is straightforward because systems are built with specific measurement goals in mind.
- Because data will exist that supports the success or failure of a project, quick decisions can be made about whether to further invest in a project or kill it. This allows customers to gain maximum value from the projects as investments will be made in those projects that are working.
For me, this model is about predictability and structure. What really matters about this model when it comes to sustaining organizational change is that people work within predictable structures to measure impact.
As I spent time with each model, I recognized that evaluation had a whole lot to learn from reflection and that I truly valued most of what reflection had to offer.
As I spent time with each model, I recognized that evaluation had a whole lot to learn from reflection and that I truly valued most of what reflection had to offer. That being said, reflection was missing the structure and predictability that I value from the evaluation model. To end the thought exercise, I asked: what if we pushed the agility and revelation of reflection so far that it actually generates the structure and predictability of evaluation? Could reflection be used to generate a more robust set of questions that we want to evaluate — questions that come from the very people whom are implementing service design and need to sustain the change they create?
When we evaluate we have already defined what we are looking for. When we reflect, we are building new knowledge. We unpeel, discover, and make a plan for what’s next. Reflection gives people the opportunity to grapple with the unquantifiable pieces of an interaction that they want to make sense of. Reflection allows people to share their thinking with one another and contribute their voice to what is working and what isn’t. Ultimately, reflection allows people to apply new meaning to information and experiences and build new knowledge. By going through a reflective process, teams come together, share thinking, and learn together what is working, what is not working, and why.
When I think about how to measure service design work, I wonder if reflection as a tool is more aligned to what service design is all about and begins to solve some of the things that make me uncomfortable with evaluating service design using predetermined indicators of success. And yet, despite its usefulness, the buck cannot stop at reflection. The reality is our world currently requires evaluation in its more traditional sense. Clear numbers are comforting and tell us a lot about how our projects are doing. But the missed opportunity with evaluation is that because success indicators are designed by an exclusive group of people and before a project it fundamentally may be missing important success indicators that were never considered. My question after all of this has become: how can we design a more robust set of indicators by employing reflection? In this way, reflection would be used not only to consolidate learning and to generate areas of inquiry that allow us to learn more, but to generate the questions we ultimately evaluate while creating group cohesiveness that supports the alignment and buy-in that change creation and adoption requires.
This year, my colleagues and I are designing a training program for one of our clients that will train client employees on the new initiative we have designed. When we inject reflection periods at the end of each session it won’t only be to help participants build knowledge and make meaning, it will be to help them become champions for the initiative itself. At Bridgeable, we create a bridge for our clients to translate knowledge into action. As a tool, reflection may just be the bridge that translates knowledge into change.
