The Business of Impact Measurement
by Anne Birch, Kristina Serbesoff-King, James Byrne, Temperince Morgan, Crystal Diaz de Villegas, Maxene Atis, Vera Agostini, Shenique Albury-Smith, Luis Solorzano, Stuart Creque, David Matheson, Somik Raha
This article chronicles a journey that began at the Gordon and Betty Moore Foundation and continues at The Nature Conservancy. Along the way, the participants made a surprising breakthrough: they discovered that metrics are valuable, but not for what they measure. This article is a valuable guide to changes your organization can make to drive productive action and improve the effectiveness of your work.
Groundwork
Scene: Conference Room in Miami at a workshop for The Nature Conservancy, 2015
“You just lifted a big weight off my shoulders,” said the scientist, with a huge expression of relief.
“What do you mean?” asked the consultant.
“Well, in the non-profit sector, we are constantly pressured to measure impact. And you are telling us that what matters cannot be measured!”
“That is correct. That is my conclusion. Good metrics drive productive action; they don’t measure value.”
It was a powerful moment in a room of forty people, most of whom were working on major environmental projects.
The consultant, Dr. Somik Raha, had studied the link between values and metrics for a Ph.D. at Stanford University, and had applied this insight in a 9-month project at the Gordon and Betty Moore Foundation. This article presents a collective view of breakthrough insights that emerged when The Nature Conservancy stopped trying to measure value and instead focused on driving productive action.
But first, why is it that we cannot measure the value that was created by our action?
Somik recalls when he began to research the linkage between values, metrics and productive action. His “aha” moment came when he came across the work of Dr. Robert Hartman, a pioneer of Formal Axiology (the study of the nature of value). He remembers, “I realized that we were all busy making algorithms that optimize value, and yet no one had defined value in our field. I asked my advisor and co-founder of the field of Decision Analysis at Stanford University, Prof. Ronald Howard, if he could help. In a strange twist of fate, he remembered a class he took five decades back from Prof. Hartman, which tackled this topic. Hartman’s work ended up becoming central to my own research.”
The key insight from Hartman’s work can be illustrated with a simple example: tooth-brushing.
If your dental health is important to you, you probably keep track of how many times a day you brush your teeth. But you’re not interested in the count of brushings: what you’re after is making sure you brush. Two people may have the same number of cavities, but you can’t distinguish their dental health with just that metric. You can engage with all kinds of other metrics and still not get close to capturing dental health in its totality. And yet, keeping count of the number of times you brush is one of the most effective ways to drive your action of brushing that brings dental health into your life. This leads us to a profound realization.
What truly matters is not countable. What is countable does not truly matter.
Try it with anything that you count. You will find, like Hartman did, that this statement shockingly holds true. So should we end our engagement with metrics? Not at all. Metrics are black-and-white constructs that drive action.
Good metrics are those that drive productive action toward value creation.
Somik took his first steps toward applying this insight when he joined Dr. Luis Solorzano at the Moore Foundation in 2010 to build a decision system for valuing priorities for forest conservation in the Amazon.
Starting with mapping sources of “forest value” and communicating them clearly, Luis and Somik created an approach that allowed them to investigate what their value positions really meant by working out the implications of those positions through decisions. Luis remembers it this way, “The analysis was powerful not because it told us what to do, but because it helped us understand who we (the Gordon & Betty Moore Foundation) wanted to be. Participants realized that some value positions fit who they wanted to be much better than others, and they could now explain why.”
The value of a decision analysis is that it takes complexity off the table so we can focus on who we want to be.
Although the insight was a huge one, this work did not have a direct impact on Moore Foundation’s decision-making process. Luis remembers, “We were so excited about our findings that we forgot how hard it was for other people to relate to our analysis. Our initial models were all spreadsheet-based, and were too complex for anyone else to internalize and find useful; we failed at socializing and communicating a novel mental framework as part of the decision-making process to allocate philanthropic assets.”
Luis left the Moore Foundation and spent the next three years as chief of staff at the world’s largest research consortium for rural development, CGIAR, where he realized that this problem of making high-quality decisions cut across many non-for-profit sectors. He subsequently joined The Nature Conservancy as the Executive Director of the Caribbean Division. He notes, “I remembered the work I did with Somik and decided to pick it up again in 2015, this time focused on building a high value conservation portfolio for the Caribbean. We had learned our lessons from the work at Moore and wanted to take a second pass at this opportunity to bring formal decision analysis to inform conservation investments.”
By this time, Somik had started building a portfolio management software solution at SmartOrg, a Decision Analysis software company, that eliminated the complexity of spreadsheets that had prevented success in his earlier work at the Moore Foundation. Together with Luis, they decided to use this solution to help identify the Conservancy’s most valuable projects in the Caribbean[1].
First Phase: Mapping Values to Drive Productive Action in the Caribbean
The project started with value-mapping of every participant and the Conservancy’s meaningful purpose was induced from these mappings. The inquiry on values proceeded along three dimensions: Head, Heart and Habit. The head, or the intellectual energy of the Conservancy centered on “Conservation by Design” — deliberately using the best science available to make thoughtful decisions. The heart, or the emotional energy of the Conservancy centered on “People and Nature thriving together,” and the habit, or the unstoppable momentum that the Conservancy has, centered on “Finding Pragmatic Solutions.”
Crystal Diaz de Villegas, Coral Project Team Manager and Board Liaison of the Conservancy’s Caribbean Division, recalls, “People were used to seeing a consultant come in, or some directive from leadership about how we’ll be doing a new process for strategic planning. The values conversation was an approach we had never taken before. What we’re doing is seeing what drives people, what drives the organization, and what you care about.
“It sounds odd to have that conversation first, especially to me, because I’m one of those people that just likes to get started with something. Stepping back to look at values was really meaningful. We spoke to a few board members and some staff, and I think they enjoyed that conversation because it was one they hadn’t had in the context of their work life in a long time.”
For Maxene Atis, Project Manager of the Conservancy’s projects in Haiti, getting to the Heart value of “People and Nature thriving together” immediately told him this wasn’t just another management process. “With the previous approach, we would talk a lot about nature and tend to forget the people in the environment. This is problematic in the Caribbean where we have a lot of small islands with a lot of people. This new approach with values gave me hope that we would better understand how the work we do impacts not just nature but also people.”
The Habit value (“Finding Pragmatic Solutions”) is also a very interesting marker. For example, the Conservancy uses market mechanisms in innovative ways to protect ecosystems, from buying endangered land or critical habitats and protecting them in perpetuity to making impact investments in debt-for-nature swaps (financial instruments that exchange a country’s debt for commitments to conservation projects). This is a distinctive aspect of their organizational values and helps explain where their natural cultural energy rests.
Aligning to the Heart value (“People and Nature Thriving Together”), Somik designed a multi-attribute metric (a composite metric derived from multiple primary metrics) for projects in the Caribbean. A portfolio evaluation approach built on this metric allowed the Caribbean team to identify their top projects and get them funded. Their board members were impressed by the rigor of the process and the logic that was used.
Crystal says, “One of the first things the process did for us is that it helped us design conversations to develop theories of change. Since I was fairly new, I was asking each of the project managers what I thought were overly simplistic questions, like, ‘What do the fishermen have to do here? What are these nets called? Why does this matter?’ While it could come off as, ‘Oh, she has no idea what she’s talking about,’ these questions actually forced managers to step back from their projects and really think about, ‘Why are we doing this? What’s the end goal here?’
“Once we had these conversations and developed theories of change, we followed it up with quantitative assessments that allowed us to stack everything up against each other for comparability. Part of that was looking at each desired outcome and asking, ‘Well, why does this matter? How do you determine that you’ve actually succeeded?’”
However, enthusiasm behind the approach wasn’t up to the level that Somik would have liked. He recalls, “I had gotten really excited about coming up with a special multi-attribute value metric, not acknowledging that such metrics don’t lend themselves well to natural interpretation. I had forgotten that the purpose of metrics was to drive productive action, and in our case, productive action meant rich conversations around value creation. With the multi-attribute metric, such conversations could only happen if I was facilitating, and that would keep the process from becoming a successful internal capability for the Conservancy.”
Second Phase: Simplifying the Metrics for the Florida Chapter
Fortunately, the Caribbean project was successful enough for the Conservancy’s Florida chapter to take notice. Temperince Morgan, Executive Director of the Florida Chapter, remembers her chapter being swamped with projects. She recalls, “we didn’t really have a good framework for talking about how these projects were organized, how they work together and what they all added up to at the end of the day. Even if we execute on all of these projects, how do I describe to our donors and our board what that accomplishes? What does it all add up to? We didn’t have the ability to answer that.”
Temperince invited SmartOrg to help and this time, Dr. David Matheson of SmartOrg joined the team. He radically simplified the metric to count acres conserved and people impacted positively. The breakthrough was in creating people and nature standards that, when passed, would allow the Conservancy to count a person as impacted or an acre as conserved. This avoided burdening a single metric with every possible nuance to account for the rich complexity of any conservation effort.
Anne Birch, Marine Program Manager of the Conservancy’s Florida chapter, says, “To whittle the complexity of doing restoration and protection of nature down to just number of people and number of acres was really hard to get our head around. But one of the major reasons why the Conservancy embarked on this was the volume of work we do. We need to focus in and do the best conservation work that we can with the resources — people, money and time — that we have. If we could simplify the metrics for each project down to something that’s standardized, where we could compare our work across projects no matter what type of work it was, and that everybody understood and agreed to use to evaluate our work, then we could see what gives us the biggest bang for our effort. This wasn’t intended to say that one project is better than another. Instead, we wanted to answer, ‘Given the amount of resources we have, what is the best effort for conservation that we can get?’ And by simplifying the metrics down to number of people and number of acres, we were able to then look across the projects.”
Challenge: Lots of value drivers cause metrics to be very complicated
Solution: Focus only on those metrics that represent real tradeoffs and have natural interpretations
Outcome: Settled on number of people and number of acres. Forced the issue of standards.
Production, Innovation and Capability: Three Project Classes
Another important step was to classify projects based on their decision context. Production projects are continuing conservation efforts, Innovation projects investigate new conservation approaches, and Capability projects develop the Conservancy’s infrastructure. Temperince points out, “We were treating these projects the same and were not accomplishing what we want, not managing them properly, or making the right decisions around them because we hadn’t made those really important distinctions across the different types of work that we’re doing.”
Challenge: The Conservancy chapters’ projects were at different levels and formed an unworkable list
Solution: Classify projects based on decision context
Outcome: Separate portfolios based on decision context:
· Production
· Innovation
· Capability
That brought up another question. Kristina Serbesoff-King, Director of Science and Planning at the Florida chapter, recalls, “Before we could even do a Decision Analysis, we needed our own standard way of saying what is a project. We thought we knew what a project was; and yet when we started to try to come up with our Portfolio projects that we would input, analyze, or come up with the metrics for, we realized that we were just all over the board on what we thought was a project.
“The hard work was the fact that we weren’t framed around project portfolio evaluation in a really systematic way. We had to figure out how to identify a project and its purpose before we could even get to figuring out metrics.
“That’s when we realized that every project had to be an instance of a product. Ideally, you would want to have multiple projects that are repeatable instances of the application of a product. We worked hard to define projects that led to products. That was a whole new way to frame projects for us.”
Challenge: It was unclear what the Conservancy chapters’ projects would accomplish
Solution: Define products as repeatable work that reliably delivers to a standard
Outcome: Clear list of products, the standards of value delivered and product instances
After defining products and projects that delivered these products to a standard, the team had to assess the impact of these projects. Project teams complained to those facilitating the process that data gathering was onerous due to the search for the right numbers. David had seen this story play out many times. He coached them, “The data you are looking for is about the future, which does not exist because you are going to create it! All our numbers here are fiction and we’re not looking for perfect numbers. We are looking for comparability.”
Anne underscored this in retrospect: “‘Embrace imperfection’: that’s what this process should be called. It’s not about getting the numbers right or finding the right answer: there is no right answer. It’s, what’s useful for you in order to deliver to a standard so you can then compare projects that follow a similar standard. Don’t let not having perfect information paralyze your decisions. You can always reevaluate.” Kristina echoed that sentiment of viewing the process as an iterative one “so it’s not just one and done.”
Challenge: Data gathering is onerous because it takes a lot of work to accurately reflect reality
Solution: Focus on credibility and comparability, not accuracy
Outcome: A clear sense of the relative merits of each project and how they stacked up
After several months of evaluation, when it came time to make decisions, the team found there was much apprehension that great projects would get killed simply because their metrics looked bad. This had to be addressed at the human level. David, Somik and Luis returned to the early insight from the Moore days.
The value of a decision analysis is that it takes complexity off the table so you can focus on who you want to be.
The team was offered the freedom to make their own portfolio proposals and pick projects even if those projects didn’t look good with the metrics selected. The analysis with the metrics let the teams make their selections with their eyes open. This proved to be the final step that allowed the team to realize that this was an empowering process and not something that would take away their decision-making power.
Anne recalls, “I’m a project manager, so I could understand when they said, ‘My project may not rank high; does that mean my project isn’t worthy?’ So we had a lot of discussions with project managers about this, and did an analysis that identified those projects that were high value in number of acres and number of people, and those projects that just didn’t get us there. We worked with the project managers to ensure that the analysis passed scrutiny.
“When we took out as much emotion as possible and just used the metrics, was the analysis validated by what we really understood the projects to be? In most cases, I think the answer was, ‘Yes.’ Some projects that were really good quality were just so small, they weren’t going to get us to our goal. The analysis gave us an unbiased way of saying project X might not get us there as well as project Z.”
Challenge: Conversations swirl — inability to get to the heart of the matter on choices. Fear that analysis will restrict choices
Solution: Use analysis to focus discussions about what we really want to do
Outcome: Florida: 26 projects whittled down to 11, 7 to transition out of, 8 to be reformulated. Freed up critical staff time and resources for innovation.
Caribbean: 47 projects whittled down to 6–10 to keep, 6–10 to transition out of, remaining transferred to working group to be improved
Phase Three: Developing a Self-Sustaining Capability
The Conservancy’s Florida and Caribbean teams subsequently took steps to turn this process into an internal capability by having staff from the Florida chapter facilitate workshops for the Caribbean chapter.
James Byrne, Senior Conservation Scientist at the Florida chapter, says, “When it came time for the Caribbean Division to have that same type of workshop that we did in Florida, I helped facilitate it with Kristina, with a little help from Somik on analytics. We wanted to circle back and apply the things we learned in Florida to the Caribbean Division.
“As Conservancy colleagues, we didn’t have to worry about establishing rapport and trust — we came into the workshop with it. When project managers and program managers saw some of their work not ranking very high, we shared how we felt when we saw some of our work not ranking very high. So it helped ease some of that tension that is naturally created when you’re evaluating projects. Participants could relate to the fact that we had been in the same position they were in now. That really helped them move forward.”
On the receiving end, Maxene felt having Kristina and James facilitate was “probably the most important decision that was made. It was really interesting as we had started this process before them, and then they really got into it, started to use it, and now they were here to share it back with us. People like James are people that we know (James had worked with us on projects before) so it’s like if they can do it we can do it too. Even more, you feel that it’s not something that somebody else is trying to force you to do or to sell, but it’s just part of how the Conservancy does things.”
James and Kristina helped the team define three categories for project disposition. James recalls, “The first were ones that made sense to keep in and move forward with. The second were projects at the end of their lifecycle, that made sense to transition out of. Then we had this middle set of projects: in Florida, we just punted and said, ‘You know what, maybe they belong in Innovation, maybe they don’t.’ In the Caribbean, we actually defined that category as ones that need to be reformulated. These projects were not meeting the same goals or delivering the same level of results as the first group, so they needed to be changed. After reformulation they could end up being on the ‘transition-out-of’ list later on or they could be moved back into the ‘go-forward’ one, or they could end up being ‘innovation.’
“In one example, the Caribbean had six different instances of one project, evaluating each instance individually, and each of those instances was quite small overall. They had a similar project that was going on in a different geography where they looked at all those instances together as a network of sites instead of individual pieces. It created a much bigger result. One workshop recommendation was to see how they manage the six projects as a single project. It would take a different management approach to the project, but it could really change the outcome and the people and nature benefits that would come out of that.”
In James’ example of reformulation, small projects got aggregated to make a bigger whole. Vera Agostini, Director of Climate Adaptation for the Caribbean Division of the Conservancy, recalls a reformulation in the opposite direction. “We approached some projects as the moon, the sky and the planets, way too big for them to be workable. In particular, with our fisheries product, we realized that the structure of the product is really quite dependent on who the audience is for that product. A hard moment there was taking a product that’s pretty big and unwieldy and breaking it into manageable pieces.”
Reformulation increases the value of existing projects, but not all projects can be salvaged. How would project leaders who had spent years of their life on a project engage with the notion of transition? Maxene recalls, “It wasn’t like somebody was asking us, ‘well do you want to let this go or not?’ After going through all the analysis and all the discussions, it’s the managers themselves saying, ‘okay, now I realize I need to let this project go.’
“I’d given up two projects of my own in Haiti so that we could focus on Three Bays National Park in Northeast Haiti, where we could contribute the most value. At first, it’s really painful to let go of your projects. I was very attached to one of them and had much hope that I could do something there. But then I realized that there are a lot of challenges that I didn’t quite take into account. I was acting out of passion. We conservation people are very passionate about what we’re doing. Although it was painful at first, I knew that there was solid analysis and good thinking behind it, not just that we impulsively decided to leave the project. So I actually felt at peace about killing my own project!”
James says, “The other thing that struck a chord with me in particular is whole project lifecycle management. A lot of times, we don’t necessarily think about our projects ending. Project lifecycle management and decision analysis really makes you think about the decision to renew, reformulate or retire: deciding whether a project should get continued support, or needs to be reformulated with innovation within the project to get going again up to a bigger scale, or has peaked and now makes sense to sunset.”
The Conservancy’s approach allowed both Florida and the Caribbean to build upon the progress the other team had made and leverage each other’s strengths. Shenique Albury-Smith, Director of the Conservancy’s Bahamas Program, remembers wondering, “How come they (Florida chapter) were able to leapfrog a little bit ahead of us (Caribbean Division)? One reason was that we had to cross some hurdles that they didn’t have to by the time it was introduced to them. They were able to use our work as a starting point.”
Another critical factor in making this work was leadership commitment. Temperince recalls, “I knew from the comments from my staff, and frankly from other places that I’ve worked, that if I didn’t make it clear that I was committed to this process and that this was my expectation, people would find good reasons to explain why they were too busy to do it. Also, I had to free up time for them to do it. The time to do this work wasn’t additive, and this is really important. People who are already working 70 hours a week don’t need a new directive, ‘work 80 hours now.’
“My directive was, ‘I know your load is full, that’s part of why we’re doing this. It’s so important that we do this now so that we can address that load issue. I am willing to delay deadlines, take work off your plate in order to give you the time that you need to do this now so that we can move the needle’. My staff told me later, ‘That was incredibly important. You made it clear that this was such a priority that you were willing to put off other tasks that are pretty fundamental to the work that we do.’”
The Conservancy’s motivation for this effort has been to increase the impact of its conservation work, an outcome that is yet to be realized. However, several intermediate outcomes have been realized:
· Improved focus on projects with high impact
· Increased understanding of the three project classes and their purposes
· Better workload balance through transitioning out of non-critical projects
The outcomes that have been realized will directly contribute to the success of the conservation projects that serve the Conservancy’s mission.
Kristina cautions that this initial success is the opening stage of a longer journey. “Our goal is to come to a point where our executives and our project managers become comfortable in using the metrics in their routine decision conversations, and we are not fully there yet. It’s critical to acknowledge that we don’t have it all figured out. We constantly have to remind ourselves that Decision Analysis is a continuous and iterative process and something you have to do to build the good decision-making muscle.”
###
This is a joint article written by The Nature Conservancy and SmartOrg on their experience together.
[1] Somik was aided by SmartOrg intern Esteban Guerrero during this phase of the project
