Impact & MLE 101: How to plan impact measurement

This post is part of a series on impact planning.

How do you actually measure change and impact? In this section, we’re going to look at how to make methodical plans to confirm evidence of change. We’re going to start with the details, and move to the overarching thoughts.

Continuing the example we’ve been building on in this guide: we understand that we have outputs, outcomes and (fleshed-out) actors. The next question is, how do we monitor change across this map?

We can monitor outputs and outcomes by using indicators. Indicators are metrics that provide quantitative and qualitative information on our activities. An indicator of our first output might be “number of men/women trained” or for our first outcome, “percentage of population using technology”.

You’ll likely have multiple indicators for each activity. More can build a better picture but concise and critical indicators will sharpen your data collection. Come up with a list of all potential indicators you could use. Narrow down to the most essential.

You can make these indicators up yourself. Alternatively, there are many examples of indicators online people have used before. The WorldBank indicators give some examples and you’ll quickly find indicators in other impact reports or online. If you find indicators elsewhere that fit with your project, that might also give you something to compare against. This idea of comparison brings us on to a very important point.

Indicators alone are not change, nor impact.

‘Impacts’ are how much those indicators have changed against a comparator.

Imagine we have a new project that aims to grows bigger trees. Our chosen indicator is, “height of tree”. If I measure the size of all the trees in our project area, all I have is the height of lots of trees. I have an indicator, but no idea of change.

So we need a comparison. Let’s say we use time. Then we can ask the question how much did the tree grow?

Great, now we can use our indicators to get a picture of change. Let’s say we measure the heights of trees before the project starts. This is the baseline measurement. We might measure tree height at a mid-point and at the end of the project. When we compare the indicators at these time points, we can confirm if the trees did get bigger. Let’s say they do. Can we say the project made the trees grow bigger?

No, we can’t.

We can’t say for sure that the project made the trees grow bigger because we’re only measuring trees in the project. It’s part of the story, but not enough.

For example, maybe all trees generally grew that amount in that time. Maybe the project trees actually grew less than other trees.

To measure the effect of the project, we need a comparison where the project took no action or wasn’t in effect. We call this comparison the “counterfactual” or our “control group”.

This time we do our measurement over time and with our project and a counterfactual. From this data we see that both groups grow over time, but trees in the project grow taller. Our project impact isn’t between the trees on the left and right of the diagram. It’s the smaller difference between the tree on the top right and bottom right. This is how we can start to capture evidence of a project having impact.

However, impact planning doesn’t always start before the project (ex ante). Sometimes we want to capture impacts after the project has finished (ex post), where no MLE took place before or during the project. In these situations, you have to find a group to use as the counterfactual to compare against.

One way might be to use data from ongoing monitoring projects. For example, there might be regulatory records or other institutions that routinely track relevant data. With our tree example, perhaps there is a forestry commission that monitors tree growth across our area and others. Alternatively, perhaps we can access satellite records to observe our target area versus others over time. You might have to get creative.

Confirming impact after a project with no prior baseline or measurement will be more tricky. The main challenge will be finding a robust counterfactual, and the data to support it. This is not impossible, but things will be much easier if you plan and collect data from the start.

Methods of collection

In our tree example, we’re just using a simple physical measurement. When it comes to data capture for impact evaluation, there are a range of approaches you can use. Some of the most common include:

• Checking official records or data sets available online or kept by institutions.
• Conducting surveys with test and non-test groups. e.g. online, over the phone, in-person.
• Observation can be used to monitor behaviour or events.
• Focus group discussions are valuable where you are wish to capture shared views or areas of debate between participants.
• Interviews (structured or semi-structured) are particularly good for getting deeper accounts and capturing impacts that might be outside the expected indicators.

Two quick tips here:

1. Keep collection tools requiring participant time as short as possible. A common bane of MLE projects is respondents not replying. A survey with 5 questions that people complete is better than one with 20 questions that no-one answers. Tell people it’s short up front.
2. Make participants feel appreciated for their time. Thank people. Offer the option of being updated. Participants who are interested in the project will be more motivated to respond. Be wary however that this can create a bias in which groups are responding.

As mentioned above, a project could lead to unexpected changes — good or bad. While indicators give us some ideas of what to look for, we must not be so bound by them that we fail to notice other effects. Qualitative data capture methods are particularly good for uncovering these unexpected results. These kinds of findings can be used in case studies that add to impact reporting.

Measurement frameworks

We now have an idea of how we can use indicators to pin-point changes. But as we’re starting to see, things can get complex quickly. We will likely have:

• Multiple indicators for each output/outcome.
• Different methods of collection.
• Different groups and sub-groups to capture data from who might have have specific working requirements.
• Data collection happening at different, time-sensitive points.

Even on a smaller project, this can turn into a complex arrangements of activities. Luckily, we can use a measurement framework to help organise things.

A measurement framework is a plan that details what is going to happen when in our MLE cycles. In the measurement framework layout I will share, we have seven headers:

1. Output/outcome: The changes that have been identified corresponding to each of the actors in the list
2. Indicator: What data (quantitative/qualitative) will you use to assess whether the output outcome has been achieved?
3. Disaggregation: How might this data need to be disaggregated? e.g. does it need to be split by different groups?
4. Source: Where can this data be found?
5. Who collects: Who will be directly responsible for collecting this data from the source?
6. Frequency: When will this data be collected and how often?
7. Collection tool: What tool or instrument will be used to gather the data (e.g. household survey, in-depth interview, verify records, observation, mobile survey, etc.)

The beauty of this approach is that it can dramatically streamline the process:

• By filling in these columns, we can plan for what needs to be measured and how.
• We can send this collection plan to a calendar app, and delegate it to a specific person/group to collect.
• The collector receives a task with a has a targeted group and tool associated with it for data collection
• We can prepare each of the collection methods long in advance with this plan. We might even be able to automate some of the process. For example, our surveys could be set to automatically send to participants long before they actually land in inboxes.

Let’s make some very simple measurement framework for our farming technology example.

Here, we’ve quickly made a plan to record the indicators we are interested in. We have different teams associated with those indicators and overlapping collection methods to triangulate our findings.

Clearly this example uses a limited number of indicators and yours will be more detailed, but the principle is the same. Here’s a copy of the measurement framework spreadsheet you can use to save you some time. Once you have your logframe/theory of change, you’ll find it surprisingly fast to fill out the form.

And there you have it. Now you have a quick, clear tool that will help you plan even the most challenging of MLE projects.

Wrapping up

With this final section complete, this guide should have given you and intro of:

• MLE and impact language.
• What impact is… and what it isn’t.
• How to frame and map pathways to impact.
• How to populate these plans with actors.
• What to measure and when.
• A way to plan your measurement schedule.

With this, you have everything you need to develop a simple but effective MLE plan. Many other approaches exist, with their own merits, but following the steps we’ve covered should start you off.

We’ve covered a lot of content very quickly. The best steps now are you try these tools on your own projects.

What we haven’t covered is how to analyse your data, nor smaller details like “how to design a good interview”. The reason for this is that there are simply too many different factors for each unique project. This ‘concise’ guide would go on forever.

Luckily, there are lots of guides out there for these more specific needs. Google and YouTube are a great start. This sounds like a bit of a cop-out but if you capture great data, your analysis will be much easier. Follow the steps we’ve covered and you’ll be off to a good start!

So that’s the guide. I hope you found it helpful. If there’s anything I’ve missed out or could clarify on, let me know. Also if you know of any other helpful guides out there, let me know and I’ll link to them on here.

Good luck!

< Previous section | Guide introduction |