Thanks, interesting. I look forward to reading the full paper, but some initial reactions.
I completely agree that the value of the evaluation depends on how it is used and by whom. So evaluations of programmes that spectacularly fail or over-deliver (rather than those that roughly fine) are most useful, because they are most relevant to decision-makers outside the immediate circle. The problem there is that we don’t know this categorisation at the start. Which in my mind means that there should routinely be more investment in capturing some basic baselines (including landscaping and qualitative data), so that evaluation is more possible later. And it also means that how much to invest in evaluation should be assessed and reviewed over time. Plan A may not be the right plan.
Secondly, evaluations are more useful if there is some kind of benchmark to serve as context. In my field of social enterprise, challenge fund support, impact investment, a share of business failures is expected. But there are so few benchmarks to know whether the evaluated impact counts as ‘good’. And a lack of sharing data. Increasing access to data across programmes would be a public good that would increase the value of each individual evaluation.
Finally, the value of evaluations comes from informing new decisions. Monitoring and learning does this too, though with more focus on adaptive management while the project is on-going. I think this is just as important. But then we get a trade-off about whether to invest in M&E that is internal, external or both. Having experienced a range of set-ups, I think an internal team is so much better positioned to truly identify lessons as they emerge and feed them into management. External monitoring is bureaucratic. But evaluations are generally regarded as more rigourous and valuable if they are external. So this makes the question of resource allocation more challenging: how much to invest in both internal monitoring and external evaluation, and how to ensure synergy and efficiency between them. I’d love to hear more views on this balance.