Can we rely on retrospective baseline data in project evaluation?
Many of us who do quantitative evaluations for a living have faced the situation of being called in to do an evaluation for a project that’s already been implemented, but which has no (pre-project) baseline data that we can assess progress against. Even if a baseline survey was carried out among the participants in the project, it’s rare for the baseline to have included a suitable comparison group. This means that we can’t make use of comparative evaluation methods, such as matching or difference-in-difference analysis.
In these cases, evaluators often ask survey respondents to recall information about the situation they were living in at a point in time before the project began, and assess what changes have happened since. This approach has been endorsed by organisations such as the OECD, the International Fund for Agricultural Development and the International Federation of Red Cross and Red Crescent Societies. Nobody expects retrospective baseline data like this to be completely accurate, but we can assume it’s better than nothing, right?
Well, perhaps not. In a new paper in the Journal of Development Effectiveness (open-access version here), Simone Lombardini, Cecilia Poggi and I have investigated the accuracy of retrospective baseline data and how useful it is for quantitative evaluation. The data we’ve used comes from a survey carried out by an international non-governmental organisation to evaluate the impact of a project on women in two rural districts in Ethiopia. The survey respondents were asked to recall some key pieces of information about their living conditions — including their household’s income sources, their housing conditions, and what assets they owned — from a time before the project was implemented, nearly six years earlier. However, in this case, the same women had in fact been interviewed at baseline. This allows us to compare the retrospective data that was collected in the final survey with the responses from the real baseline data.
The results are troubling for anyone who has come to rely on retrospective data in project evaluations. We find that the information that survey respondents gave when we asked them to recall their baseline situation was much more closely correlated with their situation at the time of the survey than it was with their actual baseline situation. In other words, they tended to underestimate how much had changed since the time of the baseline. As a result, if the retrospective data were used to construct an index of households’ socio-economic status, nearly three quarters of the households would be allocated to the wrong quintile of the index. Relying on the retrospective data alone would have had a major impact on the evaluation results. Using one of the key indicators of success for the NGO project, we find that controlling for the retrospective data rather than the true baseline data leads to the impact of the project being considerably underestimated.
Of course, this single study is not conclusive evidence that we should always avoid using retrospective baseline data. We do not know whether our findings would be replicated in other settings. The survey in this case was asking people to recall information over a particularly long time period (nearly six years), and it’s possible that recall over a shorter period would be more accurate. Many of the respondents in the survey struggled to identify a memorable ‘landmark’ event that would help them recall the relevant period — but in situations where such a memorable event has occurred, this may also improve accuracy. As we discuss in the paper, there is also a lot of potential for experimenting with questionnaire design to see whether it’s possible to improve the accuracy of recall data.
For now, though, we believe that these findings should at least give evaluators pause before relying on retrospective baseline data. Using retrospective data in this way can damage rather than improve the accuracy of your results.
