Do’s and Don’ts of measuring research impact
“Measuring impact, particularly for research that involves people and is spread out over space and time, is never a straightforward matter. It is almost never as simple as collecting data before and after the intervention, since before-after estimates can be seriously biased by factors outside of the research. Even internally, selection biases and other confounding factors are often at play.”
Professor Paul J. Ferraro, economist at the Andrew Young School of Policy Studies at Georgia State University, said this while speaking on the issue of impact measurement at the World Agroforestry Centre (ICRAF) headquarters in Nairobi on 4 September. He emphasized, however, that despite the difficulties associated with impact measurement, it is the only way to tell for sure whether or not an intervention has done what we hoped it would.
Ferraro said that the best impact analyses try to determine what the situation would have looked like after the passage of time both with and without the intervention. “By evaluation through a contrast of status or trend with the counterfactual status or trend, researchers can measure actual impact,” he said.
With examples from impact analyses of a large residential-water-conservation initiative in southeastern United States; studies in Costa Rica and Thailand on whether protected-area proximity had any relation with poverty levels of communities; and public health campaigns, Ferraro showed that without careful attention to the methods used in impact analysis, results can be exaggerated, underrated, or even wildly off-course.
Ferraro recommends that impact analysis be carefully thought through at the project design stage, as it is exceedingly difficult to evaluate the impact of completed projects whose design was not powered for such analysis. He warned against the common confusion of impact evaluation with monitoring. “Monitoring is the process of measuring the trends and status of project performance indicators, while impact evaluation is the process of attributing changes in the trends and status of project performance indicators to the project actions separate from other factors.”
Ferraro took the audience, made up of World Agroforestry Centre researchers and partners, through some of the common research techniques that allow projects to measure impact with reasonable accuracy. These methods aim to control for common confounding factors and biases over space and time, and include:
- Simple randomization: candidate units (individuals, plots, households, etc.) are assigned to treatment and control groups on the basis of a chance mechanism, like a random number generator, and their outcomes are compared.
- Randomization in oversubscribed projects: Units to treat are selected through a lottery among eligible candidates.
- Randomized phase-in projects: Randomization is used as a fair way of determining the order of phase-in, in projects that don’t have the immediate capacity to recruit a large population all at once.
- Randomized encouragement: This is where a subset of all eligible candidates is vigorously encouraged actively to join the study, while the recruitment of the other subset is more passive.
- Discontinuous Eligibility Criterion.The project selects an eligibility criterion, such as a cut-off score, to generate treatment and control groups.
Even after impact assessment shows that an intervention works and is beneficial, impact evaluation must be built into roll-out programmes. “Science doesn’t stop when people and human behavior enter the picture: it’s even more important that we apply scientific principles to measuring success,” said Ferraro.
He added that yet another compelling reason to conduct long-term experiments that are well designed, publicized and documented is that “years or even decades later, someone will surely return to your experimental site and do follow-up research, and perhaps they will look for the long-term impact of your study.”
Professor Ferraro’s lecture, titled ‘The Social and Environmental Impacts of ICRAF Initiatives,’ was among three invited lectures during the Centre’s week-long Science Forum, 3-7 September.
Additional stories from the World Agroforestry Centre Science Week: