Impact Evaluation: How far have we come?

When I embarked on my dissertation on impact evaluation work 15 years ago, I observed the new trend of rigorous impact evaluation in international development. Criticisms of aid effectiveness have been raised for decades, for example the Paris Declaration on Aid Effectiveness (2005) called for evidence-based delivery of development interventions. Impact evaluation has become a valuable tool to provide greater accountability and determine the true effectiveness of development interventions. However, the international evaluation community has not yet reached consensus on the ideal tools and methodologies to be applied by these impact evaluations.

Startling the evaluation community. 

In 2006, the Center for Global Development (CGD) published the report “When Will We Ever Learn?” to make things seemly clear, calling for more and more rigorous impact evaluations. The CGD referred to the low quality of up-to-date evaluations, lacking the needed rigour to make causal statements. The new impact evaluations would need to be able to directly attribute the “net effect” to a development programme. The report referred to the medical model of clinical trials as medical standard: “no responsible physician would consider prescribing medications without properly evaluating their impact or potential side effects” (p.3). This experimental approach was applied to evaluations of development interventions. The so-called randomized controlled trials would randomly allocate the subjects to treatment and comparison groups – thus creating a counterfactual: “Impact evaluation asks about the difference between what happened with the program and what would have happened without it.” (p.12). This “net effect” typically focuses on a single number (going up or down) and its confidence interval – which opens the door for new criticism and suggestions for alternative methods. The evaluation community responded swiftly: the bi- and multilateral Network of Networks for Impact Evaluation was formed to provide guidance on impact evaluation, not just on the net effect of experimental trials, but on an integrated methodological approach. The International Initiative for Impact Evaluation (3ie) was launched to develop evidence and promote a repository of impact evaluations.

The continuance of rigorous impact evaluation. 

More recently, several multinational organizations launched impact evaluation strategies. For example, UNICEF created a Strategy and Action Framework on Evaluation of Impact (2022–2025). UNHCR is about to publish its strategy. WFP launched the WFP Impact Evaluation Strategy (2019–2026). The terminology is similar to the one from the CGD. The strategy calls for “robust evidence” (p.4) and “rigorous impact evaluation” (p.15), measuring the “net effect” of an intervention to understand whether something works (p.6). It calls on the “academic research organizations” (and not evaluation shops) to provide expertise in impact evaluation (p.24). Among others, the strategy refers to:

• the 3ie (see above);
• the Abdul Latif Jameel Poverty Action Lab (J-PAL). J-PAL’s founders Esther Duflo and Abhijit Banerjee were recipients of the Nobel Prize in Economic Sciences for using experimental methodology in international development; and
• the World Bank’s Development Impact Evaluation (DIME) unit, which has excelled in conducting experimental evaluations across the world.

WFP’s approach is a bit more nuanced, as it also refers to the WFP Evaluation Policy (2016–2021), which aligns the definition of impact evaluation with the OECD DAC definition of impact as “assessments of the positive and negative, direct or indirect, intended or unintended changes in the lives of affected populations in receipt of WFP interventions” (p.8). It also aims to harness the “best possible tools for capturing and analysing data” about “what works best, how, and for whom” (p.15).

However, the document then focuses on “counterfactual” and “net effect” – terms that resemble the experimental definition of impact, as used by the CGD a decade earlier and promoted by DIME and J-PAL. In short, there seem to be different impact evaluation strands intertwined in a somewhat unresolved way. It would be important to clarify the exact stance in the following iteration of the WFP strategy.

How far have we come? 

Over the past 15 years, the impact evaluation debate has not been fully settled, but has led to some approximation of perspectives that any (experimental) impact evaluation would benefit from other methodologies to increase relevance and use. Based on my analysis of the impact evaluation debates, I suggest the following good practices:

1. Weigh different possible impact evaluation tools: Ideally, at the beginning of each impact evaluation, there is a question for possible tools. Consider not just the experimental approach, but the full set of social science methodologies that contribute to causal analysis.
2. Evaluate early: Make sure that evaluations start early, ideally in the planning phase of interventions, and not during or even after implementation. This also ensures that the evaluators can collect baselines and adjust evaluation processes, if needed.
3. Understand impact (qualitatively): Every experimental or quantitative evaluation includes qualitative components. For example, the evaluator needs to (qualitatively) clarify evaluation questions, assess prior knowledge, interpret findings, etc. Evaluators need to have interpretive-qualitative skills when understanding and assessing the impact of an intervention.
4. Expand the evidence base: An experimental impact evaluation primarily answers the “what” question, but not the “why” and “how”. Therefore, any quantitative impact evaluation benefits from integrating qualitative tools to provide a richer picture of impact. Using multiple tools from the methodological toolbox in tandem ensures more robust evaluation findings than solely relying on a single method.
5. Ensure relevance: By combining impact evaluation tools, evaluators make their evaluations more relevant to stakeholders. Applying impact evaluation findings to other contexts or moving from a pilot to a larger implementation requires a strong theory of change – a task that is qualitative in nature.
6. Create multi-methods evaluation teams: Combining skill sets on an impact evaluation team makes it possible to navigate the various methods.

By following these good practices, impact evaluations will more likely make an impact.