Five Myths about Impact Evaluation in the Humanitarian Space

@WFP-Saikat Mojumder

Impact Evaluation in the humanitarian space Five Myths about Impact Evaluation in the Humanitarian Space

Impact evaluation has a long history– but can it be used in humanitarian contexts?  We’re busting five common myths below.

  1. Humanitarian contexts are too difficult for impact evaluations

    Yes, conducting an impact evaluation becomes particularly challenging in crisis- and conflict-affected settings, but it can be done.  The evaluation of Impacts of the World Food Programme’s interventions to treat malnutrition in Niger found that children who received both food for assets (FFA), and treatment and prevention assistance were 20 percent less likely to suffer moderate acute malnutrition (MAM) than children receiving no assistance. The World Bank’s impact evaluation of Afghanistan’s National Solidarity Programme – with WFP as an implementing partner – is another good example. In contexts with levels of insecurity, WFP regularly relies on community-level committees to help distribute food.  The impact evaluation in Afghanistan had a good question: what types of membership in the committees results in more equitable food distribution to those who are most in need?  Testing this aspect of implementation helps WFP to understand how to best use community-level institutions to distribute assistance in times of high insecurity.

    WFP just launched its first Impact Evaluation Strategy, an exciting step towards improving how the organisation produces and uses evidence.  The strategy calls for impact evaluations that can respond rapidly to evolving contexts, and which harness the best tools and technologies to address challenges unique to the humanitarian space. 

  2. Humanitarian assistance is too unpredictable for an impact evaluation

    We recognize that humanitarian assistance often needs to be deployed quickly and unexpectedly. That’s why we’re investing in what Dr. Paul Christian at the World Bank’s Development Impact Evaluation (DIME) unit calls “the value of being there”. His research on the impact of cyclone Phailin in Odisha, India illustrates this. Since DIME was already supporting the country to generate evidence of their programming, they had data before the storm hit.  The resulting evaluation shows that women shouldered more of the negative effects of the storm, but that the rural livelihoods programme in place helped to mitigate some of the reductions in women’s consumption.

    The WFP Impact Evaluation Strategy recognizes the value of being there. With support from Germany’s Federal Ministry for Economic Cooperation and Development (BMZ), we’re developing a data ecosystem in the Sahel where frequent data collection feeds into a publicly available dataset.  WFP and our partners can use this data to generate evidence on the interventions in an area where climate-related and economic shocks are forcing migration, making it difficult to collect data. Want to follow our progress (or contribute your own data to the ecosystem)?  Let us know through the comment box below.

  3. It’s unethical

    We won’t do it if it is unethical!  The key to ethical impact evaluation is finding opportunities where rigorous impact evaluation methodologies can be applied to answer relevant questions.  In WFP, we often see this happening in a few ways:

    • When a country is trying something new or in a new context.  Sometimes an intervention is new and should be tested before expanding or replicating in a different context. This is the prime opportunity for learning – impact evaluation can help us answer questions to make sure we don’t make any mistakes when the country is ready for full implementation.
    • When we want to test different ways of helping (and don’t know which is better) an impact evaluation doesn’t always need a pure control group (i.e. without any WFP assistance).  Sometimes we want to compare different ways of achieving the same outcomes.  In this case, the country office delivers assistance to all those in need (of course!) and varies the implementation style to learn which approach works best. This can also help WFP understand the most cost-effective strategies.
    • As WFP works across the humanitarian and development continuum, more people in a country could benefit from WFP supported interventions but WFP starts with limited resources.  Our country offices don’t always start with all of the financial and human resources necessary to deliver all assistance to the entire population.  When a country office can only reach a limited number of beneficiaries to start, we think of ways to provide assistance in a way that allows us to compare outcomes with those not yet reached by the programme and improve targeting strategies.

  4. It’s too expensive

    Try not having the answers.  Remember: impact evaluation allows us to test what kinds of changes we can make to programmes to make them more effective.  These optimizations are even more important when the assistance may be life-saving.  It’s important to remember the funding gap to meet all humanitarian needs is hovering around 50%.

    In addition, the Impact Evaluation Strategy calls for the use of new technologies which can help us collect data in hard-to-reach areas to ensure we get high quality data with limited resources.  And let’s not forget – WFP already collects a wealth of information on the ground as part of their routine monitoring activities.  As we build capacity with our country office colleagues, we will explore how to make sure this monitoring data is fit for multiple purposes, including impact evaluations, to get the most value out of the data.

  5. We already know what works

    Sure, in the most basic sense: food is the only solution to extreme hunger.  But WFP, and many other humanitarian organizations, are doing more than just giving handouts.  We’re concerned about sustainable transitions, promoting gender equality, helping communities respond to shocks and adapt to a changing climate – to name a few.  In fact, our Impact Evaluation Strategy identifies the evidence areas that WFP colleagues have flagged as high priority, and develops ‘windows’ of impact evaluations which look at similar questions across contexts to build a body of evidence on how humanitarian assistance works best, why, and for whom. The first windows are about cash-based transfer and gender, and climate change and resilience.  

    Want to know more?  Find our new Impact Evaluation Strategy here, or reach out to us using the comment box below.