I was recently involved in a mid-term evaluation of a multi-agency 2016-2020 programming to enhance resilience in Karamoja, the least-developed region of Uganda.
This impact evaluation was particularly interesting in that it looked at various interventions by different agencies and also accounted for changing circumstances over time.
Karamoja, in the northeast of the country, is highly vulnerable to food insecurity, limited resources and climate variability. The Food and Agriculture Organization of the United Nations (FAO), the United Nations Children’s Emergency Fund (UNICEF) and the World Food Programme (WFP) have been working together in that area for more than twenty years and recently developed a resilience strategy to improve food security and nutrition for beneficiary households.
The planned interventions included Farmer Field Schools, school-feeding programmes and improved water, sanitation and hygiene projects.
Challenges and opportunities
The impact evaluation of the resilience-enhancing programming presented various challenges but, also, provided us with some key learning opportunities. For one, it gave us the chance to examine and strengthen the adoption of the FAO-Resilience Index Measurement and Analysis (RIMA) methodology, which looks at how households cope with shocks and stressors. The beauty of the RIMA tool is that it can be used for assessing the impact of different types of projects and programmes implemented by national and international actors. Additionally, it can be used to assess combined interventions, taking into account the many, interconnected components of household resilience.
In this case, the RIMA tool was contextualized and agreed on by the three UN agencies involved and fine-tuned to include indicators which reflected their respective programmatic efforts. The RIMA uses household surveys to offer a multi-dimensional and household-level index from approximately 30 variables.
A dynamic situation
The main challenge of this impact evaluation was accounting for the change in the UN implementation plan over time-between the (2016) baseline analysis and the (2019) follow-up gathering of data. The baseline analysis focused on the following groups of households:
- Targeted households - slated to receive UN interventions in selected areas of two districts;
- Within-district spillover - households located in the remaining areas of the targeted districts, which are not involved in the UN programming but which may indirectly benefit from it;
- Control - households located in one remote district where there were no immediate plans to implement the UN programming;
- Between-district spillover - households located in two districts where the UN programming was not planned;
- Different ethnicity - the households located in two districts populated with non-Karamojong households and no interventions.
Between the baseline and follow-up surveys, the implementation plan was modified and the three agencies decided to intervene in different locations of the region. Having collected the baseline data from a representative sample of more than 2,000 households, we managed to adapt the analysis to the implementation changes and to simplify the groups of the analysis to:
- Targeted households located in selected areas of four districts targeted by UN programming;
- Control households living in the districts where no interventions have been implemented by the three agencies;
- Spillover group composed of households living in different areas of the treated districts.
Another intriguing challenge grew out of the prevalence of pastoral migration in the region. There was a need to collect a panel dataset in this context and conducting interviews in the same households which were first interviewed back in 2016 was a challenge, due to household dynamics and migration patterns.
We worked on the tracking aspects of the survey and learned how to improve the process for future surveys, for example, by using the GIS household localization in the tablets for the enumerators. This means that, during field activities, the enumerators will now be able to rely on the Google Earth application (installed in the tablets) to indicate the exact baseline location of the households to be followed.
The lessons learned from the impact evaluation in Karamoja will be used for forthcoming impact evaluations using RIMA in other countries. It will be interesting, indeed, to see the results!
FAO (2017), Karamoja 2016, FAO Baseline Impact Evaluation Report N.02, Rome