What works in improving food security and nutrition in very poor communities?
Many African countries are heavily reliant on agriculture and characterized by high poverty rates, as well as food insecurity and malnutrition. Development projects adopt a wide range of interventions to address these issues and help rural communities access opportunities and resources, thus improving their livelihoods.
I would be interested in examples from your evaluation experience of projects that have been successful in improving food security, and in particular food access, in extremely poor communities: could you please share them? What approach did these projects adopt? What data and indicators (quantitative and qualitative) you were able to use to monitor if and how progress in food security was achieved?
If you know of evaluations that are relevant to these questions please share them (reports or synthesis).
Dr Emile N. HOUNGBO, PhD
Lecturer & researcher at the National University of Agriculture, Benin
I would concur with Eoghan and Mustapha on their experience in evaluating CA projects in Africa (and similar agriculture technology projects in Australia and third world countries).
Farmers are usually prompt adopters of appropriate technologies that take account of the risks and benefit profiles of the innovations. Subsidized inputs unless used in a very strategic and transparent way supported by proven culturally appropriate extension and communications programs for the costs/benefits will not lead to sustained change.
Mustapha MalkiIndependent consultant
Thanks, dear Eoghan, for taking time to go through my contribution and give more information about the evaluation.
The picture on introducing CA and get it adopted by farmers is very similar to what we have done for the last 3 decades in technology dissemination and adoption (intensive package on cereal cropping, use of quality seeds, mechanization, herbicides use, water-saving irrigation techniques, etc.). That general picture shows always some of the following aspects:
1. Project's technical staff are very enthusiastic to show their "successes" in the field by showing large numbers of farmers being enrolled by the project, and jump without hesitation to consider as a huge rate of technology adoption. They are very defensive when one tries to ask them questions if they took the time to know in deep their beneficiaries.
2. Farmers are keen to apply a new technology when someone else is covering the cost. But when the project is closed, then we see properly what is happening among farmers. Most of time, farmers who participated in a closed project start asking when the next new project will start and if they will be part of it, as if the closed project was just a game and then the game was over (I am becoming a bit cynical on this).
3. Little is done in terms of evaluation of the project outcomes, impacts, sustainability of both and so forth...
I am telling you this because I was involved in 2013 a 4-year Maghrebin CA project funded by Australia and implemented by ICARDA in Algeria, Morocco and Tunisia. I was involved in setting the M&E plan for that project and trained a bunch of Maghrebin researchers and dev practitioners on Results-Based Management so that they could make a good use of that M&E plan. All social actors involved in that project praised the work done (M&E Plan + RBM Training), especially the Austalians who were very keen to put a strong pressure on ICARDA to setup the M&E Plan. But the project was closed after 4 years in the same as I saw many projects closing (you can imagine the picture - business as usual).
But, in your case, and the case of your CA evaluated project, I am happy to see that you paint in your message the picture as it is in reality, i.e. that CA was not that "rosy" technology that could fit most farmers in Africa, especially that is was applied in a "one-size-fits-all" approach, with a little knowledge - to not say "no" knowledge - on the beneficiaries, and that the case presents some shortcomings that you are not hiding. And what and how evaluation has to do. Good to read a balanced contribution on a new technology.
As for the issue of sampling, especially with a "fixed constituency" for 4-5 years between baseline and project end, it is always a tricky issue to get that required robustness in our survey. But you tackled the issue through triangulation, using multiple sources of data, and honestly I would have go the same way. But locating 317 farmers among 385-390 at the end of the project is quite an endeavour by itself. That's why I mentioned in my previous contribution the need in such cases to make the sample bigger at the baseline in order to cover such turmoil at the end.
Finally, the way you presented the things made me more curious and "hungry" to look at the evaluation report. Without engaging in a formal commitment, I will download the evaluation report for which I am very thankful to you and try to squeeze some time to read (summer time is rushing away and missions and travels will start again very soon in September).
Good luck and kind greetings
Thank you for you insightful response regarding the methodologies used in the Final evaluation of the Conservation Agriculture Scaling Up (CASU) project in Zambia.
On your first point regarding the sustainability of the adoption of conservation agriculture, indeed this was one of the primary concerns of the evaluation team. Although all data, both qualitative and quantitative, points to high levels of adoption of the techniques amongst the beneficiary farmers, the history of conservation agriculture promotion in Southern Africa suggests that this may not be the case in 5 - 10 years from now. The evaluation sought to highlight some of the reasons that may result in farmers 'disadopting' the conservation agriculture techniques. Among these:
One important aspect of the CASU project was that participating 'lead farmers' were given a package of inputs and tools through a voucher scheme to allow them to demonstrate the benefits of using conservation agriculture. There were no inputs distributed during the final season of the project, although this did not affect farmer's application of the techniques - which is somewhat encouraging, as it shows that farmers continued using the techniques, even when they had to buy inputs themselves.
However, while the quantitative data pointed to high rates of adoption across the different conservation agriculture principles (minimum tillage, soil cover, crop rotation), the qualitative data painted a more nuanced picture - farmers had strong reservations regarding the extra labor required for conservation agriculture, particularly with regard to weeding, and many farmers said they would only continue applying the techniques if they could afford to buy chemical herbicides. Furthermore, those farmers who claimed to be strongly committed to practicing conservation agriculture noted that they would only do so on a small area of land, as it was too labor intensive to do so on a larger-scale, particularly without access to mechanization services.
For these reasons and more, the evaluation offers some critical findings related to the sustainability of the intervention, and calls for further research into alternatives for weed-control and the provision of mechanization services.
On your second point, regarding the sample-size for the household survey, indeed, the original proposal for the study outlined that the sample size for the final survey should be no smaller than 385 (calculated using a margin of error of 5% and a confidence level of 95%, based on the total population of beneficiary farmers). The intention was to revisit the same farmers who had been interviewed during the project's baseline survey. In practice, this was more difficult than anticipated, and the researchers could only locate 317 of the proposed 390 respondents. While this may have some implications on the robustness of the survey findings, it is important to recall that this is one of several data sources for the evaluation, including the project's own substantial M&E data, as well as the extensive qualitative data collected by the evaluation team across the different agro-ecological regions of Zambia, and all data was triangulated before drawing findings and conclusions. The report from the University of Zambia has been published as a standalone annex (Annex 2) to the main evaluation report, and you can find more information there.
The full evaluation report, as well as the annexes, are now available on the FAO Office of Evaluation website at: http://www.fao.org/evaluation/evaluation-digest/evaluations-detail/en/c/1147949/
Thank you again for your very insightful comments.
Office of Evaluation (OED)
Food and Agriculture Organization of the United Nations Viale delle Terme di Caracalla
00153 Rome, Italy
Thank you for initiating such a wonderful discussion around evaluation of Food Security, Nutrition and Agriculture programs, it indeed is a crucial issue, especially in the context of poor countries, and communities reliant on agriculture as a means of subsistence.
I would like to briefly share the followings from my experience. I served as a National Consultant in 2016 to conduct an evaluation for a SIDA (Swedish International Development Cooperation Agency) funded project of UN-ILO (International Labour Organization), the project was a job creation project (Road to Jobs).
The overall goal of the Road to Jobs (R2J) project is ‘More and better jobs in selected Northern Provinces of Afghanistan to contribute to improved livelihoods and poverty reduction. The project also aims at addressing the underlying causes of poor market systems performance in selected agricultural subsectors.
The summary of evaluation findings published by ILO is available at: http://www.ilo.org/wcmsp5/groups/public/---ed_mas/---eval/documents/publication/wcms_571956.pdf
I hope this is helpful.
Find below links to WFP website. There is substantial material on food security and nutrition in vulnerable communities.
Also see the links to the WFP evaluation reports and to the global Food Security Cluster
Register on Food Security Cluster website to receive the newsletter.
Thania de la Garza Navarrete
To promote the use of evidence on the effectiveness of different public policies CONEVAL (the National Council for the Evaluation of Social Development Policy, Mexico) developed a strategy to share relevant evidence for public policy in a friendly way, especially for policy makers. The purpose is summarizing available evidence from impact evaluation through accessible tools to inform the decision-making process.
In this email we have attached two Excel files written in Spanish:
1. REM_CONEVAL_Food Security https://dgroups.org/?0mb4xzf0
2. Food Security Evidence https://dgroups.org/?0mb4xzf0
The first file contains an impact evaluation search regarding interventions, policies and programs focused on food security in Latin America. It has two sheets, one called “IE Codebook” that includes the 72 impact evaluations found. The other one called “Evidence Gap Map” classifies the evaluations compiled during the search. The columns show the food security dimensions (availability, Access and use) and the rows, the intervention types (Indirect or Direct). Each cell contains a number representing the evaluations found, matching those categories. Also, these cells have a comment specifying the evaluations considered.
The second file consists in a synthesis of evidence that results from the search of studies regarding food security. This search was made by the combination of key words in both languages: Spanish and English separately. This file has 4 sheets: “base de datos”, “Tabla A1.1”, “Tabla A1.2” and “Resumen”. The first sheet (“base de datos”) contains all the documents found during the search, as well as their details. The studies considered are highlighted in blue. The second and third sheets (“Tabla A1.1” and “Tabla A1.2”) include the results of the key words combination search for both languages. The last sheet (“Resumen”) is an abstract of the selected studies and their main findings.
I hope this information will be useful for you.
Thania de la Garza Navarrete
Emile Nounagnon HOUNGBOAgricultural Economist, Associate Professor, Director of the School of Agribusiness and Agricultural Policy National University of Agriculture
Dear Eoghan, Mustapha, Isha and Jackson,
Thank you for your contributions, I will take them into account.
Please keep sending examples of projects that have been successful in improving food security in poor communities as well as approaches, data and indicators used to monitor progress, based on your evaluation experience.
Isha MirandaVisiting Lecturer and Independent Evaluator Independent Consultant
Dear Dr Emile
I trust this mail finds you well. As an answer to your questions, I have done a few national and cluster studies on this subject. Please find attached some indication on strategies and issues to achieve food security and nutrition. You can create indicators based on the tables.
I hope this makes sense to you.
Keeping in touch
This is an excellent piece of information on handling of adoption studies and sample size determination.
Mustapha MalkiIndependent consultant
Dear Mr. Molloy,
I have read with a great attention your contribution referring to the evaluation of CASU in Zambia. I must congratulate your department for such an achievement. However, I have two points to make here.
You mention at the start of your contribution that the entire population of the Conservation Agriculture project is "targeting over 300,000 smallholder farmers". That is the entire population of the project. You also mention that "the main focus of the evaluation was to assess the extent to which conservation agriculture has been sustainably adopted by Zambian beneficiary farmers … also sought to assess what outcomes were evident (positive and/or negative) from the project’s activities, and what were the impacts on food security, income, and soil health"
The first point I want to raise concerns the adoption study that you highlighted in your message. Though I don't have all details about how such a study was conducted and what results it did achieve, I would like use this opportunity to share some experiences on adoption studies, a sort of outcomes evaluation and if these outcomes are sustainable over time. Everett Rogers, one of the gurus on technology adoption by farmers, instruct us not to check the adoption rate at once or at any time. Adoption studies require that one is aware of the technology adoption process among farmers in order to understand how to work with adoption studies and set up appropriate protocols to study technology adoption among farmers. I saw many of adoption studies giving high rates of adoption at the end of a project and very low numbers of farmers are still keeping the technology 5-10 years after the end of a project. This is because what seems adoption to researchers is just experimentation to farmers, so real adoption for farmers will come far away after that moment of project end.
The second point I want to raise concerns the household survey undertaken by the University of Zambia and the sample size used by the research team. Besides other activities conducted within this evaluation (among which focus groups with 650 beneficiary farmers), you mention that "a household-level impact assessment survey to collect quantitative data amongst a sample of over 300 farmers, in order to assess progress against the baseline survey".
Nobody can deny that a survey can only be truly valuable when it is reliable and representative for the entire population of project's beneficiaries. This is why determining the ideal survey sample size with robust external and internal validities is quite important as it will help the research team to infer and extrapolate the results obtained on the research sample over the entire population of the project's beneficiaries.
Using a correct survey sample size is crucial for any research, and project evaluation is a research. A too big sample will lead to the waste of precious resources such as time and money, while a too small sample, though it can yield sound results (strong internal validity), but will certainly not allow inference and extrapolation of its results on the entire project population (weak external validity).
So, the sample size cannot be by how much a research team can handle but on how accurate the survey data ought to be. In other words, how closely the research team wants the results obtained on a sample to match those of the entire project population.
In statistics and statistical probabilities, we use two measures that affect the accurateness of data and which have a great importance as for the sample size: (1) the margin error, in most cases, we use 5%; and (2) the confidence level, in most cases, we use 95%. Based on these two measures, and given the population size, the research team can calculate how many respondents (people who might completely fill the survey questionnaire) it may actually need; that is the survey sample. Beside all this, the research team must consider a sufficient response rate – that is the number of "really exploitable" survey questionnaires – so that they include additional questionnaires beyond the sample so that the research team has sufficient number of completed questionnaires to exploit. The table can give an idea on the sample size for a project population of 300,000 individuals. For example, if we target 380-390 "exploitable" questionnaires, we allow 20-25% more questionnaires so that the survey is not put at risk of weak robustness.
As a conclusion, I believe that the sample size for the mentioned household survey, as part of the undertaken CASU evaluation, was a bit lower than what a probabilistic law would accept. Of course, this statement has no consequence on the results obtained within the sample as such, but the survey findings cannot be strongly and robustly inferred and extrapolated to the entire population of project's beneficiaries because of the weak external validity of the sample, due to no respect of the principles of probabilistic law.
Dear Dr. Emile Houngbo,
The FAO Office of Evaluation has just recently conducted the final evaluation of the Conservation Agriculture Scaling Up (CASU) project in Zambia, funded by the European Union, which had the objective of sustainably increasing crop productivity and diversity through widespread promotion of the three principles of conservation agriculture across the entire country, targeting over 300,000 smallholder farmers. The main focus of the evaluation was to assess the extent to which conservation agriculture has been sustainably adopted by Zambian beneficiary farmers, although the evaluation also sought to assess what outcomes were evident (positive and/or negative) from the project’s activities, and what were the impacts on food security, income, and soil health.
The evaluation used a mixed methods approach – using qualitative and quantitative methods – to assess project results. The evaluation team met with over 650 Zambian farmers across the different agro-ecological regions in Zambia during April 2018, conducting focus group discussions and key informant interviews with project stakeholders from FAO, the government, private sector and civil society. In addition, the University of Zambia, on behalf of the FAO Office of Evaluation, conducted a household-level impact assessment survey to collect quantitative data amongst a sample of over 300 farmers, in order to assess progress against the baseline survey that had been conducted in 2013. Meanwhile, the evaluation drew upon the monitoring and evaluation (M&E) data collected by the project team throughout the project’s implementation period.
Farmers met by the evaluation team reported (in focus group discussions) that food security and dietary diversity has greatly improved, and that nutrition objectives are on track. The focus on including women in the project activities has led to more legumes being grown and maintained for home use. In the focus group discussions there was clear feedback that legume quantities and varieties have increased. Many households grew small quantities of groundnuts earlier (typically considered a woman’s crop), but with the focus on legume production by the project, farmers noted that they now have a wider range of crops – now they are growing soya, cowpeas, pigeon peas, Bambara nuts, other bean varieties as well as groundnuts, depending on the area. They said that earlier they had many hungry months most years, but since CASU began they have maize year-round, and most households also have legumes available year-round (or at least ten months of the year). Many households have also begun to grow vegetables at home.
One of the project’s impact indicators was household dietary diversity (HDD) and women’s dietary diversity (WDD). This was assessed in the project baseline questionnaire in 2013, by asking farmers to select from a list of 9 food groups those they had consumed in the past 24 hours. At the time of baseline data collection, the average Household Dietary Diversity Score (HDDS) was 7.79 (out of a possible score of 9) and the average Women’s Dietary Diversity Score (WDDS) was 5.09 (out of 6). These were already quite high (therefore there was limited room for improvement), but showed increasing trend over the project period. The project’s monitoring data (collected at regular intervals between 2013 and 2018) showed an improvement in HDD and WDD. Final scores reported by the project monitoring system were 8.31 and 5.41 respectively.
The findings of the University of Zambia household survey corroborated the projects monitoring data, and showed that the project’s activities have had a positive effect on both the HDDS and WDDS.
All data sources (i.e. the project M&E data, the University of Zambia survey, as well as the qualitative focus group discussions) showed that applying conservation agriculture techniques had a significant and positive impact on maize yields, for both men and women, and in general there was in increase in yields for other crops too (sorghum, soybeans, groundnuts and cowpeas). The evaluation concluded that the project had successfully supported improved nutrition and food security through increased farm yields and increased production and consumption of legumes.
However, while adopting conservation agriculture was found to have a significant impact on production, and an increase in food availability was certainly observed among project beneficiaries, the evaluation found that farmers faced significant barriers in fully adopting conservation agriculture techniques – particularly with regard to labour constraints and the increased burden of weeding under a conservation agriculture farming system. Meanwhile, a lack of reliable markets, particularly for legumes, may deter farmers from fully incorporating legumes into crop rotations, which not only affects the sustainability of adopting conservation agriculture, but also has implications on the sustainability of food security outcomes observed during the project’s implementation. The evaluation recommends that future projects of this nature should incorporate linkages to markets and the private sector, while promoting further research on labour-saving techniques including mechanization and sustainable weed-management practices.
The full evaluation report and annexes (including the study conducted by the University of Zambia) would be available online in the coming days at http://www.fao.org/evaluation/en/.
Food and Agriculture Organization of the United Nations
Viale delle Terme di Caracalla
00153 Rome, Italy