Evaluating agroecology: what is your experience?
Dear EvalForward Community,
As evaluators working in the agriculture field, we increasingly need to assess the environmental and socioeconomic sustainability of agricultural projects and programmes.
FAO has developed a tool called TAPE aimed to support the data collection and analysis of the transition of agriculture systems towards agroecological practices. For more information about its functionalities and potential uses, see my blog here.
I would like to invite members to share their experiences in evaluating the sustainability of agricultural, and specifically agroecological, programmes:
- Have you used or tested any specific tool?
- Do you think TAPE could be useful in M&E for your projects and programmes
Please feel free to send us references to tool and guidance via email or in comments section.
Thank you, and looking forward to speaking with you all!
Office of Evaluation
Food and Agriculture Organization of the United Nations
Viale delle Terme di Caracalla
Seda KojoyanUN FAO
Dear colleagues and readers, thank you for your audience and active participation in the discussion. It makes me really glad that more people know about TAPE now, and can exchange practices.
Several of you suggested mapping the TAPE criteria against existing evaluation frameworks, notably the DAC criteria. I also thought about this when writing the blog post. For instance, aligning the TAPE criteria on productivity to the standard evaluation criteria on effectiveness and impact; the TAPE criteria on income, added value and resilience to evaluation criteria of economic sustainability, and so on. However, in consultation with colleagues and prior to the blog’s publication, we thought a good first step would be to present the TAPE without this mapping, to avoid confusion (also because the placement of some of the criteria is really open to debate).
Some of the comments which are really technical on TAPE have been forwarded to the TAPE tool developers and managers at FAO. In the meantime, thank you Olivier Cossée and Laurent Barbut, among others, for your comments about the definition of the concept itself, as well as the consequent challenges. Despite the limitations, one powerful aspect of integrating an assessment and monitoring tool like the TAPE, is that “it offers an alternative to the mainstream evaluations approaches” (to quote Anna Maria Augustin’s comment).
Some concrete (and valuable!) examples of avenues for TAPE’s use from the discussion were:
Thank you, Elias Kuntashula, for sharing that you had a positive experience already applying the TAPE questionnaires in Zambia. And Malika Bounfour, how wonderful to know that contributed to a project evaluation where women empowerment, youth employment and traditional knowledge were all added to the assessment areas. It is interesting to also hear about monitoring agroecology practice in DRC (thanks Anne Scarpitta and Jean Marie Ruhanamirindi for sharing).
Finally, we welcome that colleagues, for example, Paul Mendy and Kewe Kane, are open to testing out the specific indicators proposed by the TAPE. Please let us know how it goes!
Malika BounfourPresident Association Ayur pour le Développement de la femme Rurale
Thank you very much Seda for interesting subject and for sharing the TAPE guideline.
I find the guideline very thorough and detailed. The tool does also give a definition of agroecology. I did appreciate that bees and pollinators were considered.
From a technical point of view, I would have some personal addings:
1. I would combine the scored 0 and 1 cells for crops (page 17). If the crop covers 80% of land, it is "monoculture";
2. For exposure to pesticides, I think the evaluator should look into the pesticides in stock on farm or territory. Both quantities and storage methods impact environment and health. Therefore waste management should be included;
3. Soil micro-fauna is indicator of healthy soil and should be considered .
From my experience, farmers and technicians consider mostly water use efficiency, pesticide and fertilizers application (type and quantity), crop diversity and soil management.
I contributed to a proposal for a project evaluation where we added women empowerment (decision making) and youth employment as well as traditional knowledge. TAPE can be used as a reference for separating areas/farms in transition to agroecology from areas/farms where agroecology is fully implemented.
With my best regards
Anna Maria AugustynInternational consultant
Dear Seda and Colleagues,
Thank you for this interesting topic for the discussion. The TAPE tool developmed by the FAO looks very promising and it would be interesting to see, whether it could be embedded into the evaluations of projects or programmes targeting agroecological transitions.
The tool is very much rooted in the systems thinking and because of this, it offers an alternative to the mainstream evaluations approaches. Maybe it would be helpful to investigate how it could be integrated with the popular frameworks using the OECD DAC, Theory of Change, Intervention Logic and Logframe approaches. In my view, the best would be to use the suggested indicators already at the project design stage.
On the other hand, the proposed indicators can be possibly helpful for facilitation purposes in the rural communities. For instance, when I worked with the project beneficiaries on developing indicators in participatory manner, I was sometimes missing a good background template with indicators to inspire them. The TAPE Tool could be a good resource for this.
In my experience, farmers and rural stakeholders often do not see a clear difference between agroecology and other system-type approaches, such as for instance conservation or carbon farming. The TAPE Tool could be possibly used to articulate more the similarities and differences.
Anna Maria Augustyn
Annette ScarpittaCo-Director and U.S. Representative, The Rwenena Project Independent Consultant and Congo Federation of Smallholder Farmer Organizations - S. Kivu (FOPAC)
Hi All - I am posting this on behalf of my colleague Jean Marie RUHANAMIRINDI, well-regarded acroecologist & community facilitator for The Federation of Agricultural Producers' Organizations of Congo in South Kivu (FOPAC SK):
Hello SEDA KOJOYAN,
Greetings to all of you who take the time to share their experiences on agroecological evaluation.
I am sharing with you the monitoring of agroecological agricultural practices carried out by FOPAC SK (Federation of Agricultural Producers' Organisations of Congo/South Kivu) in the DEMOCRATIC REPUBLIC OF CONGO; because in our opinion, we first need to define the evaluation reference system, i.e. what are agroecological agricultural practices? On our side we consider: agroforestry, permaculture, the use of organic fertilizers, the use of local or adapted seeds, water management and the integration of agriculture and livestock.
We don't have a specific tool but we base ourselves on the sustainability of the system:
- For the forest area, the degree of practicability of permaculture taking into account the biomass production power of intercrops. The upper natural forest is retained but reduced to allow for the integration of crops. We see in all cases the number of men, women and youth involved in the system.
- For the Ordinary Rural zone: the focus is more on agroforestry, agriculture-livestock integration practices, recycling of crop residues. An analysis of cultural and food traditions is also attempted for the definition of species (Acceptability). Integrated soil fertility management practices, protection of land capital and the degree of application of animal and plant pharmacopoeia.
- Wetlands: attention is focused on water management, whether the systems pollute water, the crop cycle and the amount of biomass left after harvesting. The protection of marshes and lowlands from the onslaught of the hills -
for FOPAC SK
Ir Jean Marie RUHANAMIRINDI
Olivier CosséeSenior Evaluation Manager FAO
Thank you Seda for highlighting the TAPE tool. I had heard about it from the SDG 2 review we did in 2019/20, which you kindly referenced.
The TAPE guidelines provide very good sample questionnaires in annex, which could be adapted locally and used by evaluators (and others) to build their own tool or questionnaire. The questions included in there also help not just measure but also define agro-ecology by expliciting a number of key variables.
So the TAPE guidelines help answer the remark of Laurent on the need to define what success looks like in the transition to agroecology. I think this is an important issue.
There has been very little progress on the transition to more sustainable agriculture, and one of the reasons may be that we don't necessarily agree on what success looks like. While it has produced interesting experiences by civil society and farmer organizations since the 1980s, agroecology has so far failed to convince decision makers in ministries of agriculture -- except in a handful of countries such as Senegal, thanks to the relentless efforts of ENDA Pronat, its secretary Mariam Sow and many others.
Agroecology is even perceived as ideological or militant by certain governments, due to its historical roots as an alternative to the Green Revolution. So defining the approach more objectively would help firmly anchor it in science, and TAPE can contribute there as well.
Evidently, what success looks like will depend on the agro-ecological context. It would make no sense to apply exactly the same criteria all over the globe. It would also contradict a basic principle of agro-ecology which is that it's supposed to be bottom up.
So it seems to me that the right way to define most precisely an agroecological product or system is to do so locally, based on minimum standards agreed with local food producers, traders and consumer organizations. This is for instance what Nicaragua has done with its Law for the Promotion of Agroecological and Organic Production (2011) followed by Mandatory Technical Standards approved and passed in 2013 to characterize, regulate and certify agroecological production units. Many countries have done the same, in an effort to promote agroecology through consumer education and food labelling.
Norbert TCHOUAFFE TCHIADJESenior lecturer / Researcher Pan-African Institute for Development Cameroon
Hello dear all,
I do share the view point of Laurent to some extent .
The comparison between agroecology and existing practices will depend on attributes chosen and the duration of the analysis.
It is difficult in short term basis but possible in long term.
Thanks for sharing the TAPE principles with us, will revert to you soon.
Elias KuntashulaSenior Lecturer University of Zambia
In August 2021, I used a structured household questionnaire co-created from the FAO’s Tool for Agro-ecology Performance Evaluation (TAPE), to assess the level of household’s AE transitions in the South Western Part of Zambia. The Characterization of Agroecology Transition (CAET) section of TAPE (using the 10 elaborated elements) was particularly useful and was contextualized within Zambia's case and used to assess the AE transitions, among a sample of the households.
My experience with the tool is that it comprehensively covers a wide aspect of rural livelihoods and the interactions of agricultural and environmental domains in the provision of this livelihood. Moreover, the tool can easily be understood and implemented within a short duration. All the questions have standardized likert scale responses on the 10 elements and the sub elements, ranging from 0 to 4, making it relatively easier to comprehend. The fact that the tool allows detailed discussions on the farmer's circumstances (by several members of the household) before arriving at what score to give, provided the farmer some participatory space in the determination of their AE transitions, and possible interventions in some areas.
The quantitative manipulation of the scores during analysis ensured that results from the mostly qualitative assessment with farmers, could inform policy in a quantitative rigorous way.
Laurent BarbutDirecteur EPICES
Thank you for this discussion proposal, and the information on TAPE that I did not know. My experience as an evaluator of many agri-environmental and agri-ecological programs and policies in France and in Europe leads me to say that the main issue of these evaluations is the definition of the evaluation frame of reference: how to define precisely what are agri-ecological agricultural practices and those which are not? Agroecology is indeed defined by principles, but the translation of these principles into concrete practices that can be compared to existing practices is difficult because it is the subject of an unstabilized expert debate. For example, what level of fertilization or phytosanitary treatments is compatible with agroecology? There is no single answer to this question.
A second important issue is the knowledge of agricultural practices in a given territory, practices that vary according to the type of farm. This knowledge requires the use of efficient statistical tools - such as the farming practices surveys in France - and the regular repetition of these surveys to understand the evolution of average practices over time. Finally, access to these data must be easy for evaluators, which is far from always being the case.
Kéwé KANECo-coordinatrice groupe thématique Evaluateurs émergents SENEVAL
In our case, the Senegal programme of Horizont3000, which intervenes in Food and Nutritional Security and is implemented through 7 different projects, we focus on agroecological practices. We do not use specific tools to assess sustainability but we define specific indicators for this. We have for example
- the percentage of production from agro-ecological practices
- the areas developed in agro-ecology
- the number of women and young people who have green jobs
- areas undergoing restoration and/or preservation actions
- % of young people and women with initiatives in the field of green entrepreneurship (NR operations)
- % of beneficiaries (men and women??) using climate change adaptation and mitigation technologies
The partners take these indicators into account when developing their projects, so these aspects are taken into account in their intervention strategy.
I will try to find out more about the TAPE tool and possibly test it for the current phase
Jillian LenneConsultant Independent consultant
I suggest that the first step is to define what you mean by agroecology and agroecological approaches to agricultural research and development. Many of the tools (e.g. quantitative and qualitative indicators) available for evaluating AR4D take into account environmental and socioeconomic issues and are suitable for agroecology.
Paul L. MendyMonitoring and Evaluation Specialist IFAD/AfDB/IsDB co-funded Gambia National Agricultural Land and Water Management Development Project - Nema Chosso
Thank you, Seda, for bringing on this important topic for an exchange at this level.
I do have a similar topic “Context monitoring” which I want to share on this platform for ideas and experiences as well.
Simply, context monitoring is a participatory tool to identify, assess and manage risks of interventions with beneficiaries. However, I have not applied the tool yet and would like to receive experiences, tools and lessons from the group.
In the meantime, I will explore the TAPE tool by FAO, as you alluded to, which I think may provide some useful insights to context monitoring.