The farmer as a key participant of M&E: lessons and experiences from Participatory M&E systems

Dear members,

I raised the discussion topic "The farmer as a key participant of M&E: lessons and experiences from Participatory M&E systems" and would like to thank you all for your contributions. I have learnt a lot from your experiences.

Below is a summary of the discussion.

• Participation of farmers in M&E activities helps in ensuring project effectiveness and relevance and quality of project deliverables. When beneficiary farmers generate project data, they take full ownership of the project achievements.
• Participatory M&E requires understanding what farmers already do in the context of their activities in order to avoid overloading them with extra work. It is necessary to develop farmers’ capacities and raise their awareness on the data collection activities and on the purposes of the project.
• Farmers participation may be replaced by good field observation. Google earth and other satellite imagery provide high enough resolution imagery of project areas to easily plot where target crops are produced, measure the hectare involved and sum that up get percent of acceptance.
• Challenges to consider when involving farmers in data collection and feedback include: i) Literacy levels can impede full participation; ii) young women involved often leave their communities of origin due to marriage; iii) sampling of participants may not be easy and it can be difficult to justify that the finding represent the community where the project was implemented; iv) famers’ feedback may be influenced by influential people in the community. This may require sensible managementand facilitation; v) agriculture innovations may take time to be effective, beyond the timing of the project cycle.
• Different levels of participatory processes in agriculture projects may be identified: 1) Deep engagement: involve farmers in all or most of the project cycle; 2) Medium or opportunistic approach: farmers are invited just before or after the initiation of project –  probably the most common; 3) Low engagement: technocrats prepared the M & E and share with farmers to provide their feedback on their already prepared M & E strategy plan.
• In general, the notion of participation acquires different connotations in various contexts and the practice of evaluation does not always reflect this participatory “vocation”. Many times programs and projects aimed at stimulating participation become a symbolic simulation, particularly when they are unaware of the reality of redistribution of power that involves encouraging a participatory process.
• Methods used in participatory M&E and Evaluation include:
  • KoBo for collection of quantitative data and Dictaphones for collection of qualitative data
  • Outcome Harvesting: this method places the beneficiary at the centre to provide relevant quantitative and qualitative data and information on how the project is changing or contributing to changes in their livelihoods.
  • Resource mapping: can be used to understand how a project helped beneficiaries improve their life in comparing their previous experience.
  • Institutional mapping can help to understand stakeholders’ engagement in the community and is useful in the planning stage.
  • Venn diagrams useful for ‘agriculture market’ projects.
  • Seasonal calendars can provide very useful information for food security projects.
  • Community scorecards can be used to assess the performance (efficiency and effectiveness) of the project.
  • Change maps participatory technique: was used in the framework of the evaluation of the economic empowerment project that worked with female farmers in Kyrgyzstan, Central Asia and allowed to turn data collection into a semi-structured discussion among female farmers about what changed in their lives as a result of the project and its worth and merit.

Thank you for all these contributions.

Regards

Emma Gausi

Another issue I have with getting farmer participation is, too often it is virtually essential to leverage their final assessment to conform with what a project has to offer. It must be recognized that before you can solicit effective farmer input, the project will be 2 or more years past conception with over a million US$ expended. That is the typical gestation period from initial conception to issuing an implementing contract and fielding an implementing team, specifically geared to the innovation incorporated in the project proposal and contract. With that much time, effort and money already spent no one wants to hear the farmers are not interested in the innovation and the implementing team as no alternative to leverage the results to the predetermined innovation or go home. The result is very small percent of potential beneficiary’s actual participate, well below the 60+% the underwriting taxpayers assume are participating and those participating only rely on the project of minimum benefits. Ultimately, the projects require continuous external facilitation and collapse immediately after the external support ends leaving little, if any, lasting impact.

 Please review the following webpages and links:

https://smallholderagriculture.agsci.colostate.edu/project-development-process-who-represents-the-smallholders/

https://smallholderagriculture.agsci.colostate.edu/development-hierarchy-four-layers-of-isolation/

https://smallholderagriculture.agsci.colostate.edu/appeasement-reporting-in-development-projects-satisfying-donors-at-the-expense-of-beneficiaries/

Dear Richard,

How did you involve farmers in the processes that you mention, during the development of the log frame and ToC for instance? Please feel free to post a follow up comment. 

Many thanks,

Renata 

From my experience working in the field of monitoring and evaluation, it depends on the project context and to some extent on the delivery of goods and services to targets beneficiaries as well as to those outside of the boundary of the project intervention. Thus, these three tools come into mind and sometimes are confusing: Logical Framework, Result Framework and Theory of Change. Though, the theoretical understanding of these tools on paper looks simple and how they are leveraged in the M&E system is okay however their applications to provide evidence based results is sometime unclear. Further, at what level of results reporting, do I tailor in Logical & Result Framework and Theory of Change to enable better understanding of the process? What I am doing in my project is that I tailor Logical and Result Framework at process evaluation stage (input, activities and outputs) to understand flow of implementations and TOC at impact level (output, outcome and impact) to understand change as a results of the intervention. I would be happy to hear from experts.

I would like to share my experience with applying a “change maps” participatory technique within the framework of the evaluation of the economic empowerment project that worked with female farmers in Kyrgyzstan, Central Asia. The project provided female farmers training on growing vegetables and preserving them and supported them to establish self-help groups and village associations to pool resources, e.g. for procurement of quality seeds and cattle. In some village the project also introduced instruments of the Gender Action Learning System (GALS). Evaluation was conducted at the end of the first phases of the project and was to inform preparation of the second phase.

“Change maps” is a participatory technique where small groups of project participants are offered blank maps (e.g. flipchart sheets) divided into several sections – one per each area where the project was or could be expected to create change – and asked to fill them based on their actual project experiences. In my case the potential change areas were identified in consultation with the project team. For the second phase the team wanted to align the project with the Women Empowerment in Agriculture Index (WEAI) so we agreed to focus the discussion about the changes induced by the project within WEAI domains. As a result our change maps included the following sectors:

•             Do you see any changes in how decisions about agricultural production are made?

•             Do you see any changes in access to and decision-making power over productive resources?

•             Do you see any changes in control over use of income?

•             Do you see any changes in leadership in the community?

•             Do you see any changes in time use?

•             Do you see any other changes?

During the meeting at villages we had up to 45 women involved in the project. Breaking them in small group was easy – each woman was a member of a small self-help group, and each self-help group developed a separate map. Then we gave women three beans each and asked to identify priority changes among those identified in their group. Then each group shared their perspective on key changes that emerged from the project with other groups. And in the end we asked women to assess the “merit” of the project for them on a scale of 10.

The lessons that we learned from application of this approach include:

•             The “Change map” technique allowed to turn data collection into a semi-structured discussion among female farmers supported by the about what changed in their lives as a result of the project and its worth and merit. This helped me to distance evaluation from “control” visits the women were used to and enable a more open conversation about their project experiences.

•             WEAI domains did not exactly match the way female farmers perceived their daily experiences, but they address this challenge by reinterpreting change sectors of the map. But in the future I would have used change sectors based on what the project was doing rather some external theoretical constructs.

•             Filled change maps and discussions around them provided evaluation team with reach material for analysis. For example, based on the content of the maps I was able to identify more nuanced types of changes induced by the project and how common these changes were. One of interesting findings was that engaging women in productive agricultural practices led to women having no free time. This was seen as a positive change by female farmers and their families but came as a negative surprise for the project team.

Sincerely,

Natalia Kosheleva

Evaluation consultant

PME is a widely used term by all USAID projects since early 2000, it is an acronym for top-down Planning, Management (Monitoring) and Evaluation. It would be better to avoid using PME as here we mean "Participatory monitoring and evaluation", a bottom-up approach. 

Sincerely

Elamin

Dear Emma,

Thanks for posing this important question. Most times, M & E is considered as an accountability function of the donors and implementing agencies also follow the same approach. In this case, participation of farmers is not considered as prerequisite but sometime viewed that farmers are not ‘knowledge’ on these technical matters and only technical persons can provide this service on behalf of beneficiaries (as the technical persons also understand the local/farmers context and needs).

Participatory M & E has emerged to rectify this challenge where farmers would be involved in all stages of M & E – from planning to final evaluation. There is however mix experience in real world situations. I have noticed three types of P M&E process while working for various development organizations. In first group, project P M & E process duly respect its basic premise of participatory principle and involved farmers in all or most of the project cycle (deep engagement). In the second project use more opportunistic approach. Farmers are invited just before or after the initiation of project (most times during the project inception phase) and share the M & E strategy/plan and claimed farmers’ involvement in M & E process. In my experience this is the most common approach applied in agriculture project in managing M & E (medium level engagement). The third category, where technocrats prepared the M & E and share with farmers to provide their feedback on their already prepared M & E strategy / plan (low engagement).  

There are many participatory P M & E tools/methods that are dependent on the context and the technical matters. For example, for planning the project, ‘social and resource mapping’ would be very useful whereas for ‘agriculture market’ project ‘Venn diagram’ might provide some good understanding. Similarly, for food security project analysis, ‘seasonal calendar’ would provide very useful information. I have also used community score card to assess the performance (efficiency and effectiveness) of the project. There are many tools available, but it is vital to understand the basic principle of the participatory process and one should have a strong rapport with farmers and patients to use the tools.

Best regards,

Ram Chandra Khanal

Dear Emma, 

let me share with you and the rest of the colleagues my experience in EvalParticipativa, the Community of Practice and Learning in Participatory Evaluation for Latin America and the Caribbean. This community has born from the interest and growth of the evaluation in the regional and global context. In recent decades, theoretical and methodological production has increased, as well as the emergence of national evaluation policies in countries on all continents, a growing institutionality of evaluation and the consolidation of various initiatives aimed at professionalizing this practice. In addition, the Sustainable Development Goals report a new agenda of priorities for global evaluation, which includes the role of civil society in these efforts and highlights the participatory dimension as a central aspect of the evaluation practice. 

However, the notion of participation acquires different connotations in various contexts and the practice of evaluation does not always reflect this participatory “vocation”. Many times programs and projects aimed at stimulating participation become a symbolic simulation, particularly when they are unaware of the reality of redistribution of power that involves encouraging a participatory process. In these cases, participatory evaluation tends to be limited to a mere consultative instance, without offering local actors the possibility of influencing the decisions of the evaluation agenda and thus enriching its products and impact.

It is true that the theoretical and instrumental deficit that characterizes these practices is not clear or evident, as well as what are the conditions and mechanisms that facilitate and impede effective citizen participation in the evaluation processes. Therefore, without revealing these issues, it will not be possible to overcome the weaknesses of many participatory evaluations.

Thus, we consider it relevant and timely to develop a Community of Practice and Learning on Participatory Evaluation for Latin America and the Caribbean that allows these issues to be deepened creatively through knowledge and analysis of concrete evaluation experiences, which will enhance their scope and socialize their methods and tools

As a community of practice we emerged from the conviction of the potential of peer work as a mechanism to deepen knowledge and experience through different instances of interaction (training, meetings and fairs of tools, webinars, publications, etc.). By "peers" we understand all those who in one way or another are involved in an evaluation process: evaluators, decision makers requesting an evaluation, civil society organizations that are users of programs and projects, among others. The creation of this community of practice and learning around participatory evaluation for Latin America and the Caribbean has as its main objective the strengthening and inclusive involvement of civil society in evaluation processes. For this we want:

1) Identify and analyze experiences of participatory evaluation in Latin America and the keys that mark the facilitation of these processes.
2) Create a virtual repository of materials for learning about participatory evaluation: manuals, guides and validated tools.
3) Systematize the experiences studied by transferring good practices to a manual for the facilitation of participatory evaluations, which has four central chapters: conceptualization, methodology (processes), tools and guidelines for facilitation.
4) Design a course to facilitate participatory evaluations and implement a pilot test to train representatives of civil society organizations in the region.

In this 5 minutes video you can see the experience we developed last year in the First Latin-american and Caribbean Workshop on Participatory Evaluation. Unfortunately, most of our materials are in Spanish, although Google Translator can give you a hand ;-). We are planning to have our website in English soon. I´m also attaching a brief publication by one of the organizations linked with EvalParticipativa.

I hope you find this useful. 

Best wishes,

Pablo 

Based on the  experience I had on implementing rural livelihood projects in international NGO, applying PME has its own advantages and limitation.

During impact evaluation, PME helped us to evaluate the project impact through how the developed resources/schemes could finally impacted on their life. One of our development scheme was developing a water source, hand dug well.  just to give you one example, The resource mapping helped us to well understand how the scheme helped them improved their life on comparing their previous experience by showing where they fetched water in the map and how far they traveled and tried also showing in picture how It was affecting women on different situation  like rape, harassment and abduction and the like. It helped us also to understand how the scheme improved social interaction and information sharing when people meet at the scheme  just by drawing pictures.

We used also the PME for planning purpose for project resource allocation and partnership. For instance We used resource mapping to get in-depth community perception and views on resource utilization and sharing  with in  and neighboring communities, when the resources are plenty and scarce, which resources are source of conflict and when. Institutional mapping also helped us to understand stakeholders’ engagement in the community and how it influence their livelihood. This information helped us to understand who works what and also to choose to whom we need to collaborate.

However, there was some limitation when conducting the PME. One of the basic challenge was to show how much  our finding are well represented due to accurate sampling was difficult. We had struggled to justify that the finding represented to all the community where the project implemented. On the other case, the discussion on some cases influenced by influential people in the community unless we made it controlled and managed discussion. On the other case, women participation were very low due to cultural influence.

Best regards

Mesfin

Effectively involving farmers in the participatory process has always been a challenge both for diagnostic and evaluation. However, much of this might be due to the tendency of relying on the interview process to solicit their input.  Thus, it might be noteworthy that much of agronomy is highly visible and thus it can be easily and perhaps more accurately passively collected with some good field observation. Once the farmer plants the seeds, he has a several month commitment the is easily seen for the entire cropping season. This be good for identifying the crops being produced, and the varieties being grown. With some good observations or questions, it would be possible to get planting dates and determine the biggest oversight in agronomy and smallholder development. This is the timing of crop establishment and other time sensitive activities. The overall time spread being a critical indicator of the severe operational limits smallholders face in terms of limited labor, energy to fuel that labor sufficiently for a full day of manual agronomic field work, or access to contract mechanization. Once this oversight is recognized it hopefully will be addressed instead of being ignored for over 40 years.

With the quality of Google earth and other satellite imagery it is now easily possible to get high enough resolution imagery of project areas to easily plot where target crops are produced, measure the hectarage involved and sum that up get percent of acceptance. Likewise, get an estimate of planting date and you can quickly plot a cropping calendar to see the overall cropping pattern of the project area, including the time spread in activities. This can be both a diagnostic tool and monitoring tool. While all this may have only minimal direct connection with the farmers, it should give you a quick appreciation as how effective your project is. Please see the example attached of a crop calendar that was developed to document actual irrigation use in Egypt as a guide for bring irrigation issues in line with actual use.

Of course, you will need to have targets as to what will qualify as a successful vs. unsuccessful project and this will need to be in line with what your underwriting taxpayers funding the project expect and will accept as success.

If the operational limits mentioned above were identified 30 or 40 years ago our overall emphasis would have shifted from concentrating on specific crops and crop management to facilitating access to contract mechanization, expediting crop establishment sufficiently to meet food security with enough surplus production to accommodate the value chain we promote as a means of stimulating production.

Please review the following webpages:

https://webdoc.agsci.colostate.edu/smallholderagriculture/OperationalFeasibility.pdf

https://smallholderagriculture.agsci.colostate.edu/calorie-energy-balance-risk-averse-or-hunger-exhasution/

https://smallholderagriculture.agsci.colostate.edu/request-for-information-basic-business-parameters/

https://smallholderagriculture.agsci.colostate.edu/mel-impressive-numbers-but-of-what-purpose-deceiving-the-tax-paying-public/

Greetings!

I hope that I may be forgiven for presenting a few justifiable reasons to demonstrate that the much-maligned top-down approach does not mean dictating to a target group be they farmers or any other category of actual workers who really do the job.

For instance, take the case of a single farmer; if he is experienced, not necessarily loaded with impressive diplomas in agriculture, would not he first consider the type of soil in his property, access to water, climate, kind of crop or animals fro which he is certain to have a demand, etc., before he begins his work? If anyone should deny this, one is just assuming that farmer is just ignorant, which is very often unjustifiable. 

If one should agree that our farmer does  so, then, he is using top-down approach to see what is most suitable for him to grow and/or raise on his land, because that will enable him to achieve his goal of succeeding in procuring enough food for himself and selling the rest to meet his other needs.

The problem is not in the method, but in its fragmented application. What happens is it is incompetently applied at highest level i.e., policy formulation and even less competently at the strategic implementation level. At regional and local levels, what comes from the top is just passed over as it is. This is a common chain of incompetence very wide-spread throughout the globe. The term top-down' is bandied about often by those who do not understand what it is, or what it involves.

It is a staggered approach where at each level both the 'big picture goal' which is general becomes more and more area-specific on implementation. Finally, at our farmer's level where it matters most, it becomes a process of reconciliation between his goal through its integration into the national whole. It is as simple as that when shorn of jargon and the clichés of vested interests.

So, farmer's wishes and feed back is an integral part of a competently applied top-down solution.

Cheers!

Lal Manavado.

Dear Emma,

Good topic for discussion. Please find my contribution below:

Nema Chosso Project in The Gambia – Experience on Farmer’s role in PM&E

The International Fund for Agricultural Development (IFAD) is the financier of the National Agricultural Land and Water Management Development Project which is locally called Nema Chosso in The Gambia. The goal of the project is to reduce rural poverty through sustainable land and water development and management practices. The project targets rural smallholder women and youth and invests in the promotion of two commodity value chains: Rice and Vegetables. Women and youth are the principal producers of these crops, which are also the staple foods in The Gambia. The project provides investments to finance approved interventions as requested in a Request for Assistance Form. Communities, Groups and Individual farmers submit their requests through the local government authorities and regional agriculture directorates across the country. These authorities are responsible for reviewing and appraising the requests and upon completion of the appraisal/feasibility submit recommendations for intervention to the Project Management Team for assistance.

The Nema Chosso Project has succeeded in mobilizing beneficiaries (farmers) to participate in monitoring and evaluation of project activities, outcomes and impact and this has proven to be effective. Here is what we did:

1. Once a community or group has been selected for intervention or support, the project sets off to conduct a sensitization meeting on the specific deliverables which the intervention entails.
2. During this sensitization meeting, beneficiaries (farmers) are made aware of their role in ensuring the anticipated project benefits are realized for their own good;
3. At this meeting, the project team emphasizes the fact that once the intervention is completed and formally handed over to the community, it is their role to ensure continuous operation and maintenance.
4. The role of beneficiaries (farmers) in sustaining the project support outcomes are thus established.
5. In order to facilitate the effective participation of farmers in the project, the Department of Community Development is engaged by the project to conduct training on group management, governance and sustainability.
6. Such training (5 days) culminates in a selection (or election) of what we call Village Farmers Associations (VFA) for each of the beneficiary communities.

The key role of the VFA is to lead the coordination of the community for effective participation in the project implementation, key among which is the monitoring of the performance of the contractors and consultants to ensure the activities and outputs actually lead to the project objectives. Once the VFA is established, all project staff and stakeholders are instructed to use the VFAs are as entry point for further development initiatives under the Nema Chosso Project.

The project recruits a consultant for the supervision of all works. The Consultant, like all implementing partners of the project, is sensitized as to the existence of the VFAs and is required to ensure they participate in the monitoring of the delivery of works. It is stipulated in the contract and further emphasized during the handover of intervention sites to contractors that the VFAs, when dissatisfied with the contractor’s performance or quality of work, shall have the powers to stop the works and report such issues to the project through the project’s regional staff. The Project M&E Team keeps a database of the VFAs and ensures they join and participate in fieldwork monitoring.

A VFA comprises 12 members, 6 of them women and 6 men. Most of the farmers in rural Gambia are women and youth. The training by the Department of Community Development guides the communities in the formation of VFAs. The guide the selection process to include the youth in the VFAs. The VFAs are supported by the project with further training specific to the monitoring mechanisms in place, and channels of communication with the Project’s regional structures. Data collection templates on yields and incomes are provided with training on their application. The project’s regional structures comprise a Regional Coordinator, two Focal Points for rice and horticulture value chains and a conservation field assistant to provide technical support in land development and conservation civil works.

Further on building the capacities of beneficiaries to effectively do PM&E, they’re provided functional literacy training for the first 2 years of the project to enable the illiterate ones to do basic record keeping and data collection. Each beneficiary community identifies 30 members to participate in the functional literacy classes. What is interesting here is that the functional literacy lessons are delivered in their languages of mother tongue and this enhances quick absorption and uptake of literacy skills. These skills allow farmers to record minutes of meetings and those are well-read out in their typical languages by their members during subsequent meetings. This ensures ownership of decisions reached.

Nema Chosso Project involves beneficiaries (farmers) in outcome and impact assessments. We use the Outcome Harvesting method to assess the project's effectiveness and this the approach places the beneficiary at the centre of the process, providing relevant quantitative and qualitative data and information on how the project is changing or contributing to changes in their livelihoods. Project beneficiaries (farmers) are engaged in important studies/surveys such as the IFAD results and impact management system (RIMS) baseline and end-line surveys. They have also participated in validating the survey findings and results.

Key challenges

• Literacy levels are low and this impedes full participation in M&E, hence the initiative to provide functional literacy training
•  
• Young women often leave their communities of origin due to marriage; this often requires replacing them with new ones who would often lack the same enthusiasm and commitment to the project
•  
• When project contracts delay, which is often the case, members of VFAs are first targeted for criticism by the rest of the communities thus creating distrust and suspicion

Key lessons:

• Beneficiary participation in M&E significantly ensures project effectiveness and relevance
• Beneficiary participation in M&E ensures quality project deliverables
• Capacity building is crucial for effective participation for beneficiary participation in M&E
• When project achievement data are generated by beneficiaries they validate such data and take full ownership of successes.

Thank you

Paul

As part of the implementation of the Global Agriculture and Food Security Program Missing middle Initiative (GAFSP MMI) Senegal project, the FAO team based in the field with had among its missions the establishment of a participatory and inclusive Monitoring and Evaluation system. Today we decided to capitalize on this experience in order to share it for beneficial purposes. In summary, the process started with a diagnosis of the essentials in terms of monitoring and evaluation or even data management near the beneficiary producer organizations at the office level with the technical team and at the field level with the farmers. This diagnostic mission made it possible to scope the monitoring and evaluation system with a clear object and its fields of application. This mission has also made it possible to have a look and discussion on the tools currently used by farmers in the context of their activities in order to integrate them in the system after discussion and reinforcement. The discussion makes it possible to see the shortcomings with farmers and other stakeholders in order to take the initiative together to strengthen the system in this direction. After this phase, the unit proceeded with the design of the device and the tools shared with the producer organizations for validation. It is in the implementation that we made the producer organizations responsible for their participation and reinforcement so that after the project they would be able to unfold the device. Each producer organization has its team formed by the coordination unit in a progressive manner for the implementation of the device. They are in the collection of quantitative data with KoBo collect and qualitative data with Dictaphones and focal points in the management of the database with KoBo collect. Today the implementation of the system is participatory in all the functions that make it up in its entirety. We will not fail to share the experience after capitalization with a lot of information on the methods and approaches adopted.