Making data collection meaningful and useful to farmers: what is your experience?

Dear members,

I would like to thank those of you who took the time to share insights on the discussion topics I proposed and those who expressed an interest in taking part to the review of the farm sustainability methodology that we are developing. We will incorporate feedback received in the documents, and share them with the community once ready.

It was extremely interesting to hear first-hand experience with these challenges from such a wide range of different geographical contexts, and  practical ways to overcome them.

Below is a summary of the lessons learnt from the discussion:

Participatory design of M&E initiatives:

I was happy to see several members underlining the importance of designing M&E activities that target farmers in a participatory way.

A key point raised is the need to devote time at the start of activities to explain the reasons why data is being collected, to understand priorities of the local community, and to identify issues—and indicators to measure these—that are meaningful to  farmers, so that these can be included in the assessment to the extent possible. Another good practice at the start of activities is the sharing of assessment questions with farmers (either directly or through their representatives or local community leaders), to gather feedback and refine questions. In order to increase engagement in the process, a kick-off event should be organised before the start of activities, to present the proposed M&E indicators and give farmers the opportunity to select  those they find most relevant, or to propose additional or alternative ones.

In this sense, while M&E exercises will have to be designed to ensure project interventions can be properly assessed, it is equally important to recognize and incorporate farmers’ needs and the issue they face, and strike a balance between their needs and those of the project/evaluation. As one member noted, in some cases this might mean setting aside some of the questions or indicators originally planned.

At data collection stage, where possible, facilitators and enumerators should be recruited from within local communities, and sufficient time should be dedicated to train them on the methods and tools used (e.g. digital data collection applications). This facilitates local ownership of the process, helps transfer knowledge and enhance local capacity, and  in turn can increase sustainability of project interventions over time. Where possible, data collection should also include participant-led methods, for example community mapping of challenges experienced locally that can add further depth to quantitative and qualitative data collected through surveys and interviews. Involving farmers in the interpretation of findings can ensure their perspective is heard, therefore improving overall reliability and depth/quality of information collected.

Interviews: logistical arrangements & practical considerations:

In this regard, a first point raised the need to ensure that farmers are interviewed at their preferred setting and time taking into account farmers’ work schedule and on-farm activities (for instance, assessment exercises could be schedule between farming seasons). If women are the target. allowing them to bring their children and avoiding lunch hours can make it easier for them to participate.  Several community members also highlighted that farmers should be compensated somehow for their time. Compensation can be either financial, especially when meetings take up a substantial amount of time and result in e.g. the loss of a full work day, or in-kind and symbolic through small gifts or drinks and refreshments. However, caution should be exercised when considering monetary compensation, as financial incentives can potentially influence assessment participation and interview results.

Members also provided valuable advice on best practices to observe during data collection. These include distributing written materials with information about the programme that participants can keep and share with other households members;  allowing space for Q&A sessions; including extension workers in the assessment, as they are generally the ones maintain relationships with farmers throughout the year; and using appropriate language—for example refraining from defining activities specifically as “monitoring” and “evaluation”, and limiting the use of technical jargon—to put interviewees at ease and ensure understanding of questions being asked.

Dissemination and discussion of results

Members highlighted how participatory approaches should extend to results discussion and dissemination, and also underlined the value of farmer-to-farmer exchange and peer learning. Comparing and discussing results from project interventions for those farmers who joined activities and those who have not can provide a valuable learning ground and encourage increased participation in the project. I see an immediate entry point for this in the case of our project, which focuses on supporting the adoption of sustainable farm management practices: farmers who may be hesitant to try out new practices may change their opinion after hearing from peers about tangible benefits from these.

In terms of practical organization of events to discuss results and facilitate peer learning, members suggested organizing farmer knowledge exchange events, farm visits, and informal gatherings to encourage information sharing among peers that use a common language. The emphasis in these events should be on facilitating a participatory discussion of findings; on discussing lessons learnt from the farmers’ perspectives; and on identifying ways that assessment results can be beneficial to them and their communities. In addition, a participatory approach to the sharing of findings can also serve to ground-truth results from evaluation surveys.

Sending a summary of results in advance to farmers can increase their engagement into discussion of findings. When presenting results, farmers should be encouraged to explain them in their own words, and to share main lessons learnt from the exercise in terms of farm sustainability management. Also, grouping farmers into interest groups (e.g. youth farmers, or farmers involved in a specific aspect of the supply chain) can increase interest in the discussion of specific parts of the assessment results that are more useful to them.

Many underlined the added value of visual aids, including videos, presentation, infographics and illustrations and using  local language to facilitate discussion around topics at hand, and of local information networks and systems (e.g. instant messaging platforms) to disseminate lessons learnt beyond the immediate group of participants.  

Thank you very much again to the EvalForward community for this interesting discussion!

David

Dear all

Let me share my experiences from my own M&E assessments.

When designing questionniares it is important to already think through the culture and farmer requirements. For example if its in rainy season, the farmers will usually be very busy with farm work. It is best to have an initial visit to plan with them in advance. This is where you go through a farmer representative to do the planning.

If survey targets women allow them to bring their toddler also make sure its not schedule around the lunch hours

When conduct focus group discussion its better to meet them at their usual meeting hours than requesting for an extra ordinary meeting.

The questionniare should be as concise as possible.

Administer the questionniare in their local language. This saves on translation time.

Explain to the farmer the purpose and expected result as well as expected time of the interview and seek consent for the interview.

If they are focus group discussion ensure that they use alot of participatory methods other than just questions

If its field based activities, ensure that the farmer takes part in other activities such field measuring. It motivates them.

 Train the enumerators. Let them understand the meaning if each to avoid redundancy

On incentives, we always try to provide the respondents a drink

Hello all,

I agree with most of what has been contributed below, especially Anna Maria's comments on involving farmers or other local stakeholders in developing the evaluation design and relevant indicators. Using a participatory approach will help uncover assumptions in program design or other local incentives that the research team did not foresee, and which therefore would not be covered in the questionnaire. Involving local research team members is helpful, but the closer you can get to bringing the individuals who would be program participants themselves into the design, the more relevant and targeted the survey will be.

Just a few thoughts in addition to what has already been said:

1.Striking a balance between depth and length of assessment:

Think critically about how the data will actually be used to design/adapt programming or inform decisions, and then eliminate the assessment questions that collect unnecessary data points. One way to do this is to conduct a simulation of use: the evaluation team brainstorms potential evaluation findings, and then has a facilitated discussion with the intended evaluation users to discuss how those findings would influence program design or funding decisions. This can help narrow down which data points you really need. For example, you might have questions about types of employment for various household members, but recognize that these data aren't likely to impact your program design - so you can eliminate them from the questionnaire.

I'd also recommend bringing in more participant-led data collection methods (e.g. Most Significant Change or other storytelling formats, participant-led stakeholder mapping exercises in which they visually map out how the local system presents challenges or changes to information flow, resource access, etc), as this will add depth to your data.

I agree with the comments made by others about involving local stakeholders in data collection (to the extent possible in light of methodological limitations in an experimental setting). This example of community-led enumeration in Ghana is good inspiration!

2. Making findings from M&E assessments useful to farmers: 

The data placemat approach, in which the facilitator presents data visually, then guides stakeholders to explore and explain the data in their own words, is a good way to get people interacting with the data. For an audience of smallholder farmers, you'd need to rely more heavily on visuals that farmers relate to (rather than bar charts!) but I think there's a way it could be done! 

After presentation of findings, you could have farmers generate short skits that represent their vision of a future in which the lessons about farm sustainability are implemented. This helps generate enthusiasm for action, as motivated by evaluation findings.

Best of luck!

Amanda 

Dear David and Colleagues,

Thank you very much for the interesting topic for the discussion. Below, I'd like to share some insights from my practice as evaluation consultant and researcher. I worked on a number of assignments from the local to global level, which involved surveys with farmers and other rural people in diverse geographic contexts. My most recent project has been focused on the capacities of evaluation stakeholders in multi-actor projects targeting agricultural innovation (https://liaison2020.eu). I've a strong background in sociology and psychology, which also affect my approaches to surveying. 

1. Striking a balance between depth and length of assessment: 

• How can the burden on smallholder farmers be reduced during M&E assessments

This usually depends on the context. For instance, I interviewed farmers who were very interested in chatting with me, both about the survey questions and non-related topics. It's important to recognize their needs and issues they face, which may be often different from what we expect as evaluators. Some people are more, some less busy, introvert or extrovert and it can also affect their eagerness to engage into the task. I normally strive towards a balance between their needs and mine. At times, one may need to compromise skipping some questions in the survey. This could be reflected at an earlier stage - the evaluation design, when decisions are to be made on the direct and proxy indicators. 

• What are the best ways to incentivize farmers to take part in the survey (e.g. non-monetary incentives, participation in survey tailoring, in presentation of results)?

It can be helpful to ask what are their evaluation needs: a problem they want to solve, in which evaluation and data could help. They may be quite different from what the evaluators intend, so one should try to negotiate and seek an optimization in the evaluation design. It’s helpful to engage farmers into defining the scope of evaluation, relevant questions and indicators. For instance, I once run a workshop where participants were presented with a list of possible indicators and could rate those, which were most relevant in their opinion. The result was quite different from what the evaluators anticipated. Non-monetary incentives are also helpful. I remember bringing a box of fine chocolate from my home city to farmers, with whom I stayed during the survey work. They were helping me to identify other survey participants (snowballing) and at the end gave me also eggs from their farm to bring home. Concerning monetary incentives, I always fear the Hawthorne’s effect, i.e. an increased performance of respondents under the pressure of being studied and rewarded. 

2. Making findings from M&E assessments useful to farmers:  

• Based on your experience, what could be the most effective ways to communicate results from the sustainability assessment to farmers (e.g. field visits and peer learning, technical information workshop)? What kind of communication materials (e.g. briefs, leaflets, others) are most appropriate to support knowledge sharing events?

Definitely, P2P learning is very useful. This way people can exchange with each other using the same language. As evaluators we often tend to communicate in a different way than farmers, hence a skilled facilitation is usually a better option than and top-down way of presenting the results. It’s good to have it as a facilitated discussion, field trip and some informal get together. In addition, various dissemination channels can be helpful, such as radio, videos or leaflets. Using visual communication is quite effective, in my experience. I remember evaluating a project where farmers had issues with recognizing grapevine diseases, which already existed in their area. They did not know actual names of those, but pictures helped to recognize them. 

• Do you have experience in comparing results among farmers in a participatory way? What method have you used to do this? Was it effective?

I remember a visioning exercise where evaluation results were presented and further elaborated. It was a mid-term project evaluation, where people who were earlier interviewed (farmers and other rural community members) participated in the event, also some contributed with their stories. Based on this, a visioning exercise was run by the external facilitators, which was intended to help in improving the project and planning other activities for the community’ future. Various methods were used, including the facilitator’s toolbox with sticky notes, flipchart and others. 

• How can the results be used for non-formal education of farmers (e.g. to raise awareness and/or build capacity on ways to increase farm sustainability)?

In principle, the evaluation results need to be translated into the farmers’ language. With these, they can be used in many ways through capacity building activities. Forms of P2P and experiential learning are in my experience most effective to maximize the uptake of the evaluation results at the farm level. Sometimes, the broader enabling environment of evaluation need to be also considered, for instance farmers may lack some incentives to change their practice, despite an increased awareness on the issue. It's important to choose the right means of communication, which can be also different in various countries, regions and depend on the literacy of the farmers and their community leaders. 

With best wishes from Budapest,

Anna Maria Augustyn

https://www.linkedin.com/in/aniaaugustyn/

LIAISON2020 | Optimising interactive innovation

Dear David,

Thanks for initiating this useful discussion. I want to share experiences from our organisation on how we have navigated some of the points you raised.

To minimise the time burden on respondents, we try to be strict during questionnaire development. This generates some payback that all the questions we ask relate to some indicator that we will analyse. It also helps us remove some useful and desirable questions to only relevant questions without which our assessment would not be complete.

We have found engaging farmers to tell their stories as we present our findings very useful. It also helps ground-truth our findings. in addition, they can enumerate lessons learnt during the assessment.

For us at Tegemeo Institute, we try and have forums with farmers where we discuss our findings and how they can use the findings for their benefit. in addition, we have found the use of infographics handy with farmers as utilising local information networks to disseminate information. Furthermore, when we have compared farmers, we have found their approaches to make comparisons and deductions quite informative. I definitely recommend participatory approaches.

To reduce the burden on small-scale farmers during monitoring and evaluation assessments, the collection tools should be adapted to the different social realities of these farmers, and farmers should be involved in the design of these tools by involving them directly in the preparatory work.

As for the best ways to encourage farmers to participate in the survey, it will be necessary to propose compensatory measures that would correspond to the time they would have lost in the monitoring-evaluation exercise, for example by offering some refreshment when they participate in the activity. Also, it will be necessary to adapt the collection tools to what they would have proposed (make proposals for the choice of tools if possible).

[Contribution originally posted in French]

The main strategy to make data collection meaningful and useful to farmers is to involve them in the process of investigation, from the beginning to the end. The best method for that is the IAR4D Approach (Integrated Agricultural Research for Development), developed by the Forum for Agricultural Research in Africa (FARA) in 1983. Then, the raison and the process of the data collection are explained to the farmers in a participatory way, and in their local language. Then, evaluation is considered as a system that is made of many sub-systems that must work together to foster development. All actors that are involved in the process, including farmers, interact and jointly foster their capacities. Thus, the IAR4D Approach simultaneously addresses research and development as a fused continuum for generation of innovation. Generally, in the process, the farmers’ analysis of the evaluation steps and their analysis of the findings are different from that of the researchers/evaluators, and then improve the quality of the evaluation. With this approach, the farmers constitute at the same time the channel for the process explanation and the results dissemination.

For more information, see https://faraafrica.org/iar4d/

Dear members,

I would like to share my experience in data and information collection among farmers with the Indonesian Farmers Association. 

Usually we contact farmers though the chief of village or farmers leader and invite them in a certain day/time and place. We have a preliminary discussion with the chiefs and send the questions in advance so that the leaders can respond generally to them to get an idea and have prior information. At the meeting we go deeper with farmers also one-by-one and get fresh information and do crosschecks when needed.

This was in the pre-pandemic situation, as currently gatherings are stopped and we only call on individuals.

It is important to note that in our rural areas there is a culture and tradition by which if we invite farmers we need to set up a type of ceremony: we have prayers, delivery speeches by chief of village or farmers leaders, open discussion and we prepare food. This is also the incentive. Meetings can be one day long and in some cases, since farmers dedicate a long time, we pay back the lost workday.

In addition to the chief, we also invite the extension workers: these are the ones who maintain the relations with the farmers and the leaders and already have a lot of data and information.

In the case of international donor-funded initiatives and the related Monitoring and Evaluation missions, in order to avoid farmers feeling this as a verification or audit, we prefer not to use the terms M&E and call them SIS - Supervision Implementation Support missions.

On communication: we communicate back results from the programme evaluations. Also in this case we have gatherings and we send in advance the highlights (summary) of the evaluation reports so that the farmers can react and disagree / respond during the meetings.

I hope this is useful information when approaching farmers for data collection and communication.

See my video here on the AFOSP-MTCP2 Indonesia Report:  https://youtu.be/kICPu8tb7jc?list=PLtXDxoTN3R8ajQmREnrdJUEvXlwCCwaB6

Agusdin Pulungan

Hello everyone,

I have tried to answer each question and my answers are below. They are based on some of my experience with small farmers.

1. Striking a balance between depth and length of assessment: 

Small farmers are very busy because they have to find alternative / complementary sources of income. In addition, social time is important (marriages, tea time, football for the young, carpet making for women...).

Thus, assessment time should fit within their schedule. I suggest short questionnaires that are meaningful to them, which brings in that the programme should take into account their actual needs and not 100% according to organizational needs.

• How can the burden on smallholder farmers be reduced during M&E assessments?
• What are the best ways to incentivize farmers to take part in the survey (e.g. non-monetary incentives, participation in survey tailoring, in presentation of results)?

1. Make it a social time and talk about what is meaningful  to them (ex cereals in the mountainous areas). The usually preferred time is the afternoon. ex. plan assessment time during tea time and work with focus group. If the questionnaire is preferred then it will take more  time for the evaluator because she / he will have to adjust to each farmer;
2. Allow women to bring in toddlers or small infants ( up to 5 years);
3. Give away written information on the programme. They will keep it and show it to their schooled children;
4. Plan on lunch or afternoon tea with snacks.

2. Making findings from M&E assessments useful to farmers: 

- Like for the assessment, plan on information workshops in "a between seasons" time to avoid getting on the way for "actual" work;

- Provide leaflets, audios, videos, pictures;

- Allow for Q&A sessions.

• Do you have experience in comparing results among farmers in a participatory way? What method have you used to do this? Was it effective?
• How can the results be used for non-formal education of farmers (e.g. to raise awareness and/or build capacity on ways to increase farm sustainability)?

1. Comparing results among farmers is effective in showing results and making farmers adopt new techniques. I used a treatment / non treatment method . The non treatment was actually from farmers not adhering to the programme. Once results were obvious, they asked to be included.

2. Results could be used in  non-formal education of farmers through exchange visits among peers, audios and videos distributed through instant messaging, result presentations on field visits of extension workers. 

Malika Bounfour

Dear David,

Thanks for bringing these important issues to this community of Practice. Based on my experience working in the developing countries, I have the following input for your reference.

With best regards, 

1. Striking a balance between depth and length of assessment: monitoring and assessment exercises based on interviews and farm surveys can put significant burden on respondents, for example diverting time that would be otherwise allocated to other activities. Respondent fatigue due to lengthy interviews/surveys can also result in lower quality of data collected, and therefore in lower reliability of results. On the other hand, a shorter assessment may result in a level of depth that is insufficient to design effective interventions.

How can the burden on smallholder farmers be reduced during M&E assessments?

I prefer to do/ am doing: 

• Making objective oriented short questionnaires
• Mostly close ended but also provision of sharing their views and perspectives
• Interview in their own setting and preferred time
• Making them feel they are also benefitted from this exercise
• Create strong rapport (interpersonal skills) – (not mechanical but also speak on their personal issues)
• Provide some present (this can be to their children)

What are the best ways to incentivize farmers to take part in the survey (e.g. non-monetary incentives, participation in survey tailoring, in presentation of results)?

I prefer to do / am doing: 

• When I was programme/ project manager I used to provide farmers some financial compensation (I strongly feel we need to pay the information provider as we information collectors are making a good sum of money for similar kind of functions)
• Compensate their time with good snacks / refreshment
• Provide them a present as a ‘token of love’
• Acknowledging their support

2. Making findings from M&E assessments useful to farmers: considering the burden on farmers resulting from M&E exercises, it is key to ensure results are meaningful and accessible to them. This is in fact an explicit objective of the M&E tool we are developing. The assessment seeks to provide an indication of sustainability strengths and weaknesses that can be used by e.g. extension agents to help farmers identify targeted practices that can increase overall sustainability of production.

Based on your experience, what could be the most effective ways to communicate results from the sustainability assessment to farmers (e.g. field visits and peer learning, technical information workshop)? What kind of communication materials (e.g. briefs, leaflets, others) are most appropriate to support knowledge sharing events?

• Clarify the objectives – how sustainability assessment are important to farmers and their groups/
• Organize sharing meeting and get their feedback on the result
• Take full use of local resources persons/ local groups or trusted partners while sharing the results
• Use illustration/visual aids/local language  
• Less of use of technical words and complex terms

Do you have experience in comparing results among farmers in a participatory way? What method have you used to do this? Was it effective?

When possible:

• Making sharing group according to the interest groups (such as women farmers, youth farmers, farmers groups based on their production or participation in different value chain)
• Use illustration / examples / visual aids/ simple demonstration (such as big maize cub vs small cub) relevant to local context  (for example if you say 50%, in many cases – they do not understand, if you give example such as 100 unit and 150 unit (50% additional)- may be will be in position to understand

How can the results be used for non-formal education of farmers (e.g. to raise awareness and/or build capacity on ways to increase farm sustainability)?

• Develop participatory based farmer centred training module considering the need of the farmers
• Provide sharing opportunity by farmers (farmer to farmer approach)
• Use of Use illustration / examples / visual aids/ simple demonstration

-----------------------------------------

Ram Chandra Khanal (PhD)

Evaluator and programme manager: Climate change/NRM/Agriculture

Dear Colleagues,

I would like to share with you our experience in the implementation of the GAFSP MMI project in Senegal.

We have set up a participatory M&E system in which the communities are the central elements. For each PO (producer organisation), we selected a team of supervisors and facilitators from the communities. We trained these people in digital data collection via kobo and also in the facilitation of a qualitative survey in order to empower them in data collection. The training was difficult, given the level of some facilitators, but it must be acknowledged that they are very much empowered and accepted by the communities for the provision of reliable data.
We are continuing this process to make monitoring booklets available to the grassroots producer organisers, which are filled in every year so that the facilitators, via the tablet, will go on missions to collect data in the monitoring booklets of the value chain.

It is necessary to have an inclusive approach and to define an M&E guideline that makes producers responsible for the implementation of the monitoring-evaluation system.

[this contribution was originally posted in French]