Renata [user:field_middlename] Mirulla

Renata Mirulla

Facilitator of Evalforward
Italy

More about me

I facilitate the EvalForward Community of Practice by supporting knowledge sharing and information exchange among members, summarizing topics discussed and highlighting good practices in evaluation, ensuring updates on events, resources, guidelines that are relevant to evaluation in agriculture, food security and rural development.

 

 

    • Dear Judith and all,

      The topic of how to move forward evaluation in agriculture is the one of ideas that brought to the development of this CoP!

      In 2019, FAO and EvalForward studied capacities for evaluation in Ministries of Agriculture, for which we interviewed officers in the Ministries of Agriculture of 23 countries. The study revealed a disparate situation depending on countries, including some with still very limited capacities in relation to evaluation, M&E or even Results-based management (here the link to the report and to the briefing note). Rwanda was not in our sample, but it looks to have well-established M&E systems and performance measurements (such as the annual imigo mentioned by Olivier).

      As known, evaluation can be useful to bring the depth of the analysis to the data and M&E system and help to identify weak spots. An institutional set up would provide the framework for carrying out strategic evaluations and support the demand for evidence and the willingness and capacity to use it.

      Based on the experience of countries that have succeeded in developing an evaluation system, some entry points to start the process can be:  

      • Finding influential leaders championing evaluation and who could influence the move towards an effective evaluation function at national and sectoral level.
      • Pilot evaluation with the involvement of public officers as an opportunity to proof the value of such exercise, including testing rapid evaluations to provide feedback in a reasonable time on pressing policy issues.
      • Connecting with countries that have established evaluation systems and initiate collaborations with initiatives such as Twende Mbele that aim to support these processes. 
      • Lobby for evaluation through key stakeholders, such as the Producer organizations, the VOPEs, academia and NGOs.

      In many countries, budget cuts in the agriculture sector have led to reduced investments in human resources and skills development undermining M&E functions and attempts to develop evaluation capacities in the sector. It is great to see that this is not the case of Rwanda. The centrality of agriculture in the economy should be leveraged to make the case to advance on the tools to improve evidence generation and use in the sector.

      We look forward to the views of other members!

      Renata

    • Dear all,

      Below are the highlights from this fruitful exchange.

      A special thanks to Paul for raising the topic and to the members who shared their knowledge! 

      KM and M&E have complementary objectives and should work in synergy

      • There are evident synergies between monitoring and evaluation (M&E) and knowledge management (KM): both contribute to organizational learning and effective programming designed to generate benefits for people and communities.
      • In practice, identifying linkages and complementarities can be more challenging.  M&E and KM require different roles and skill sets, and are often managed by separate units which, in turn, follow different organizational practices and cycles.

      Programmes and projects are moving towards better integration of these functions  

      • In the Global Agriculture and Food Security Program Missing Middle Initiative, Senegal, the M&E and the KM functions work jointly on data treatment, capitalization, and workshops on lessons learned. Knowledge Management activities involve members of the Coordination Unit to ensure that the lesson learning and knowledge capitalization form the basis for follow-up decisions.
      • The Nema Chosso Project (National Agricultural Land and Water Management Development) in the Gambia had M&E and KM working under separate units and faced challenges when it had to document and capitalize on successes and on lessons learned, use them to develop knowledge products around climate resilience interventions, and ensure publication in local press and social media. In response, the follow up project is merging the two units into one whilst including a specialized partner to support communications.
      • IFAD-led rural development and smallholder projects in East and Southern Africa have integrated online M&E platforms to support data collection used by the KM function to generate stories from the field and systematic learning. This approach was pushed at the time when projects where asked to demonstrate their policy impact and contribution to country targets.
      • An earlier KM Project developed at WFP included an after-action review methodology to facilitate learning for field-based operations, which benefitted from evaluation-based inputs. It was field-tested and refined in collaboration with experts from their Office of Evaluation.

      Challenges, risks and elements to consider

      • Challenges and tensions can emerge when KM and M&E work together but use different approaches to communication. Knowledge management, for example, might not fully communicate certain M&E evidence. Conversely, the programme and M&E team might d expect more consistent and specific communication to sustain programme level uptake, to a greater extent than is considered necessary by the knowledge management colleagues.
      • There may be instances when M&E and KM are skewed towards “appeasement reporting" to the donors, showing results under a positive light in order to promote funding and disregarding real beneficiaries impact. This may happen in the absence of ethical considerations and social responsibility.
      • There is still little evidence, if any, on how project initiatives and lessons feed into policy, despite the participation of policymakers.

      Integration at the organization level: an example from IFAD

      • The Independent Office of Evaluation of IFAD has acknowledged the natural relationship between evaluation and communication and KM. KM’s role is to foster the development of “feedback loops from evaluation to policy makers, operational staff and the general public” through the production and dissemination of communication products and the facilitation of learning events. These should help ensure that the information contained in evaluation reports is widely disseminated, thus triggering further learning and feedback.

      Way forward

      • To ensure that these functions complement each other, they have to be thoroughly planned and synced, preferably at the design stage of the project with clear lines of communication.
      • Steps in this direction could be: 1) discussing with programme management and beneficiaries what they need/expect from M&E and KM products; 2) Agreeing on a joint plan of M&E and KM activities, focusing on complementarities and sequencing; 3) Developing a plan for effective utilization and dissemination of M&E and KM products.

       

    • Dear all, 

      here is a summary of the discussion and main issues raised by participants. 

      • Gender should be addressed at the time of formulation of the development action and design of its results framework, including the choice of indicators and data collection methods.
      • If gender was not included in the programme design and baselines are not available, it will be more difficult, though still possible, for the evaluation to address it and for baselines to be replaced by historical analyses;
      • The type of evaluation (process, impact or outcome evaluation) and the target audience will influence the way the evaluation considers gender aspects.
      • Given the complexity and multisectoral nature of gender, evaluations of gender should include social analysis, which is necessary to understand social systems and the constraints that perpetrate the women’s condition in the context of food security. 
      • To evaluate gender equality and empowerment of women, gender-responsive methodologies, methods, tools, and data analysis techniques should be selected (see references below).
      • It is important to consider possible trade-offs related to gender: an example from a project that aimed to support women by assigning them livestock with higher productivity but did not consider additional workload and the missing links to market opportunities that could allow to translate the higher production into income.
      • Examples of gender-responsive indicators were referenced and on how to make indicators gender sensitive.
      • Women, along with other stakeholders, should participate in co-producing the evaluation framework and take part in evaluations themselves, in a participatory approach. The evaluator / evaluation team should play the role of a facilitator and recognize the experiential knowledge of stakeholders.
      • Gender evaluations in Africa, as in other regions, must consider that unfortunately the level of understanding of the gender approach still varies a lot, which does not facilitate its application.
      • Some country initiatives:
        • Benin: a study on the sensitivity of the national system of monitoring and evaluation in relation to gender allowed to identify national indicators by sector to evaluate the gender aspect and to incorporate norms and standards to take gender into account in all evaluations.
        • Burundi: the Ministry of Local Government is looking at how to integrate gender in the five-year development planning process of municipalities, defining the basic "gender" indicators that will be monitored and evaluated during the life of this plan.
        • Costa Rica: The Ministry of Planning worked with UN Women on a guide on how to evaluate gender and human rights to complement the evaluations carried out within the National Planning System of Costa Rica.

      References and links shared by participants / Références et liens partagés par les participants / Referencias y enlaces compartidos por los participantes 

      Other references / Autres références (en anglais) /Otras referencias (en inglés)

       

    • Dear members,

      Thanks to Georgette Konate Traoré from Burkina Faso for raising these concerns on evaluation of gender with the EVAL-ForwARD community. Please feel free to share your view on any of the issues she raises by replying to this email.

      I take the opportunity to invite you all to use this space in the weeks to come to:

      • Start discussions: post queries on topics of your interest inviting members to respond and to share their experiences and views.
      • Post information on and links to publications, events, calls and other activities of interest to other members.

      Discussions, resources and information shared will start forming the knowledge base of the EVAL-ForwARD community. Please contact me for practical support or any further information at Renata.mirulla@fao.org

      Related to quantitative and qualitative indicators to evaluate gender (question 3 below), FAO is adopting the pro-WEAI index, which helps assess women’s empowerment within agricultural developmental projects using 12 indicators. Here is a link to the WEAI webpage weai.ifpri.info  and to a recent presentation in FAO www.fao.org/webcast/home/en/item/4695/icode