Renata [user:field_middlename] Mirulla

Renata Mirulla

Free lance consultant
Italy

International development professional with a longstanding experience in knowledge management and communication for development. Focused in particular on communities of practice and policy dialogue, she launched EvalForward and facilitated the community until 2024.  

 

 

My contributions

  • Since 2009, the United Nations Development Programme Independent Evaluation Office has been organizing the biennial National Evaluation Capacities (NEC) Conference, engaging influential participants from government, civil society and academia from more than 180 countries, representatives from numerous United Nations agencies, as well as bilateral and multilateral partners.

    The Seventh NEC Conference took place at the International Training Centre of the International Labour Organization (ITC-ILO) in Turin, Italy, on 25-28 October 2022. Following the upheaval and socioeconomic fragility brought about by the global pandemic, the conference focused on “Resilient National Evaluation Systems for Sustainable Development”.

    The session

    This blog shares the

  • The old refrain that there are not enough skilled evaluators in Africa has passed its sell-by date. Realizing the need to offer solutions, the Centres for Learning on Evaluation and Results – Anglophone Africa (CLEAR-AA), the South African Monitoring and Evaluation Association (SAMEA) and the World Food Programme (WFP) recently joined forces to develop a tailored Emerging Evaluator Programme.

    The work immersion programme was launched in June 2021 during gLOCAL Evaluation Week, bringing six emerging evaluators on board for a year. The programme is taking them on a “deep dive” into evaluation work from different perspectives. For example, with WFP’s

  • Measuring without understanding “puts the cart before the horse”. If we measure something that is unclear or based on wrong assumptions, we are likely to end up with irrelevant data and or misleading conclusions. It is crucial, therefore, that we understand before we decide what to measure or, indeed, whether to measure at all. This was the main message by Silva Ferretti, the freelance evaluation consultant who opened the third EvalForward Talks, who astutely pointed out that this was not always the case and should be common practice in evaluation.

    The EvalForward Talks session underscored the need to reaffirm understanding

    • Dear Judith and all,

      The topic of how to move forward evaluation in agriculture is the one of ideas that brought to the development of this CoP!

      In 2019, FAO and EvalForward studied capacities for evaluation in Ministries of Agriculture, for which we interviewed officers in the Ministries of Agriculture of 23 countries. The study revealed a disparate situation depending on countries, including some with still very limited capacities in relation to evaluation, M&E or even Results-based management (here the link to the report and to the briefing note). Rwanda was not in our sample, but it looks to have well-established M&E systems and performance measurements (such as the annual imigo mentioned by Olivier).

      As known, evaluation can be useful to bring the depth of the analysis to the data and M&E system and help to identify weak spots. An institutional set up would provide the framework for carrying out strategic evaluations and support the demand for evidence and the willingness and capacity to use it.

      Based on the experience of countries that have succeeded in developing an evaluation system, some entry points to start the process can be:  

      • Finding influential leaders championing evaluation and who could influence the move towards an effective evaluation function at national and sectoral level.
      • Pilot evaluation with the involvement of public officers as an opportunity to proof the value of such exercise, including testing rapid evaluations to provide feedback in a reasonable time on pressing policy issues.
      • Connecting with countries that have established evaluation systems and initiate collaborations with initiatives such as Twende Mbele that aim to support these processes. 
      • Lobby for evaluation through key stakeholders, such as the Producer organizations, the VOPEs, academia and NGOs.

      In many countries, budget cuts in the agriculture sector have led to reduced investments in human resources and skills development undermining M&E functions and attempts to develop evaluation capacities in the sector. It is great to see that this is not the case of Rwanda. The centrality of agriculture in the economy should be leveraged to make the case to advance on the tools to improve evidence generation and use in the sector.

      We look forward to the views of other members!

      Renata

    • Dear all,

      Below are the highlights from this fruitful exchange.

      A special thanks to Paul for raising the topic and to the members who shared their knowledge! 

      KM and M&E have complementary objectives and should work in synergy

      • There are evident synergies between monitoring and evaluation (M&E) and knowledge management (KM): both contribute to organizational learning and effective programming designed to generate benefits for people and communities.
      • In practice, identifying linkages and complementarities can be more challenging.  M&E and KM require different roles and skill sets, and are often managed by separate units which, in turn, follow different organizational practices and cycles.

      Programmes and projects are moving towards better integration of these functions  

      • In the Global Agriculture and Food Security Program Missing Middle Initiative, Senegal, the M&E and the KM functions work jointly on data treatment, capitalization, and workshops on lessons learned. Knowledge Management activities involve members of the Coordination Unit to ensure that the lesson learning and knowledge capitalization form the basis for follow-up decisions.
      • The Nema Chosso Project (National Agricultural Land and Water Management Development) in the Gambia had M&E and KM working under separate units and faced challenges when it had to document and capitalize on successes and on lessons learned, use them to develop knowledge products around climate resilience interventions, and ensure publication in local press and social media. In response, the follow up project is merging the two units into one whilst including a specialized partner to support communications.
      • IFAD-led rural development and smallholder projects in East and Southern Africa have integrated online M&E platforms to support data collection used by the KM function to generate stories from the field and systematic learning. This approach was pushed at the time when projects where asked to demonstrate their policy impact and contribution to country targets.
      • An earlier KM Project developed at WFP included an after-action review methodology to facilitate learning for field-based operations, which benefitted from evaluation-based inputs. It was field-tested and refined in collaboration with experts from their Office of Evaluation.

      Challenges, risks and elements to consider

      • Challenges and tensions can emerge when KM and M&E work together but use different approaches to communication. Knowledge management, for example, might not fully communicate certain M&E evidence. Conversely, the programme and M&E team might d expect more consistent and specific communication to sustain programme level uptake, to a greater extent than is considered necessary by the knowledge management colleagues.
      • There may be instances when M&E and KM are skewed towards “appeasement reporting" to the donors, showing results under a positive light in order to promote funding and disregarding real beneficiaries impact. This may happen in the absence of ethical considerations and social responsibility.
      • There is still little evidence, if any, on how project initiatives and lessons feed into policy, despite the participation of policymakers.

      Integration at the organization level: an example from IFAD

      • The Independent Office of Evaluation of IFAD has acknowledged the natural relationship between evaluation and communication and KM. KM’s role is to foster the development of “feedback loops from evaluation to policy makers, operational staff and the general public” through the production and dissemination of communication products and the facilitation of learning events. These should help ensure that the information contained in evaluation reports is widely disseminated, thus triggering further learning and feedback.

      Way forward

      • To ensure that these functions complement each other, they have to be thoroughly planned and synced, preferably at the design stage of the project with clear lines of communication.
      • Steps in this direction could be: 1) discussing with programme management and beneficiaries what they need/expect from M&E and KM products; 2) Agreeing on a joint plan of M&E and KM activities, focusing on complementarities and sequencing; 3) Developing a plan for effective utilization and dissemination of M&E and KM products.

       

  • What about evaluation? Is it truly reaping the benefits of ICTs or merely looking at potential applications with caution? Should evaluators embrace the ICT revolution or shy away from engaging more closely? What about big data, their challenges and risks?

    Information and Communication Technologies for Development Evaluation, a recently-released book edited by Oscar Garcia and Prashanth Kotturi,  sheds light onto the relationship between ICTs and evaluation, discussing ways of harnessing the rapid evolution of technologies and the internet for development evaluation work, and includes examples of frontrunner applications.

    ICTs can certainly go a long way to improving the efficiency

  • Over 500 participants from 100 countries joined this 6th NEC Conference to exchange and learn how to advance national evaluation agendas and to discuss the use of evaluation to accelerate the implementation of the SDGs.

    Leaving no one behind amid the rise in inequalities in society is a mounting concern in both the developed and developing world. Captured in Sustainable Development Goal 10, addressing inequalities needs to go beyond looking at the increasing gaps between higher and lower levels of income and wealth. Inequalities, in fact, touch on a multidimensional set of social, cultural, geographical, ethnic and other drivers that

  • Its 650 registered participants from over 60 countries showed the vitality of the African evaluation community and the attention that evaluation attracts among national and international institutions. This success is certainly a result of 20 years of active engagement of the African Evaluation Association.

    The conference theme: “Accelerating Africa’s Development: Strengthening National Evaluation Ecosystems” stimulated debate on ways to develop and affirm Africa’s body of knowledge related to evaluation, on how to promote Africa-rooted and Africa-led evaluation through sharing African evaluation perspectives and to gain recognition of the role of Africa in the evaluation “ecosystem”.  The African evaluation

  • The aim of this EvalMENA pre-conference workshop was to take participants through a journey of reflection and deep thinking around the fundamentals of evaluation.

    This particular type of reflection is something that evaluation practitioners should regularly set aside time to do, as it is difficult to think deeply when fully involved in day-to-day work. As our practice develops, we should then take time to re-focus on particular aspects of evaluation and its distinctive features, in order to continuously nurture our evaluation knowledge and practice.

    Evaluation is confronted with a set of other practices (such as performance audits, evaluative research, etc.)

  • There were among some 200 evaluation experts, researchers and representatives from governments, international cooperation agencies and NGOs who gathered in Rabat for the 7th Conference and General Assembly of the Middle East and North Africa Evaluation Network (EvalMENA), which was held from 27 to 30 November 2018.

    The meeting was hosted by the Moroccan Evaluation Association and focused on the theme “National Evaluation Policies in the MENA Region: Institutional Measures and Regional Evaluation Processes".

    In the MENA region, there has been a growing recognition of the importance of evaluation of government policies and programs over the last years, with Morocco being

  • The African Evaluation Association’s 9th International Conference will be held in Abidjan, Cote D’Ivoire from 11 to 15 March 2019. The Conference theme is Accelerating Africa’s Development: Strengthening National Evaluation Ecosystems. It aims at expanding the “Made in Africa” evaluation approaches, supporting knowledge sharing, capacity development and networking opportunities among a wide range of organisations and individuals working on evaluation.

    EVAL-ForwARD will actively promote and support one of 12 work strands of the Conference, titled “Improving Agriculture and Food Security through Evaluation”.  

    Those interested in contributing to the conference may propose: papers, roundtables, posters, exhibitions and workshops

  • More than 200 evaluation experts, development partners and parliamentarians representing 70 countries from around the world convened at the EvalColombo 2018 forum in Sri Lanka, from 17 to 19 September, to discuss the benefits and challenges of using evaluation evidence in government policymaking.

    EvalColombo was jointly organized by the Global Parliamentarians Forum for Evaluation (GPFE), EvalPartners and the Sri Lanka Evaluation Association. The conference focused on the role played by parliamentarians in promoting the use of evaluation and in driving the Sustainable Development Goals (SDGs) agenda. The discussions brought up many issues to address if we wish to

  • Gender and evaluation of food security

    Discussion
    • Dear all, 

      here is a summary of the discussion and main issues raised by participants. 

      • Gender should be addressed at the time of formulation of the development action and design of its results framework, including the choice of indicators and data collection methods.
      • If gender was not included in the programme design and baselines are not available, it will be more difficult, though still possible, for the evaluation to address it and for baselines to be replaced by historical analyses;
      • The type of evaluation (process, impact or outcome evaluation) and the target audience will influence the way the evaluation considers gender aspects.
      • Given the complexity and multisectoral nature of gender, evaluations of gender should include social analysis, which is necessary to understand social systems and the constraints that perpetrate the women’s condition in the context of food security. 
      • To evaluate gender equality and empowerment of women, gender-responsive methodologies, methods, tools, and data analysis techniques should be selected (see references below).
      • It is important to consider possible trade-offs related to gender: an example from a project that aimed to support women by assigning them livestock with higher productivity but did not consider additional workload and the missing links to market opportunities that could allow to translate the higher production into income.
      • Examples of gender-responsive indicators were referenced and on how to make indicators gender sensitive.
      • Women, along with other stakeholders, should participate in co-producing the evaluation framework and take part in evaluations themselves, in a participatory approach. The evaluator / evaluation team should play the role of a facilitator and recognize the experiential knowledge of stakeholders.
      • Gender evaluations in Africa, as in other regions, must consider that unfortunately the level of understanding of the gender approach still varies a lot, which does not facilitate its application.
      • Some country initiatives:
        • Benin: a study on the sensitivity of the national system of monitoring and evaluation in relation to gender allowed to identify national indicators by sector to evaluate the gender aspect and to incorporate norms and standards to take gender into account in all evaluations.
        • Burundi: the Ministry of Local Government is looking at how to integrate gender in the five-year development planning process of municipalities, defining the basic "gender" indicators that will be monitored and evaluated during the life of this plan.
        • Costa Rica: The Ministry of Planning worked with UN Women on a guide on how to evaluate gender and human rights to complement the evaluations carried out within the National Planning System of Costa Rica.

      References and links shared by participants / Références et liens partagés par les participants / Referencias y enlaces compartidos por los participantes 

      Other references / Autres références (en anglais) /Otras referencias (en inglés)

       

    • Dear members,

      Thanks to Georgette Konate Traoré from Burkina Faso for raising these concerns on evaluation of gender with the EVAL-ForwARD community. Please feel free to share your view on any of the issues she raises by replying to this email.

      I take the opportunity to invite you all to use this space in the weeks to come to:

      • Start discussions: post queries on topics of your interest inviting members to respond and to share their experiences and views.
      • Post information on and links to publications, events, calls and other activities of interest to other members.

      Discussions, resources and information shared will start forming the knowledge base of the EVAL-ForwARD community. Please contact me for practical support or any further information at Renata.mirulla@fao.org

      Related to quantitative and qualitative indicators to evaluate gender (question 3 below), FAO is adopting the pro-WEAI index, which helps assess women’s empowerment within agricultural developmental projects using 12 indicators. Here is a link to the WEAI webpage weai.ifpri.info  and to a recent presentation in FAO www.fao.org/webcast/home/en/item/4695/icode