Serdar [user:field_middlename] Bayryyev

Serdar Bayryyev

Senior Evaluation Officer
FAO
Italy

My contributions

  • Developing effective, inclusive and gender responsive MEAL systems

    Discussion

    • Serdar Bayryyev

      Senior Evaluation Officer FAO
      Italy

      Dear Eriasafu,

      Thank you for raising these important questions and emphasizing the need for integrating the evaluation function with those of monitoring and learning. All these important functions are inter-connected and as such are integral parts of the effective project management cycle. It would, indeed, be the most effective approach if an evaluation could be integrated in the overall results-focused management system from the start, rather than be carried out as a “one-off” exercise occurring only at certain point of the project/programme cycle.

      I would like to highlight a few points that may stimulate further discussion on the questions that you have posted.

      Purpose and utility

      Before any monitoring and evaluation system is established or developed, its main users/stakeholders and developers need to be clear on the purpose and utility of the system. For example, the system that is mainly generating data for higher-level (corporate) reporting is different from the system that focuses on measuring benefits to the local community or beneficiary-level impact monitoring.  Any system developed need also be pragmatic, and take into account available capacities and resources, without creating or adding levels of complexity or functions that may not be used subsequently.

      Human rights and gender-related considerations.

      The monitoring and evaluation systems should adhere to the rights-based approaches, potentially identifying the effect of programmes on people realizing their human rights and identifying potential best practices of ensuring respect for human rights while implementing operations and programmes. The monitoring and evaluation processes and activities should duly integrate gender-related considerations, measuring any effect of programmes on women, girls, men and boys, and assessing benefits and deprivations. These can be achieved by integrating gender-sensitive indicators of performance in projects’ results and resources frameworks, identifying potential sources of gender-disaggregated information, and ensuring that gender-disaggregated data is collected and used for monitoring and evaluation, to the extent possible. 

      Addressing constraints

      Monitoring and evaluation activities should take due consideration of the projects and programmes’ operating environment and in particular potential constraints and risks. In emergencies and crisis-affected settings, there are multiple constraints and limitations, including issues of secure access to project sites and intended informants, availability and reliability of data, possible biases of key informants in crisis-affected settings due to their location or affiliation. These constraints can be addressed by using local partners/experts, using third party monitoring, and/or use of modern technology that allows remote monitoring and data collection (e.g. remote sensing via satellites, geospatial data available, digital data collection, mobile phone-based data platforms, remote sensing with satellites, etc.)

      Do No Harm

      In all contexts, and particularly in crises-affected settings, “Do No Harm” principles should be applied in planning and undertaking the monitoring and evaluation activities, bearing in mind potential sensitivities and tensions. The proposed M&E approaches should be planned in a manner that do not affect providers of monitoring and evaluation data, and do not exacerbate existing tensions, and/or worsen relationships between the informants and the other local actors/communities. 

      Kindest regards,

      Serdar Bayryyev (FAO)

  • Principles for a healthy relationship between evaluators and experts

    Discussion

    • Serdar Bayryyev

      Senior Evaluation Officer FAO
      Italy

      Dear Ibtissem,

      Thank you for posting a blog on this important issue. The complexity of development interventions requires that evaluation teams have various types of expertise. Yet, it is often quite difficult to secure right mix of expertise, and it is even more challenging to ensure that this expertise is effectively utilized during a short span of the evaluation process.

      Evaluation managers/leaders (hereinafter referred as Evaluation Team Leaders) need to make their thoughtful decisions on what expertise they need to obtain for the evaluation team, and what combination of expert knowledge and experience would work best for their evaluation.  In making these decisions and in leading the evaluation processes, Evaluation Leaders may indeed be guided by the five competency domains highlighted by Ibtissem  (or the Competencies for Canadian Evaluation Practice), which resonate with the 2016 UNEG Evaluation Competency Framework (http://www.unevaluation.org/document/detail/1915 ).  

      The role of the evaluation leader in managing the teams of experts, is indeed strategic and challenging, and I would like to add just a few considerations in this regard, to stimulate further discussion on this important topic:

      Establishing strong leadership

      Evaluation Team Leaders need to effectively lead the evaluation team, coordinating effective engagement of each expert in the team, validating their individual inputs and undertaking quality controls at key milestones. Continuous communication with each team member throughout the process is essential to ensure that any questions are potential risks of drawbacks are timely and effectively addressed.

      Ensuring compliance with organizational standards

      The Evaluation Team leaders need to require compliance of each team members with applicable rules and regulations, and standards of conduct established by the organization, and establish quality parameters of expected deliverables. In collaboration with each team member, Evaluation Team Leader should ensure that there is a shared understanding of what is to be evaluated and how, clarifying how organizational standards and processes are to be applied, and what is expected from each individual team member.

      Focus, focus, focus

      Biases are unavoidable, yet their influence can be minimized by focusing on evaluation objective and clearly establishing the scope of evaluation analysis. Furthermore, it should be made clear that any expert inputs and viewpoints need to be validated and supported with solid and relevant evidence.

      Building relationships

      Teambuilding skills are essential, to ensure that team members support each other and works towards achieving common objective.

      Engaging external expert panels

      If at all possible, external panels of experts may be engaged to provide feedback on key thematic areas of your evaluation analysis. Recognized experts from counterpart organizations may assist by serving as high-level expert panel members, to review findings produced.

      These are just a few examples, and I anticipate continued discussion on this topic. 

      Kindest regards,


      Serdar Bayryyev

       

  • How to evaluate science, technology and innovation in a development context?

    Discussion

    • Serdar Bayryyev

      Senior Evaluation Officer FAO
      Italy

      The need to evaluate science, technology and innovation in a development context

      The significance of science, innovation and research in supporting global efforts towards more sustainable and climate-friendly development is growing. There is an urgent need for relevant science, quality research and innovations that are ground-breaking, as the world is experiencing new and unprecedented challenges and crises. Evaluation of the quality of evaluation and research is essential in determining the usefulness of and effectiveness of science, innovation and research activities.  Evaluation findings should help decision-makers in determining important priority areas for further investigation and facilitate the decision-making on allocating resources for future research activities.

      Key limitations

      The evaluation of science and research is, however, quite complicated, and facing numerous methodological challenges. For example, assessment of the relevance and significance of scientific or research products is mostly based on the use of bibliometric methods. This is a quantitative method that may in fact produce solid evidence-based findings, yet its use is constrained by major limitations.  For example, not all science, innovation and research products are included and properly recorded in the bibliographic databases, or not even published, hence not all products can be assessed.

      The bibliometric methods often are based on calculating average number of citations, which also presents the basis for some biases. For example, there is sometimes an overly exaggerated attention to a specific author, who is known for previous work in a specific field or is affiliated with institutions that have strong political or financial support. In terms of citations, some authors may also deliberately exclude certain reference materials from their publications. Henceforth, whenever the bibliometric data analysis is used, it should be used with caution and should be combined with the use of other methods for validity purposes.

      The other major limitation is that in today’s complex world of science and innovation, there are various standards or criteria of assessing quality of research, science and innovation in various parts of the world, and various parts of science and innovation.

      Assessment of science and research products may also be biased due to the differences in political affiliation, beliefs, culturally or religiously-based perceptions of those who undertake these assessments or evaluations. 

      Key considerations

      As this stream of evaluation function is still evolving, there are few key considerations that need to be taken into account in undertaking relevant evaluation, or in developing appropriate evaluation tools and methods.

      Assessing relevance/significance of science and research.

      Assessment of relevance or significance of science, innovation and research products need to take due consideration to the context in which these products are to be used. What works in one context may not be suited for the other, and what constitutes innovation and ground-breaking science varies substantially depending on the intended use or users.

      Assessing effectiveness (or quality)

      In assessing the effectiveness of the research and scientific analysis, the key is to assess “influence” of these activities, or extent to which the science, innovation and research products have influenced the policies, approaches or processes.   

      Assessing the degree of “networking”, i.e. the degree to which the researchers and scientific institutions have interacted with all relevant stakeholders, including those that may have had a “negative” or opposing stance to the subject research theme/topic.

      Assessing Transformational nature

      In today’s world, perhaps, the most important criteria for assessing the relevance, use and efficiency of science, innovation and research activities, is whether these activities cause truly transformational change, or at least trigger important policy discourse on moving towards such transformational change.

      The above-written are suggestions for consideration, that are aimed at stimulating further feedback into this important discussion.

      Kindest regards,


      Serdar Bayryyev

      Senior Evaluation Officer

       

       

  • Can we use an evidence-based, evolving Theory of Change to achieve "local learning” during project design?

    Discussion

    • Serdar Bayryyev

      Senior Evaluation Officer FAO
      Italy

      Dear John,

      In my view, the Theory of Change, or logical framework, or any other method used to guide the design of the development intervention (project) is critical. These methods should be based on comprehensive analysis of the development context and the critical issues to be addressed to meet the needs and desires of local communities.

      Multilateral organizations have some examples of "community-driven development" projects. For example, the International Fund for Agricultural Development (IFAD) has recently published an evaluation synthesis of "Community-driven development in IFAD-supported projects" (IFAD, April 2020), which is based on the review of case studies of community-driven development projects. The theory of change used for this synthesis was based on the assumption that social capital and empowerment are at the center of the community-driven development approach. This theory of change assumes that participatory implementation process "...is expected to achieve a truly sustainable transformation of rural livelihoods by building poor peoples' capacities to make use of a wider range of livelihood options and by transforming community-government relations to better support people-centred development processes". This theory of change is illustrated in Figure 2 on page 5 of the synthesis paper accessible via the following link:

      https://www.ifad.org/documents/38714182/41898849/ESR+CDD+-+final+with+c…

      Kindest regards,

      Serdar Bayryyev

      Food and Agriculture Organization

       

  • Cohérence des interventions dans le secteur agricole

    Discussion

    • Serdar Bayryyev

      Senior Evaluation Officer FAO
      Italy

      Dear Nabyouré Jean Stanislas OUEDRAOGO,

      Thank you very much for raising this strategic question, which is at the heart of the discussions held today in the context of best practices and development pathways towards achieving the Agenda 2030. All countries, all developments stakeholders, communities and individuals should act in collaborative and productive partnerships to make progress towards achieving the Sustainable Development Goals. 

      The proliferation of development interventions could indeed become a negative factor, if their universe is composed of projects that do not exploit potential synergies and apparent complementarities, and are not based on partnerships based on solid analysis of mutual benefits generated from joining forces, capacities and resources toward common goals.

      We live in communities and environments that are affected by a multitude of factors that are interconnected, broadly defined as social, economic, environmental, health-related and other development factors. Accordingly, the development interventions should be developed with due attention top and in full consideration of these inter-linkages, inter-connections and trade-offs. There are examples of good practices used by development organizations to coordinate and consolidate the universe of development interventions to take full account of development context and exploit potential complementarities in addressing the inherent interlinked development issues and challenged. Some of these examples are show below: 

      • The United Nations Development Assistance Framework, if planned with due consideration of local context and with robust analysis of development challenges, would guide UN entities and other development actors  in producing coherent and well-coordinated package of development support towards achieving national development goals and objectives.
      • United Nations Global Compact that helped creating multi-stakeholder initiatives, supported by the UN, international financial institutions, private businesses and small and medium-sized enterprises, to address development challenges in a more coordinated manner.
      • The United Nations pooled funding mechanisms, which serve as channels for directing flows of development and humanitarian assistance from diverse groups of external factors through national budgeting and financing schemes, helped to improve effectiveness, reduce duplication and promote alignment among a wide range of actors. 

      These are just a few examples from the recent past. The new generation of the United Nations Sustainable Development Cooperation Frameworks (https://unsdg.un.org/sites/default/files/2019-11/UNSDG-SDG-Primer-Repor…), are being developed to move the development aid paradigm from assistance to cooperation, and from individual contributions by development agencies to a collective and coherent response to countries' opportunities, gaps and challenges. 

      The new Cooperation Frameworks will consider development priorities from multiple perspectives of the diverse groups of stakeholders, taking their views as the basis for developing coherent development support package. In doing so, the Cooperation Frameworks will aim at developing interventions that take full consideration of potential effects among different sectors. If done and implemented right, these Cooperation Frameworks will guide the transformation of development projects into a coherent and well-coordinated package of development assistance, aligned with the national plans towards achievement of Sustainable Development Goals.

      Just a reflection.

      Kind regards,

      Serdar Bayryyev

      Food and Agriculture Organization

       

       

       

       

       

       

  • How to use Knowledge Management to strengthen the impact of Evaluation on smallholder agriculture development?

    Discussion

    • Serdar Bayryyev

      Senior Evaluation Officer FAO
      Italy

      Dear Paul,

      Thank you for raising this important issue, frequently faced by development practitioners. While both functions, the M&E on one hand, and the knowledge management, on the other, contribute to organizational learning and effective programming towards generating intended benefits to the people and communities, these functions quite often are managed by different units and follow different organizational practices and cycles. To ensure that these functions complement each other, these have to be planned and synced, preferably at the design stage of the project. For example, the results from periodic monitoring conducted at quarterly intervals, may be used to produce knowledge products, such as newsletters and case study brochures to raise awareness on the results achieved. Annual monitoring exercises could be informing another type of knowledge products - such as the lessons learned. Utility of evaluations can be enhanced by wider dissemination and broadcasting (e.g. via social media, TV, radio), with support from knowledge management professionals.

      In a effort to ensure that both M&E and KM teams work collaboratively together, the following initial steps could be considered and applied: 1) discussing with programme management and beneficiaries their needs in M&E and KM products; 2) Agreeing on a joint plan of M&E and KM activities, focusing on complementarities and sequencing; 3) Developing a plan for effective utilization and dissemination of M&E and KM products.

      Kindest regards,

      Serdar Bayryyev

      Senior Evaluation Officer (FAO)

       

  • Evaluation to support the “leaving no one behind” agenda: insights from the UNDP NEC Conference 2019

    Blog

    Over 500 participants from 100 countries joined this 6th NEC Conference to exchange and learn how to advance national evaluation agendas and to discuss the use of evaluation to accelerate the implementation of the SDGs.

    Leaving no one behind amid the rise in inequalities in society is a mounting concern in both the developed and developing world. Captured in Sustainable Development Goal 10, addressing inequalities needs to go beyond looking at the increasing gaps between higher and lower levels of income and wealth. Inequalities, in fact, touch on a multidimensional set of social, cultural, geographical, ethnic and other drivers that

    Posted by

  • Challenges of evaluation

    Discussion

    • Serdar Bayryyev

      Senior Evaluation Officer FAO
      Italy

      Dear Hynda,

      You have raised a very important question, which affects the quality of evaluation work. Evaluations of development programmes in the broadly defined areas of rural development, agriculture and food security are inherently complex. The assessments of results in these areas are affected by a multiplicity of biophysical, economic, and social systems and factors. There are different types of constraints and challenges in evaluation work that depend mostly on the context of the programmes or policy work being evaluated.  For example, accurate and timely assessments of potential impact and development change may be affected by the remote location of project sites, social stratification of rural communities, time required to produce productivity gains, adoption capacities of local communities, and many other factors.

      Evaluators often encounter issues with availability of baseline data, or information on the prevailing conditions of the development situation at the start of the projects or programmes addressing food security and agriculture development. This issue could be addressed by reconstructing baselines, for example, using ‘recall’ technique, i.e. requesting key beneficiaries or stakeholders to recollect information about these conditions in the past.

      Security situation in the country may also have a huge impact on the access to data and methods we chose for evaluation. The choice of evaluators could also be highly limited, as not all may have necessary clearance to visit high-risk areas, or experience in working in similar situations. 

      Accessibility of project sites may also be restricted or banned. To address these constraints, local consultants with access to restricted zones may provide support in data collection, and potential alternative evaluation methods could be also considered. In recent FAO’s evaluation of the large irrigation rehabilitation programme in Afghanistan, evaluation team faced a constraint of accessing some of the programme sites. The team opted for alternative method by using the open-source data from Google to assess the potential impact of the programme on the livelihoods in those specific sites. Google Earth maps were utilized to measure the expansion of the irrigated area and the vegetative cover along different sections of the rehabilitated canals. The methodology for measuring these areas was also using preliminary information from enumerators in the field who had access to the restricted zones, and were engaged in supporting collection of necessary data and information for the evaluation (e.g. the GPS coordinates of the irrigated areas in the vicinity of the irrigation canals). Then this information was analyzed based on historic data available from Google Earth on before- and after-project conditions and the changes based on vegetative cover at different periods during a year.

      These are just a few highlights of the constraints and challenges that evaluators may encounter in their work and an example of possible ways to address those. The range of such constraints is quite broad, and we encourage all members of this community to share their experiences in addressing different types of constraints and limitations.

      Kind regards,

      Serdar Bayryyev,

      Evaluation Officer

      Food and Agriculture Organization (FAO)