Evaluation and, more generally, Results-Based Management (RBM) are amongst the best-known instruments to strengthen accountability and learning, and, ultimately, improve policies and practices.
How are these applied by institutions concerned with agriculture, and are officials working in this sector equipped with adequate capacities and resources to carry out evaluation?
Knowledge seems to be scarce on these questions. Hence, the FAO Office of Evaluation and EvalForward started exploring the dynamics of evaluation within Ministries of Agriculture and their relations with other institutions who play a role.
While the study is ongoing, the Francophone International Forum on Evaluation - FIFE2019, held in Ouagadougou from 11 to 15 November, provided an opportunity to exchange ideas on these questions with Conference participants. In a Round Table with representatives from M&E units in Ministries of Agriculture in Cameroon, Benin and Burkina Faso, and from the AVANTI project, who presented relevant experiences, I shared some preliminary results emerging from the study as a basis for the discussion. These are presented in brief, to generate further reactions from the community.
A first finding is that evaluation is still largely led at central level, and although most countries contacted have M&E units within the Ministry of Agriculture, their roles in measuring results in the sector vary widely across countries and are often limited.
These units are most often tasked with a broad mission, covering both the planning of activities, including budgeting, and programme performance against work plans. In many cases, their evaluation mission, as final stage of the RBM cycle, is not prioritized. Evaluation calls for specific skills and investments. Yet, budgetary resources dedicated to these units are usually insufficient for a genuine investment in evaluation, and the staff made available to the M&E units of the Ministry of Agriculture are limited in number (12 staff nationwide on average) and lack appropriate training.
Evaluation is more systematic in the context of projects, when it is levied by external donors who add a budget line, and when it is a condition of future aid. Evaluation is thus rarely conducted at the level of national policies and programs, being in financial competition with programming activities, most often prioritized by policy makers looking to show tangible deliverables in their short term time span.
The pivotal influence of political ‘champions’ of evaluation is also demonstrated by the few countries where evaluation has been well developed and could be said to be institutionalized: they owe it to the support of conscious interlocutors at the highest level of the government.
Looking at the relationship between the Ministry of Agriculture and the central units responsible for evaluation in the countries, it looks like this is usually a formal reporting system rather than a collaboration between evaluators. In addition, the dynamism of the central units is a factor that strongly influences their ability to promote the evaluation culture within countries, and especially in sector ministries.
To date, three types of situations are emerging:
- Advanced countries, where national institutions sustain an evaluation function in the agricultural sector;
- The countries or M&E units of Ministries support results-based monitoring but do not practice evaluation;
- Countries where the concepts underpinning RBM do not yet seem to have penetrated institutions of the sector.
The preliminary results of the study are summarized here (in French). These initial results will have to be confirmed with the collection of more cases, with the aim to finalize the study early in 2020. Some questions remain open:
- Is there a need to internalize the evaluation function in sector-based Ministries, to ensure that the sector policies and programmes are appropriately evaluated? Or instead, should the priority be on better linking professional evaluators to these public institutions, while simply increasing the sectoral ministries’ capacities to plan evaluations and maximize their benefits? The latter would also responds to the need for evaluation to remain independent.
- Which levers may influence the emergence of evaluation champions within public institutions and which may help to ensure adequate investments in the evaluation function?
We invite members of the Community to share their perspectives around these and the broader question of institutionalization of evaluation in the agriculture sector.
John Hugh Weatherhogg
Agricultural Economist ex-FAOThe main weakness in M&E is the weakneess in collection of field data and its subsequent analysis. This is the aspect which needs the most attention. Local universities are the best way for these surveys to be carried out, as presented in the attachment
Diagne Bassirou
Responsable Suivi-EvaluationResponsable Suivi-Evaluation WACA- West Africa Coastal Areas Management programAu Sénégal, comme dans beaucoup de pays la disponibilité des données agricoles reste toujours un challenge. LA FAO Sénégal dans le cadre du programme GAFSP MMI, est entrain d'expérimenté une nouvelle initiative dans ce domaine pour faciliter la disponibilité des données du secteur agricole. le principal souci est l'agriculture des exploitation familiale ou des petits exploitants agricoles qui représente plus de 60% de la production agricole au Sénégal. Ainsi dans cette nouvelle initiative, les données sont cherchés à la base par le biais des organisations de producteurs. il 'agit de former des animateurs et superviseurs issus des organisations de producteurs en collecte de données quanti et quali d'évaluation et de suivi agricoles via des tablettes Android qui sont mis à leurs dispositions contenant l'application KOBOcollect. dans cette perspectives avec l'entrée des organisations de producteurs qui sont sous la supervision des Directions régionales de l'agriculture nous pensons établir un dispositif qui permettra d'avoir en permanence des données actualisées. Aujourd'hui, nous sommes en phase pilote de cette initiative dans la région de Tambacounda avec le projet GAFSP MMI inscrit dans une logique de renforcement de l'autonomisation des organisation de producteurs pour contribuer aux renforcements de la productivité des petits exploitants qui sont affiliés. la mise à l'échelle de cette nouvelle dispositive avec l'entrée OP, pourrait permettre au Sénégal de faire face à ce challenge de données de la production agricole au niveau national. Ainsi l'expérience pourrait aussi partager par d'autres de la sous région voire ailleurs dans le monde.
Tim Njagi
Research Fellow Tegemeo InstituteHi Aurelie,
our experience shows that although we have gained some ground with policymakers to make them recognise the need for investment in data, we need more traction to get them to commit resources. An ideal strategy in my view is to get a partner willing to put up investments for a pilot. once we demonstrate the effectiveness of such a system, it would be easier to attract more resources especially when demonstrating value for investment.
Thanks,
Tim
Aurelie Larmoyer
Senior Evaluation officer WFPDear friends,
Thank you for these very rich and sensible comments, which demonstrate a solid experience and thinking on these issues.
Dear Tim, the idea of a system that relies on enabled collaborations with research institutes is indeed interesting. You point to the need to unifying data systems, and investing in technology, alongside with the necessary capacity development: does your experience help identify who could support such investments? Is this a matter of inciting the political or finding financing partners?
As regards the need to better connect evaluators with the Ministry of Agriculture, and finding change agents or leaders, as also pointed by our colleagues from AVANTI, we agree these are important levers to promote effective evaluation. The question remains: what incentivizes these positive spins? If anyone want to share any elements of success in this respect, we will welcome your thoughts!
Thank you, Aurelie
Abdulkareem Lawal & Kai Schrader
AVANTI ProjectThe AVANTI AG-Scan approach aims to help strengthen the ability of national governments to effectively engage in results-based management (RBM). It starts by creating the conditions for a meaningful self-assessment by governments and other key players around monitoring, reporting and acting in support of Agenda 2030’s Sustainable Development Goals (SDGs) in the agricultural and rural development sector.
Our experiences from conducting AG-Scans in ten countries so far shows varied levels of M&E capacity in countries and stakeholders are often very clear about existing challenges, including those relating to routine evaluations. There also exist good knowledge of RBM – although there are slight variations in how countries use the language of RBM; some simply situate M&E within the broad confines of RBM and stick to the “M&E language”. Overall, key challenges relate to political will and funding to undertake the evaluations – there is evidence that agriculture ministries do have M&E plans in place, which include periodic evaluations. However, for evaluations to take place, there has to be the commitment and support at senior leadership levels. In addition, there are often capacity gaps to undertake such evaluations, and ministries often cannot afford to engage independent consultants to undertake evaluations. Moreover, taking findings from evaluation into account during the planning and budgeting phase is still a rare habit.
The AG-Scan process advocates for behaviour change that is focused on a knowledge sharing and learning culture and achieving results and robust evidence for decision making; and stresses the role of leadership and management for making this effective. In the countries where AG-Scans have taken place, policy makers alluded to the challenges that they face such as political restrains and institutional, routines, personal motivations and limitations on resources. There was, nonetheless, evidence of commitment and involvement from leaders and senior ministry officials to create and drive a culture of RBM and managing for results, either at the level of the agriculture and other ministries or at the level of agencies within the ministry.
Abdulkareem Lawal & Kai Schrader (AVANTI)
Seyni SANOGO
Chargé du bureau des ressources humaines à l’inspection de saponéSur l'institutionnalisation au sein du ministère de l'agriculture, j'estime qu'il faut susciter l'engagement des acteurs de l'agriculture. Faire prendre conscience de la place des évaluations dans l'amélioration de leur performance est un élément fondamental pour leurs activités quotidiennes, leur service et pour leur ministère. Cela pourrait se faire par l'identification des motivations des acteurs en matière d'évaluation ou suscité des motivations en matière d'évaluation par l'information et la sensibilisation sur qu'est-ce que le suivi, l'évaluation, leur lien et l'importance de l'évaluation pour les acteurs eux-mêmes et pour la réussite de leur mission ainsi que pour le ministère.
Ensuite de les sensibiliser sur le concept de la GAR et ses avantages par rapport au PPO. Enfin identifié les capacités existantes pour l'évaluation afin des outillés en suivi-évaluation. Aussi, de chercher à savoir les capacités financières et organisationnels de la structure devant assurer le suivi-évaluation: si ces capacités sont inexistantes nous devrons songer à trouver des créneaux pour les accompagner à en acquérir.
Enfin, la présence des champions pourrait davantage consolider la pratique et la culture de l'évaluation. Pour cela, il faudrait que ces champions aient du poids et de la voix dans les structures. Cela pourrait se faire par la prise des lois encadrant et protégeant les évaluateurs de telle sorte qu'ils ne puissent pas être influencés dans leurs missions, que les résultats des évaluations puissent être utilisés véritablement pour des changements innovants, que les champions puissent être responsabilisés dans leur structure si possible et introduire des questions d'évaluations dans les lettres de missions des agents du ministère de l'agriculture. A ce niveau, les professionnels de l'évaluation pourront être un pilier incontournable pour les champions: pour le partage d'expérience et le raffermissement de la pratique et la culture de l'évaluation au sein de ce ministère.
Richard Tinsley
Professor Emeritus Colorado State UniversityMy biggest concern for all host government activities such as managing evaluations is "is there sufficient financial resources to do an effective job or if not will tasked be highly compromised by officers attempting to complete evaluation based on personal expectations rather than solid fact". The concern is based on most host governments FAO works with are what I refer to as financially suppressed economics where so much of individual wages or labor goes to meet basic subsistence needs that there is not sufficient discretionary funds to form a viable tax based to support government services such as evaluations. This then results in a lot of informal income activities interfering with effective governance. Governments are for all practical purposes financially stalled. Please review the webpages:
https://smallholderagriculture.agsci.colostate.edu/financially-suppress…
https://smallholderagriculture.agsci.colostate.edu/financially-stalled-…
https://smallholderagriculture.agsci.colostate.edu/impact-of-financiall…
https://smallholderagriculture.agsci.colostate.edu/informal-income-oppo…
Tim Njagi
Research Fellow Tegemeo InstituteIn Kenya, like other developing countries, data within the agriculture sector remains a challenge. This has constrained the RBM model where a lot of focus remains on the processes rather than moving on to outcomes. Many of the ministry's reports give a lot of prominence to the processes and targets achieved for process indicators, and use this to claim success even when independent studies have pointed to little or no progress has been achieved in outputs and outcomes of these processes. At Tegemeo Institute, we have been tracking key indicators and trends within the agriculture sector and have learned some key useful lessons. First, there is need to have a unified data system. A unified system becomes more relevant in devolved systems like Kenya's where the devolved units are semi-autonomous. Second, there is need for continuous capacity development within the systems to ensure that quality data is continuously collected, managed, analysed and shared. Third, leveraging on technology ensures to ensure that credible and quality data is collected and used to inform decisions.
The Ministry of Agriculture should strengthen the linkages with evaluators within the public and private sectors to improve the quality of evaluations while maintaining the independence of evaluators to make assessment from the evaluation studies and recommend action to be taken to maximise impacts. In addition, identifying change agents who disrupt the BAU position is necessary to be outcome-oriented and use lessons from evaluation to design and implement impactful policies and programs