Challenges of evaluation

@FAOEvaluation

Challenges of evaluation

[English translation below]

Bonsoir à tous,

Je travaille au Ministère des Finances Algérien et j’ai récemment rejoint cette Communauté de Pratique.

Je viens vous proposer d'ouvrir un débat qui m'intéresse particulièrement.

Il s'agit de parler des différentes contraintes/limites que vous rencontrez lors des différentes évaluations de programmes et interventions de développement rural, agriculture et sécurité alimentaire.

Comment les contournez-vous? Et quelles sont celles que vous n'arrivez pas à contourner ?  

Merci à tous.

Hynda Krachni

Ministère de Finances

Algeria

***

Good evening everyone,

I work at the Algerian Ministry of Finance and recently joined this Community of Practice.

I would be particularly interested in opening a debate on the different constraints / limits that you encounter during the different evaluations of programs and interventions on rural development, agriculture and food security.

How do you get around them? And which are the ones you cannot get around?

Thank you all.

 

This discussion is now closed. Please contact info@evalforward.org for any further information.

Thanks, dear Naser, for bringing this issue again to the forefront.

We should not stop 'hammering' that evaluation cannot and should not be disconnected from monitoring and we should do all we can to connect them from the start, at the moment of developmental action formulation, albeit a project, a programme, or a policy.

It is a fact - and nobody can deny that - that most of the time developmental actions are:

  • lacking a clear theory of change, and hence a thorough and sound results framework; and,
  • not founded upon a robust M&E system which will systematize monitoring from the start and prepare the ground for evaluation.

But why this is still happening after eighteen years of the MDG endeavour?

Because of weak or insufficient M&E capacities within national systems in almost all developing countries, but also a 'stricking' reluctance and lack of political will to adopt a national M&E framework for national development. Again this fear of M&E as a control and audit system is in the air...

Besides, whenever international organizations is pleading the need to build national capacities on this issue, stress and focus are rather put on evaluation and very low consideration is allotted to monitoring.

And again, I would claim that monitoring and evaluation - and not monitoring or evaluation - are the two 'legs' of a system on which will stand a developmental action seeking to ensure achieving its expected results; choosing the one or the other would just mean that our development action - as a person standing on one leg - will certainly fall short of achieving its expected results.

That's what I wanted to say as a rejoinder to Naser's contribution...

Mustapha

Bonjour chers amis,

Je suis vraiment contente. Les dernières contributions de Mustapha, Nasser et Raoudha sont tout à fait pertinentes. Merci de les partager avec nous.

Le respect des principes de l'évaluation est capital dans la conduite de toutes les évaluations pour en garantir l'objectivité et par conséquent apporter les réponses aux questions posées et y introduire les correctifs nécessaires.

Le riche débat qui a découlé de ma question, met en exergue la grande diversité des contraintes rencontrées et leur impact sur les résultats de l'évaluation.

Certaines incombent aux évalués, d'autres aux évaluateurs et d'autres encore aux politiques et dispositifs fixés.

Encore une fois, merci à tous pour votre réactivité et la pertinence de vos contributions.

Hynda

Alger

***

Hello dear friends,

I am really happy. The last two contributions of Mustapha, Nasser and Raoudha are quite relevant. Thank you for sharing them with us.

Compliance with the principles of evaluation is crucial in the conduct of all evaluations to ensure objectivity and therefore provide answers to the questions asked and introduce the necessary corrective measures.

The rich debate that emerged from my question highlights the great diversity of constraints encountered and their impact on the results of the evaluation.

Some fall under the responsibility of the evaluators, others to the evaluators and others to the policies and devices set.

Once again, thank you all for your responsiveness and the relevance of your contributions.

Hynda

Alger

Bonjour de la Tunisie,

Merci beaucoup cher-e-s collègues pour ces échanges riches et fructueux. Je pense que parmi les contraintes majeures de l'évaluation est la culture d'évaluation elle même quand elle n'est pas vraiment bien ancrée. La plupart du temps on se contente d'une auto-évaluation qui n'est pas fondée sur une base scientifique et ne répond pas ou peu à des standards internationaux en matière d'évaluation. Il faut qu'on soit convaincu que l'évaluation va nous permettre soit de rectifier le tir pour reprendre le bon chemin ou de continuer sur la même voie qui s'avère la bonne, et dans les deux cas ça va nous permettre d'avancer. Ceci nous mènera à la deuxième contrainte qui entrave l 'évaluation, à savoir la personnalisation du projet ou programme. En effet la personne qui est chef de file du projet (je parle là des institutions publiques) n 'accepte pas que, soit disant, son projet soit évalué parce qu'il croit que cette évaluation va toucher sa crédibilité et là c'est vraiment un grand problème et ça renvoie à la question relative à la culture d'évaluation.

Trés bonne journée

Raoudha Jaouani

Directrice en charge de la Pauvreté, des politiques sociales et de la migration

Ministère du Développement, de l'Investissement et de la Coopération Internationale

Tunisie

***

Hello from Tunisia,
 
Thank you very much, dear colleagues, for these rich and fruitful exchanges. I think that one of the major constraints of evaluation is the evaluation culture itself when it is not really well anchored. Most of the time we are satisfied with a self-assessment that is not based on a scientific basis and does not meet or only partly meets international evaluation standards. We need to be convinced that the evaluation will allow us either to adjust the direction to resume the right path or to continue on the same path that is the right one, and in both cases it will allow us to move forward. This will lead us to the second constraint that hinders evaluation, namely the personalization of the project or program. Indeed the person who is the leader of the project (I am talking about public institutions) does not accept that, supposedly, his project is evaluated because he believes that this evaluation will affect his credibility and this is really a big problem that goes back to the question of the evaluation culture.
 

Dear Colleagues,

Good morning from Palestine.

The evaluation challenges in rural development/ food security or agriculture are not much different from what is related to other sectors and even evaluation of policies.

I agree with what colleagues mentioned earlier. From my experience in evaluating agricultural projects in Palestine I found one major issue complicating the evaluation which is the design of projects. Even with the international organizations, results frameworks of programs/ projects are not well defined. Indicators are not well adopted or formulated. The whole theory of change is not clear. This is reflected on the evaluability of the program. For examples, baselines studies when present, are not related to indicators. 

Another major issue: clients of the evaluation (implementers) do not have clear understanding of the evaluation process and methodological approaches. Therefore, the ToR would not be clear, the expectations from the evaluation become not realistic, and as earlier stated, the program design does not allow good monitoring. 

To end up with, a good evaluation implementer should start planning M&E at the first phase of the program cycle, have enough resources, do right things at the right time, especially monitoring. 

Clients of evaluation should understand evaluation practices and know that no M&E can be done without working together with the evaluation team. And they should give enough time for evaluation, not leave it to the last month of the project.

The discussion on this issue never ends. I think evaluation networks (like EvalMENA) should reach clear set of recommendations to enhance evaluation culture and reach common understanding of the M&E on both supply and demand sides.

Good luck

Naser Qadous

Palestinian Evaluation Association

[English translation below]

Bonjour à tous,

Un grand merci à notre chère Hynda pour avoir ouvert un débat fort intéressant sur les défis et les contraintes qui entravent l'émancipation de l'évaluation dans certains pays. Tout ce qui a été dit est tout à fait valable dans son intégralité, mais néanmoins le manque de compréhension de la fonction d'évaluation, telle que l'évoque Hynda, très souvent perçue comme un contrôle et forçant beaucoup d'individus à des positions de résistance pour différentes raisons, reste un des défis qu'il est nécessaire de prendre en charge. De par ma modeste expérience dans les différents ateliers de formation sur la Gestion axée sur les Résultats que j'anime, dans son volet suivi-évaluation, je commence toujours par démystifier les fonctions de suivi et d'évaluation parmi les participants en posant une question simple: est-ce que nous faisons du suivi-évaluation dans notre vie quotidienne? Et j’engage ainsi un débat franc et serein avec les participants en les emmenant à évoquer des exemples de la vie courante où l'être humain pratique le suivi-évaluation de manière assez intuitive et fortuite. L'exemple d'un voyage en voiture vers une destination où on n'a jamais été pour y arriver à une date et une heure précise, selon un itinéraire précis que nous n'avons jamais emprunté, est l'exemple qui revient assez souvent. Et là nous commençons à décortiquer nos actions pour enfin découvrir que nous faisons assez fréquemment du suivi-évaluation, parfois sans se rendre compte, et concluant que finalement le suivi-évaluation est plutôt en notre faveur qu'en notre défaveur.

Cependant, d'autres défis guettent la fonction d'évaluation que je peux personnellement avancer, à titre d'illustration et sans être exhaustif, et qui sont plus logés dans l'environnement immédiat de la fonction d'évaluation, notamment:

  • L'auto-censure pratiquée par certains évaluateurs dans certains systèmes politiques afin de rester dans le "politiquement correct", poussant parfois les choses jusqu'à faire entendre aux responsables politiques et autres officiels ce qu'ils aiment entendre;
  • L'interférence de certains responsables politiques et autres officiels et la pression exercée sur les évaluateurs afin de changer certaines conclusions dans le rapport d'évaluation, voire même déguiser la réalité mise en évidence par l'exercice d'évaluation;
  • La rareté – voire l'absence – de statistiques officielles fiables et d'études sectorielles de qualité permettant de trianguler les "findings" d'une évaluation;
  • L'éloignement de certains évaluateurs de l'objectivité et la neutralité requises dans la fonction d'évaluation pour rester toujours dans le "politiquement correct" tout en pensant aux contrats futurs;
  • Le foisonnement de nombreux universitaires et académiciens ayant pour vocation d'élaborer depuis des décennies des études socio-économiques (tels des états des lieux, des diagnostics, etc.) et qui prétendent être des évaluateurs sans comprendre les fondements et les principes de la fonction d'évaluation et sans mise à jour préalable et nécessaire de leurs connaissances vis-à-vis de l'évaluation.

Voilà ce que je voulais partager avec les collègues à titre de contribution dans ce débat.

Cordialement

Mustapha Malki

Canada

***

Hello everyone,

Many thanks to our dear Hynda for opening a very interesting debate on the challenges and constraints that hinder the emancipation of evaluation in some countries. All that has been said is quite valid, nevertheless the lack of understanding of the evaluation function, as evoked by Hynda, very often perceived as a control and forcing many individuals to positions of resistance for different reasons, remains one of the challenges that must be addressed. From my modest experience in the various results-based management training workshops that I lead, in their monitoring and evaluation component, I always start by demystifying the monitoring and evaluation functions among participants by asking a simple question: do we do monitoring and evaluation in our daily life? And I engage in a frank and serene debate with the participants by taking them to evoke examples of the everyday life where the human being practices the monitoring-evaluation in a rather intuitive and fortuitous way. The example of a trip by car to a destination where we have never been to arrive at a specific date and time, according to a precise route that we have never taken, is the example that comes up quite often. And here we begin to dissect our actions to finally discover that we do quite often monitoring-evaluation, sometimes without realizing it, and concluding that eventually the monitoring-evaluation is rather in our favor than to our disadvantage.

However, there are other challenges for the evaluation function that I can personally advance, by way of illustration and without being exhaustive, and that are better housed in the immediate environment of the evaluation function, including:

  • Self-censorship practiced by some evaluators in some political systems in order to remain "politically correct", sometimes pushing things to the point where politicians and other officials hear what they like to hear;
  • The interference of some politicians and other officials and the pressure on the evaluators to change certain conclusions in the evaluation report, or even disguise the reality highlighted by the evaluation exercise;
  • The scarcity - or absence - of reliable official statistics and quality sectoral studies to triangulate the "findings" of an evaluation;
  • The remoteness of some evaluators from the objectivity and neutrality required in the evaluation function to remain always in the "politically correct" while thinking about future contracts;
  • The proliferation of many academics whose aim has been to develop socioeconomic studies for decades (such as baseline studies, diagnoses, etc.) and who claim to be evaluators without understanding the foundations and principles of the evaluation function and without prior updating and necessary knowledge of the evaluation ...

This is what I wanted to share with colleagues as a contribution to this debate ...

Cordially

Mustapha

[English translation below]

Chère Isha, chère communauté,

C'est exact. L'environnement est déterminant dans la réussite des évaluations, notamment celles liées à l'agriculture et au développement agricole. Le nombre important d'intervenants, limite quelque peu ce travail de communication et de sensibilisation à l'importance de l'évaluation. Il me paraît capital, aujourd'hui, de développer les compétences de communication et de "crédibilité" de l'évaluateur. Cette crédibilité n'a pas uniquement une portée technique, mais aussi une dimension formative très importante. Notre vis à vis, en comprenant la portée de l'évaluation, développera une confiance avec nous et plutôt qu'une relation de méfiance, un véritable travail de collaboration entre les parties sera développé, ce qui nous permet à coups sûrs d'atteindre de très bons résultats pour notre évaluation et contribuer par la même, au développement des différents dispositifs et politiques agricoles.

Merci à tous!!!

***

Dear Isha, dear community,

That's right. The environment is critical to the success of the evaluations, particularly those related to agriculture and agricultural development. The large number of stakeholders somewhat limits this work of communication and awareness of the importance of evaluation. It seems to me essential today to develop the communication skills and "credibility" of the evaluator. This credibility has not only a technical scope, but also a very important formative dimension. Our face-to-face, understanding the scope of the evaluation, will develop a trust with us and rather than a relationship of mistrust, a real work of collaboration between the parties will be developed, which allows us to achieve very good results for our evaluation and contribute to the development of different agricultural policies and systems.

Thank you all !!!

 

Dear Hynda,

Very true. Most think that evaluation is assessment of finding faults rather other way around. 

I think that the evaluation community does not consider enough the enabling environment for evaluation but focuses too much on conducting the evaluation based on TORs. 

I take a step further and before the assignment, I conduct a basic awareness raising on evaluation for the contracting organization and their stakeholder, which makes things easier.

[English translation below]

Bonsoir chère communauté,

Je vous remercie vivement pour votre réactivité.
Les points de vue sont très pertinents et émanent de véritables professionnels, c'est ce qui rend le débat intéressant.

Vos contributions se complètent et indiquent que les entraves à l'évaluation sont nombreuses et peuvent réellement impacter ses résultats.

Pour ma part, je voudrais rajouter en plus du déficit de la culture de l'évaluation, particulièrement dans les pays en voie de développement, la confusion entre l'évaluation et le contrôle, l'audit ou l'enquête. 

Pour moi, évaluatrice rattachée à une institution publique, c'est la contrainte principale que je rencontre.

Je m’explique: une évaluation participative repose sur des entretiens. Cependant, lorsque votre vis à vis pense que vous opérez un contrôle, il se ferme car il a peur de la sanction qui peut être le résultat d'un contrôle. 

Dans ce cas, un travail de sensibilisation est fait, dont le résultat n'est pas toujours satisfaisant.

La compétence et la distanciation de l'évaluateur, sont également capitales pour une évaluation objective, sans aucun parti pris et qui prenne en compte l'ensemble des facettes.

Merci encore à notre chère communauté. C'est un grand honneur d'être parmi vous. Cet espace d'échange nous permettra d'apprendre davantage sur la pratique de l'évaluation.

Je souhaite à tous une bonne continuité. 

***

Good evening dear community,

I thank you very much for your responsiveness.

The points of view are very relevant and come from real professionals, which makes the debate interesting.

Your contributions complement each other and indicate that the obstacles to evaluation are numerous and can really impact its results.

For my part, I would like to add in addition to the lack of evaluation culture, particularly in developing countries, the confusion between evaluation and inspection, auditing or investigation.

For me, evaluator attached to a public institution, this is the main constraint that I encounter.

I will explain: a participatory evaluation is based on interviews. However, when the person interviewed thinks you are doing an inspection, he stops because he is afraid of the penalty that may be the result of an inspection.

In this case, an awareness raining work is carried out, but the result is not always satisfactory.

The competence and distance of the evaluator are also crucial for an objective evaluation, without any bias and that takes into account all facets.

Thanks again to our dear community. It is a great honor to be among you.

This space of exchange will allow us to learn more about the practice of evaluation. I wish all of you good continuity.

Dear Hynda,

I am taking your question on challenges in evaluation from a broad/philosophical perspective.

The question begs us to scrutinize, the many reasons why we should evaluate interventions, plans, programmes, projects, strategies, policies, processes, and so forth.  The reasons give us an indication of the hoped-for benefits expected from evaluation. We should remember that there should be at the minimum, a set of principles shared by the evaluation team and the target groups/object for the evaluation should we desire a purposeful/impactful evaluation. As such, we can categorize the challenges as technical and non-technical. This response focuses only on these two defined categories.

In the first instance, technical challenges are so many, and this is rightly so, given the multiple realities that exist in this world).

Technical challenges at the very least may be addressed with less difficulty provided the appropriate authority figures are consulted and appraised of what’s at stake as well as the communities affected by the evaluation activities. The saying ‘it matters who you know and not what you know’ is closer to the truth that we would dare to imagine.  The other challenge that could arise is the degree to which ‘surprises’ are embraced and accommodated before, during, and after the evaluation exercise. Such technical challenges could be addressed through specific agricultural and professional training in evaluation approaches, methods, and processes, among other topics. These trainings would also incorporate elements or aspects of the Sustainable Development Goals (SDGs) and how the SDGs present big evaluation opportunities at the intersection of food security, agriculture and rural development.

In the second instance, non-technical challenges especially human-human interactions are a feature to deal with.  Such interactions partly dictate whether participants in the evaluation exercise would be willing to share information and knowledge to further the evaluation agenda.  An analysis of how societies are governed and function in any part of the world sometimes leaves us with wondering whether humans are ever going to get along anytime soon.  These short-comings in the human-human interactions call for the need for skills in creativity, people management, negotiation, and cognitive flexibility.

I would want to end this note on a sanguine tone.  It is the potential and ability to get along as humans that opens possibilities for the evaluation processes. The exciting thing is that the greater the possibilities opened the richer the human experiences, and consequently, the easier it becomes to realize the objectives of any evaluation exercise and derive meaning from the exercise. Evaluation should after all be a ‘fun and joyous’ exercise.

Raymond Erick Zvavanyange

Country Representative

YPARD - Young Professionals for Agricultural Development

[English translation below

Bonjour,

Une petite contribution en réponse à l'importante et grosse question soulevée par Hynda sur les contraintes relatives aux évaluations des projets de développement rural ou de sécurité alimentaire. Difficile d'épuiser le sujet dans ce cadre...

  • Une grosse contrainte c'est le manque ou la qualité des études sur la situation de base ou situation de référence dans les zones d'intervention des projets. Quand ces études ne sont pas faites dans les règles de l'art (bon diagnostic, analyse complète de la situation de départ avec la participation des hommes et des femmes) l'évaluation par la suite n'est pas aisée. C'est une étape primordiale quand on veut travailler dans une perspective de changement qualitatif d'une situation donnée car elle permet de bien orienter les actions et de faire un choix pertinent des actions et des acteurs et actrices... Dans le cadre du contrôle d'avancement d'un projet de développement de l'agriculture dans une région au Burkina il y a quelques années, le projet avait identifié l'aviculture, entre autres, comme activité génératrice de revenus au profit des femmes. Cette activité n'a jamais prospéré dans les localités choisies tout simplement parce que culturellement c'est une activité toujours dévolue aux hommes pour préserver l'harmonie dans les familles... Un bon diagnostic et une bonne analyse au départ aurait permis de choisir des activités socialement acceptées ou de réfléchir à des stratégies devant aider à apporter les changements nécessaires pour le bien- être de tous.
  • Une autre difficulté c'est la formulation des indicateurs quantitatifs et qualitatifs: les indicateurs quand ils existent ne permettent pas toujours de mesurer comme il se doit les progrès ou les changements induits par les interventions au niveau des communautés.
  • La prise en charge des questions de genre: cette thématique est souvent oubliée ou rajoutée dans les documents de projets comme une pièce jointe; ce qui ne facilite pas les évaluations. Nous en avons déjà parlé je crois lors des échanges sur ce thème dans ce groupe (https://dgroups.org/?8cvm4jpt) . Le genre est une question transversale et être pris en charge dès le début du processus de formulation d'un projet. C'est une question très importante dans nos pays en Afrique ou les femmes contribuent en raison de 70 à 80 % dans la production agricole et maraîchère
  • Autre contrainte: le temps imparti à l'évaluation ainsi que le budget prévu à cet effet : il arrive que lors de la formulation de projets l'on ne prévoit pas suffisamment de ressources pour le suivi et l'évaluation. Cela porte préjudice par la suite pour une bonne réalisation de cette activité pourtant très importante pour la bonne mise en œuvre et l'atteinte des objectifs visés par les interventions.

Ma modeste contribution sur ce sujet.

Georgette

***

Hello,

A small contribution in response to the big and big question raised by Hynda about the constraints on evaluations of rural development or food security projects. It is difficult to exhaust the subject in this context ...

  • A major constraint is the lack or quality of studies on the baseline situation in project intervention areas. When these studies are not done in the state of the art (good diagnosis, complete analysis of the starting situation with the participation of men and women), the evaluation afterwards is not easy. It is an essential step when one wants to work in a perspective of qualitative change of a given situation because it makes it possible to orient the actions and to make a relevant choice of the actions and the actors and actresses ...
  • As part of the monitoring of the progress of an agricultural development project in a region in Burkina some years ago, the project identified poultry farming, among other things, as an income generating activity for women. This activity never flourished in the localities chosen simply because culturally it is an activity always devolved to men to preserve the harmony in the families ... A good diagnosis and a good analysis at the beginning would have made it possible to choose socially accepted activities or to think about strategies to help make the changes necessary for the well-being of all.
  • Another difficulty is the formulation of quantitative and qualitative indicators: indicators, when they exist, do not always make it possible to measure progress or changes induced by interventions at the community level. Gender issues: this theme is often forgotten or added to project documents as an attachment; which does not facilitate evaluations. We have already talked about it earlier with this  group (https://dgroups.org/?8cvm4jpt ). Gender is a cross-cutting issue and should be taken care of from the beginning of the project formulation process. This is a very important issue in our countries in Africa where women contribute 70% to 80% of agricultural and vegetable production.
  • Another constraint is the time allocated to the evaluation and the budget allocated for this purpose: in the formulation of projects, there are not enough resources for monitoring and evaluation. This is detrimental to the successful completion of this activity, which is very important for the proper implementation and achievement of the objectives of the interventions.

My modest contribution on this subject.

 

Georgette

Dear Hynda,

You have raised a very important question, which affects the quality of evaluation work. Evaluations of development programmes in the broadly defined areas of rural development, agriculture and food security are inherently complex. The assessments of results in these areas are affected by a multiplicity of biophysical, economic, and social systems and factors. There are different types of constraints and challenges in evaluation work that depend mostly on the context of the programmes or policy work being evaluated.  For example, accurate and timely assessments of potential impact and development change may be affected by the remote location of project sites, social stratification of rural communities, time required to produce productivity gains, adoption capacities of local communities, and many other factors.

Evaluators often encounter issues with availability of baseline data, or information on the prevailing conditions of the development situation at the start of the projects or programmes addressing food security and agriculture development. This issue could be addressed by reconstructing baselines, for example, using ‘recall’ technique, i.e. requesting key beneficiaries or stakeholders to recollect information about these conditions in the past.

Security situation in the country may also have a huge impact on the access to data and methods we chose for evaluation. The choice of evaluators could also be highly limited, as not all may have necessary clearance to visit high-risk areas, or experience in working in similar situations. 

Accessibility of project sites may also be restricted or banned. To address these constraints, local consultants with access to restricted zones may provide support in data collection, and potential alternative evaluation methods could be also considered. In recent FAO’s evaluation of the large irrigation rehabilitation programme in Afghanistan, evaluation team faced a constraint of accessing some of the programme sites. The team opted for alternative method by using the open-source data from Google to assess the potential impact of the programme on the livelihoods in those specific sites. Google Earth maps were utilized to measure the expansion of the irrigated area and the vegetative cover along different sections of the rehabilitated canals. The methodology for measuring these areas was also using preliminary information from enumerators in the field who had access to the restricted zones, and were engaged in supporting collection of necessary data and information for the evaluation (e.g. the GPS coordinates of the irrigated areas in the vicinity of the irrigation canals). Then this information was analyzed based on historic data available from Google Earth on before- and after-project conditions and the changes based on vegetative cover at different periods during a year.

These are just a few highlights of the constraints and challenges that evaluators may encounter in their work and an example of possible ways to address those. The range of such constraints is quite broad, and we encourage all members of this community to share their experiences in addressing different types of constraints and limitations.

Kind regards,

Serdar Bayryyev,

Evaluation Officer

Food and Agriculture Organization (FAO)