I recently raised a topic for discussion, asking EvalForward members how they practised and experienced evaluator neutrality, impartiality and independence at different stages of the evaluation process.
Drawing from the wide array of experiences and views shared, here some of the key aspects I retained along with some personal reflections.
What exactly do we mean by independence?
The discussion raised clear questions as to definition. What does independence mean and what does it entail? A 2016 UNDP report [1] defines evaluation independence as a “twofold concept and refers to formal independence on the one hand and substantial independence on the other. Formal independence means structural freedom from control over the conduct and substantial independence can be described as the objective scientific assessment of a subject free from undue influence meant to distort or bias the conduct or findings of an evaluation.”
Evaluation users, managers and beneficiaries, however, may have different interpretations of independence, neutrality and impartiality (the latter two generally being roughly synonymous). Hence, as evaluators, we should seek to come up with a shared understanding of these key concepts for our practice and profession.
What are the factors that may influence an evaluator?
Most participants in the discussion seemed to agree on the importance of these principles in evaluation, but cited some key factors that might influence the independence, neutrality and impartiality of evaluators. Some participants also questioned the extent to which we should hold onto these concepts in concrete terms.
First, while evaluating can require the evaluator to distance themselves from their culture, logical flow and value system, their individual choices may still be influenced by their culture and background, making complete neutrality a challenge. Evaluators need to take time to understand and integrate the logic of different contexts and stakeholders in order to conduct an objective evaluation and reduce potential cultural bias.
Evaluation commissioners and terms of reference can also influence the evaluator and, consequently, their independence. Some participants cited the financial dependence of the evaluator, with an emphasis on cases in which the evaluation commissioner (person or organization) has also been responsible for the implementation of the intervention, so is particularly keen to see a positive evaluation outcome. Under such conditions, there can be a tendency to put pressure on the consultant, leading to reduced neutrality and impartiality in carrying out the mission. Similarly, other participants noted that even when funds were available, evaluators might lack neutrality to ensure future assignments.
There are also occasions when evaluators are recruited based on personal ties or established working relationships with commissioners. This can affect the quality of evaluation and reporting. Evaluators should remain reasonably neutral, regardless of their experience or connections to commissioners, who should also strive to select neutral candidates.
The availability of data, information and knowledge is crucial throughout an evaluation, as is the ability of evaluators to synthesize and put them into perspective. Often, however, data may be difficult for evaluators to access. They may face certain limitations (a lack of time, for instance) on contacting stakeholders, beneficiaries and other relevant groups, which may influence the neutrality of the evaluation process. Inclusiveness and stakeholder participation in the evaluative process, combined with conditions that encourage the “open-mindedness” of evaluators to diverse views, are key to achieving objectivity and impartiality and to creating a valuable learning outcome.
There may also be a loss of quality and depth of analysis where there is an excess of objectivity, as the evaluator may not be fully informed about the intervention and the sociodemographic conditions of the area, resulting in a superficial analysis.
Strive for neutrality or take an openly biased approach?
Some participants in the discussion opposed equating evaluation to research, saying that evaluation should not aim for the degree of independence and neutrality of research. However, this debate is also pretty relevant to research, especially in the social sciences, where the positioning of the researcher with regard to the object of investigation remains a continued topic of debate. Even scientific research cannot be completely independent. Moreover, even laboratory experiments carry some margin of error.
On the other hand, some participants took an opposite stand point: rather than pretending that we can achieve true impartiality, it makes more sense as evaluators to be aware of (as much as possible) and open to what our choices are and be honest about the dynamics at play and choices made. One participant suggested that we be openly biased while striving to give greater voice to those who are usually less represented.
Because it is highly challenging to get all stakeholders to participate, it is also difficult to get an unbiased result. One participant shared a fairly old, but still relevant paper by Weiss [2], which analyses programme development, implementation and evaluation. The paper observes that programmes are decided in the context of a given political environment. Consequently, as evaluation is meant to guide decision-making, it is also subject to pressure from political participants in that programme. In other words, evaluation is never free from the political context in which it occurs, and evaluators should be mindful of this. They should, therefore, strive to exercise to a reasonable degree their independence, impartiality and neutrality to bring the added value they are expected to have in assessing a programme.
In conclusion: aim for the greatest possible level of objectivity
As long as the terms of references are clear, reliable data are available and the independence of the evaluator is ensured through sufficient funding and administrative independence, the evaluator must work on himself in terms of his beliefs, culture and biases. Methodological approaches could help reverse any undue bias.
As evaluators, we should be aware of our own biases and prejudices and try to manage these limitations, striving for neutrality, independence and impartiality, even if these ideals may not be fully achievable in practice.
As this is not an exhaustive summary of the discussion, please feel free to read all of the contributions on the discussion section of the website and to share additional thoughts or comments using the chat box below, or via info@evalforward.org.
With thanks to participants: Abado Bienvenue EKPO, Abubakar Muhammad Moki, Bamlaku Alamirew Alemu, Bassirou Diagne, Egwuatu U. ONYEJELEM, Emile N. HOUNGBO, Isha Miranda, Jean de Dieu BIZIMANA, Khalid El Harizi, Lal Manavado, Lasha Khonelidze, Mohammed Lardi, Olivier Cossée, Pamela Dianne White, Ram Chandra Khanal, Richard Tinsley, Roxana Marcela Arce, Sébastien Galéa, Silva Ferretti, Svetlana Negroustoueva, Thierno Diouf, Umi Hanik, Una Carmel Murray