RE: Neutrality-impartiality-independence. At which stage of the evaluation is each concept important?   | Eval Forward

Dear Umi,

Many thanks for sharing this excellent paper by Carol Weiss [earlier contribution here]. Old is but gold is. I just finished it and I want to carve some of its sentences in the concrete of my office walls. For instance:

Weiss brings to light an impressive series of assumptions baked in evaluation practice. Not all of them are always assumed true, but I think she is right that they tend to “go without saying”, i.e. be silently and even unconsciously accepted most of the times. Here is a list of such assumptions, based on her piece:

  1. The selection of which programs or policies get to be evaluated and which do not is done fairly – i.e. there’s no hidden agenda in the evaluation plan and no programs are protected from evaluation.
  2. The program to be evaluated had reasonable, desirable and achievable objectives, so that it can be assessed based on these objectives.
  3. The explicit program objectives can be trusted as true; there’s no hidden agenda, they reflect the real objectives of the intervention.
  4. The evaluated program is a coherent set of activities, reasonably stable over time and independent from other similar programs or policies, so that it makes sense to focus on it in an evaluation – it is a valid unit of analysis.
  5. Program stakeholders, recipients and evaluators all agree about what is good and desirable; any difference in values can be reconciled, so that the discussion is generally limited to means to get there.
  6. Program outcomes are important to program staff and to decision makers, who can be expected to heed the evidence collected by the evaluation in order to improve outcomes.
  7. The questions in the TORs are the important ones and reflect the preoccupation of program recipients, not just of program implementers.
  8. The evaluation team as composed can achieve a fair degree of objectivity (neutrality-impartiality-independence…) in its analysis.
  9. In most programs, what is needed to improve outcomes is incremental change (renewed efforts, more resources) as opposed to scrapping the program altogether or changing radically its approach.

The last assumption is based on the fact that most recommendations emanating from evaluations are about minor tweaks in program implementation. Weiss relates this to the type of messaging that can be accepted by evaluation commissioners.

In practice they are all problematic, at least occasionally, and Weiss does a great job at showing how some of them are often not borne by facts. For instance on assumption 1, she shows that new programs are typically subjected to more evaluative pressure than old, well-established ones.

So thank you again for a very insightful paper. Evaluation is indeed political in nature and evaluators can only benefit from understanding this.

Olivier