Thanks for starting this interesting discussion! The approach that you outline is very similar to how Outcome Mapping (OM) is used for planning and monitoring purposes. In the FAO Evaluation Office we have used Theory of Change (ToC) and OM for evaluation purposes only.
I believe you can find some examples on evolving ToC and the application of OM in the real world at the Better Evaluation website: https://www.betterevaluation.org/en/plan/approach/outcome_mapping
Thanks for your very interesting post, and for sharing your experience in grappling with this enormous challenge. I wish to share a couple of thoughts, from my role as evaluator and as a commissioner of evaluations.
In the short-term, I wonder the value that stakeholders may give to evaluations done at distance, and in the medium-term, the threat this pose to evaluation as a profession.
What is the added value of an evaluation that is done from distance? In a couple of ongoing evaluations with large field components we are facing some issues whose consequence we should not underestimate as they might reduce the credibility of the whole exercise (inability to observe first-hand changes, reliance on the evaluand to select who participates and who does not, limits to triangulation with beneficiaries and local partners, etc.) and put our teams in danger of being challenged in case they come up with negative or erroneous if not inaccurate findings.
Then, evaluation as a profession: if we are doing things from distance and without credible triangulation and bottom-up participation, what makes us different from those doing reviews or even performance audit? If we advocate for distance evaluations, and colleagues/partners realize that these can be done cheaply and in a non rigurous manner, we may have issues in the future i) selling evaluation as a distinctive and truly learning tool, and ii) getting adequate evaluation provisions/budgets.
Linked to this, the moral imperative for evaluators of not making harm. In view of all the unknowns that this pandemic is bringing, it is our duty not to put more people at risk, neither local evaluators nor beneficiaries. Trying to postpone evaluations if feasible, at least till it becomes clearer what we could safely do in the field and what we cannot, it will just be a fair and ethical thing to do.
Thank you Jackie and Richard, and all the previous commenters! It has been an interesting discussion, with so many different points of view and insights. We will soon be wrapping this up and summarize the learning in an Evalforward's blog. Keep an eye on it!
Best to all,
I wanted to raise one aspect that has apparently become a standard procedure when doing ToCs - many colleagues have said that they use this tool largely for programme design/implementation/evaluation. However, literature suggest that "Theories of Change may start with a program, but are best when starting with a goal, before deciding what programmatic approaches are needed." (see AEA presentation shared in a previous post)
Thus, the starting point for ToC should ideally be development/humanitarian goals in a particular theme (poverty/hunger reduction, climate change adaptation, rural development, women empowerment, saving lives) that have been identified by key stakeholders (usually Government, since they represent us all, or humanitarian actors in their absence) for a given geographical area (country, state/region, province, district), and not the programme (or project) specific goal.
As an example of this, in a recent evaluation of FAO's contributions to the development of the food and agricultural sector in Mexico (http://www.fao.org/evaluation/evaluation-digest/evaluations-detail/en/c/1202316/), we used the Mexican government's theories of change to map and then evaluate FAO's contributions. The Mexican government (an OECD member) indeed had by law to develop theories of change at different thematic and geographic levels (national/state) as part of their long-term (national development plans) and medium-term (strategies and programmes) planning process, often with CONEVAL advice (https://www.coneval.org.mx/Paginas/principal.aspx). This together with the fact that FAO had planed its programme of work along the lines of the Mexican's theories of change enabled the evaluation to assess FAO's contributions against these frameworks.
I was wondering if other colleagues have experienced developign ToC having locally-agreed/owned development/humanitarian goals as starting point (and not the specific funding agency goal in mind), and whether they think this is a feasible way forward in their own countries/agencies.
Thanks for your question. A few years ago the American Evaluation Association had a discussion on this topic (ToC vs Logic Models). Below is a link to the presentation made by Helene Clark (The Center of Theory of Change) during this session https://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA.pdf
Thanks for your very interesting contributions. If anyone could contribute with specific examples of ToC application, either in developing or developed countries, and highlight how this was useful for the programme under evaluation it would be great.
Thanks to those who contributed to the discussion on the use of synthesis and meta-analysis in development evaluation. The exchange supported my preparation for the What Works Global Summit 2019 (wwgs2019.org), where synthesis and meta-analyses are discussed as tools for designing, implementing and assessing programmes and policies. A synthesis is the integration of existing knowledge and findings relevant to a topic, and has as its main objective to increase the applicability of evaluation findings and develop new knowledge through the integration process. It is promoted as an approach that addresses the challenge of "information overload", delivering products that distil relevant evidence for decision-making.
Here are the main issues shared by participants:
Taking on this last point, EvalForward will organize a webinar soon for members interested in learning more on synthesis and meta-analysis. Stay tuned!
Thanks for sharing your experience with document analysis. It is indeed a great way to have early insights into the effectiveness of a programme or policy, although -depending on the amount of materials for review- it could also be a very demanding task.
You raised a very good point regarding the issue of accessibility to documents. With the advent of the internet it is assumed that information is becoming globally available, while digitalization is making reports and research more and more accessible through online means.
What is the perspective in the global south? Are government (evaluation) reports and research from academia easier to access? Are they available in a format and language that make them suitable for synthesis and meta-analysis?
A colleague has shared with me a link to a series of synthesis reports of impact assessments on various agricultural topics done by the CGIAR:
Hope you find it useful too!
Dear Olivier and Lal,
Thank you very much for your contributions.
Regarding a definition of synthesis (or synthetic approach), for the purpose of this discussion, we can define it as the process of reviewing, assessing and synthesising existing literature or data to produce a series of outputs (products and services).
A first step in this process is to review the quality of the literature or data that will be aggregated to ensure that is comparable and meet the research protocol requirements. Afterwards, the synthesis is conducted often by academic disciplinary experts, but can also be done by inter- or transdisciplinary working groups or evaluators drawing on knowledge from across academia and beyond, the latter to ensure a comprehensive analysis and avoid the pitfals raised by Lal.
There are many guidelines out there on how to conduct synthesis reviews, especially in the areas of health and education. Perhaps the ones better know and applied in the field of agriculture and rural development are those developed by 3IE and Campbell Collaboration, accessible at this link: https://www.3ieimpact.org/our-expertise/synthesis
Thanks to all of you for contributing to this discussion.
Here is a summary of the suggestions and ideas shared:
I look forward to further exchanges with the Community!