Excellent question, which sparked reflections based on insights from the recently completed independent reviews of 12 CGIAR research programmes (CRPs).CRPs are global research-for-development programmes covering themes from single-crop programmes like RICE, to integrated cross-cutting programmes like Climate Change, Agriculture & Food Security (CCAFS).
How does your own work relate to the topic question?
Following earlier announcement, the evaluation function of the CGIAR Advisory Services only recently completed independent & rapid reviews which covered the quality of science as well as the effectiveness of the outcomes achieved, zooming in on progress along ToC and usefulness of the ToC.
What is a real-world example of a localized project design?
The evidence need not be de-coupled from the risks and assumptions as together they give a big picture of the ground realities, irrespective of the size and type of intervention/program/initiative. To put things in perspective, ToC for CRPs are layered. First, all CRPs have a ToC which contribute to the CGIAR overall Strategy and Results Framework. Cascaded down, the CRPs in turn, have different Flagship programmes (FP) - each FP contributes through specific impact pathways nested within the overall ToC. The FP ToC were co-designed and developed in collaboration with project teams, reflecting bottom-up approach- the process much appreciated overall in reviews.
CRP reviews found that, although most CRPs incorporated evidence fed in from previous independent evaluations and impact assessments from conceptualization and during implementation, the ToC had varying levels of use and evolution. Overall, for some of the CRPs, the reviews found value in the process- in cultivating ToC-thinking even among scientist but limited evidence in its use as a measurement tool, linking it to the results framework.
What would an evidence-based, evolving Theory of Change look like for that project?
Given the global nature of CGIAR and majority of CRPs, grounding in the context has been key. ToC are very context/programme-specific. Framed within the context, one of the CRPs (Forests, Trees and Agroforestry-FTA) had a considerably evolved ToC-use. It had annual targets adapted and indicators suited to the field realities. Some CRPs did not make any changes to their ToC (WHEAT). One of the conclusions for its Review (WHEAT) was that its’ ToC was good for “(1) priority setting, (2) assessing contribution of scientific outputs, (3) seeking and justifying funding, (4) mapping trajectory to impact and (5) reporting” but unsuitable for assessing the effectiveness of CRP or flagship. Why? The review report says “because that was not its purpose.” Intentionality matters when developing ToCs, in order not to limit its usage in evidence-generation, learning, reporting and reprogramming, ToC development and iteration teams have to be intentional about co-designing it as an iterative evidence tool, tying in the indicators, linking the drivers and risks, testing the assumptions and causal pathways.
What opportunities and obstacles do you see?
Adaptive management was not found to be necessarily tidy, having revisions of ToC based on evidence, assumptions and risks could make the process as well as aggregation and reporting of results cumbersome. Yet this can be managed if reporting is consistently structured based on the indicators and targets linked to the (updated/revised) ToC. The suite of metrics have to reflect design, implementation and scale-up of scientific innovations on the ground and be flexible, useful and coherent to allow progress to be tracked in a way that gives a clear picture of progress and the context has to promote a learning-by-doing approach. When the underpinning ToC, the evolution of the system and CRP metrics, and the evidence, with associated risks and assumptions, are revisited, captured and tracked coherently, then process tracing or contribution analysis of particular causal pathways is made easier. On the other hand, when ToC are not context-specific (time and place), which was the case in one of the CRPs (Grain, Legumes and Dryland Cereals-GLDC), accurate reflection on progress is challenging, as some TOC impact pathways may become obsolete.
Reading other responses has been interesting, obviously, your question sparked an intriguing discussion. Should you and colleagues be interested in more information on earlier reflections around ToC in CGIAR and actual CRP Reviews, you can check out the hyperlinks.
RE: Can we use an evidence-based, evolving Theory of Change to achieve "local learning” during project design?
Dear John and colleagues,
Excellent question, which sparked reflections based on insights from the recently completed independent reviews of 12 CGIAR research programmes (CRPs).CRPs are global research-for-development programmes covering themes from single-crop programmes like RICE, to integrated cross-cutting programmes like Climate Change, Agriculture & Food Security (CCAFS).
How does your own work relate to the topic question?
Following earlier announcement, the evaluation function of the CGIAR Advisory Services only recently completed independent & rapid reviews which covered the quality of science as well as the effectiveness of the outcomes achieved, zooming in on progress along ToC and usefulness of the ToC.
What is a real-world example of a localized project design?
The evidence need not be de-coupled from the risks and assumptions as together they give a big picture of the ground realities, irrespective of the size and type of intervention/program/initiative. To put things in perspective, ToC for CRPs are layered. First, all CRPs have a ToC which contribute to the CGIAR overall Strategy and Results Framework. Cascaded down, the CRPs in turn, have different Flagship programmes (FP) - each FP contributes through specific impact pathways nested within the overall ToC. The FP ToC were co-designed and developed in collaboration with project teams, reflecting bottom-up approach- the process much appreciated overall in reviews.
CRP reviews found that, although most CRPs incorporated evidence fed in from previous independent evaluations and impact assessments from conceptualization and during implementation, the ToC had varying levels of use and evolution. Overall, for some of the CRPs, the reviews found value in the process- in cultivating ToC-thinking even among scientist but limited evidence in its use as a measurement tool, linking it to the results framework.
What would an evidence-based, evolving Theory of Change look like for that project?
Given the global nature of CGIAR and majority of CRPs, grounding in the context has been key. ToC are very context/programme-specific. Framed within the context, one of the CRPs (Forests, Trees and Agroforestry-FTA) had a considerably evolved ToC-use. It had annual targets adapted and indicators suited to the field realities. Some CRPs did not make any changes to their ToC (WHEAT). One of the conclusions for its Review (WHEAT) was that its’ ToC was good for “(1) priority setting, (2) assessing contribution of scientific outputs, (3) seeking and justifying funding, (4) mapping trajectory to impact and (5) reporting” but unsuitable for assessing the effectiveness of CRP or flagship. Why? The review report says “because that was not its purpose.” Intentionality matters when developing ToCs, in order not to limit its usage in evidence-generation, learning, reporting and reprogramming, ToC development and iteration teams have to be intentional about co-designing it as an iterative evidence tool, tying in the indicators, linking the drivers and risks, testing the assumptions and causal pathways.
What opportunities and obstacles do you see?
Adaptive management was not found to be necessarily tidy, having revisions of ToC based on evidence, assumptions and risks could make the process as well as aggregation and reporting of results cumbersome. Yet this can be managed if reporting is consistently structured based on the indicators and targets linked to the (updated/revised) ToC. The suite of metrics have to reflect design, implementation and scale-up of scientific innovations on the ground and be flexible, useful and coherent to allow progress to be tracked in a way that gives a clear picture of progress and the context has to promote a learning-by-doing approach. When the underpinning ToC, the evolution of the system and CRP metrics, and the evidence, with associated risks and assumptions, are revisited, captured and tracked coherently, then process tracing or contribution analysis of particular causal pathways is made easier. On the other hand, when ToC are not context-specific (time and place), which was the case in one of the CRPs (Grain, Legumes and Dryland Cereals-GLDC), accurate reflection on progress is challenging, as some TOC impact pathways may become obsolete.
Reading other responses has been interesting, obviously, your question sparked an intriguing discussion. Should you and colleagues be interested in more information on earlier reflections around ToC in CGIAR and actual CRP Reviews, you can check out the hyperlinks.
Best regards,
Erdoo Karen Jay-Yina