Happy 2021 to you and all our colleagues on the platform!
Thanks for raising a critical and much-intriguing question worth looking into as evaluators. I am sure I cannot do justice to the important points you have raised but at least I can share my two cents. I hope colleagues will also keep coming in for a richer discussion.
It is true we assume we understand issues affecting local communities. We thus design interventions to meet their needs. I completely agree with you. There are important factors unknown at the design stage of development interventions. When little is empirically and theoretically known about a community, little may be done and achieved. Ideally, we need to known the unknowns to design proper interventions and serve better the target communities. But it is unfortunate that it does not work all the time like that, it is not linear, more so in the pandemic-stricken era. We base on what we know to do something. In that process, we learn something new (i.e. evidence) which is helpful to redefine our design and implementation. The complexity of our times, worsened by COVID-19, has pushed all evaluators to rethink their evaluation designs and methods. It would be an understatement to point out that we all know the implications of social (I personally prefer physical) distancing. Imagine an intervention designed through face-to-face results chain as its underlying assumption to achieve the desired change! Without rethinking its Theory of Change (ToC), the logic underlying such an intervention may not hold water. This scenario may apply and rightly prove we need time-evolving ToC. In my view and professional practice, my answer is in the affirmative. We need time-evolving, evidence-informed ToC. We use assumptions because we do not have evidence, right?
Keeping the ToC intact throughout the life of a project assumes most of its underlying assumptions and logical chain are known in advance and remain constant. This is rarely the case. I believe the change of the ToC does not harm but instead it maximizes what we learn to do better and benefit communities. Let’s consider this scenario: assume X outputs lead to Y outcomes. Later on one discovers that A and B factors are also, and more significantly, contributing to Y than their initial assumptions on X outputs. Not taking into account A and B factors would undermine the logic of the intervention; it undermines our ability to measure outcomes. I have not used outcome mapping in practice but the topic under discussion is a great reminder for its usefulness. Few development practitioners would believe flawed ‘change’ pathways. Instead, I guess, many would believe the story of the failure of the ToC (by the way I hate using words fail and failure). Development practitioners’ lack of appetite to accommodate other factors in the time-evolving ToC when evidence is available are possibly the cause of such failure. In the end, evaluation may come up with positive and/or negative results which are counterintuitive, or which one cannot be linked to any component of the intervention. It sounds strange, I guess, simply because there are pieces of evidence which emerged and were not incorporated in the logic of intervention.
With the above, a localized project would be a project with full local colours. With different sizes and forms, all coming in to play their rightful place. This does not mean being too ambitious (too many colours can blur the vision, just kidding but never mind, I wear glasses!). A project which discovers new evidence should incorporates it into the learning journey. Such project is more likely to achieve its desired outcomes. In view of time-evolving context, a project with a static ToC is more likely to become irrelevant over time.
In my view, a ToC needs to be dynamic or flexible in complex and time-evolving settings. Is there any development context which can be fully static for a while? I guess no. This reminds me of systems theories and complexity theories without which we would easily fall into the trap of linearity. In my view, there is no harm to start with assumptions but when evidence emerges, we should be able to incorporate new evidence in the implementation theory and program theory which, if combined, may constitute the full ToC for development interventions. No longer are projects looked at in silos (I guess we have seen coherence as a new OECD DAC evaluation criteria!). In my view, there is a need to understand the whole picture (that is, current + future knowns) to benefit the part (that is, current knowns only). But understanding the single part will less likely benefit the whole.
The challenges with evolving ToC are related to impact evaluations, mostly Randomized Control Trials. With the evolving ToC, the RCT components or study arms will get blurred and contamination uncontrollable. In statistical jargon, unexplained variance will be bigger than necessary. While there are labs for natural and physical sciences, I believe there are few, if any, reliable social and behavioural science labs. The benefit of knowing how to navigate complex ToC is that one may learn appropriate lessons and generate less questionable evidence on impact of development projects.
I guess I am one of those interested in understanding complexity and its ramifications in ToC and development evaluation. I am eagerly learning how Big Data can and will shed light on the usually complex development picture, breaking the linearity silos. As we increasingly need a mix of methods to understand and measure impact of or change resulting from development interventions, the same applies to the ToC. Linear, the ToC may eventually betray the context in which an intervention takes place. Multilinear or curvilinear and time-evolving, the ToC is more likely to represent the real but changing picture of the local communities.
I would like to end with a quotation:
“Twenty-first century policymakers in the UK face a daunting array of challenges: an ageing society, the promises and threats for employment and wealth creation from artificial intelligence, obesity and public health, climate change and the need to sustain our natural environment, and many more. What these kinds of policy [and development intervention] challenges have in common is complexity.” Source: Magenta Book 2020
All evolves in a complex context which needs to be acknowledged as such and accommodated into our development interventions.
Once again, thank you John and colleagues for bringing and discussing this important topic.
RE: Can we use an evidence-based, evolving Theory of Change to achieve "local learning” during project design?
Dear John,
Happy 2021 to you and all our colleagues on the platform!
Thanks for raising a critical and much-intriguing question worth looking into as evaluators. I am sure I cannot do justice to the important points you have raised but at least I can share my two cents. I hope colleagues will also keep coming in for a richer discussion.
It is true we assume we understand issues affecting local communities. We thus design interventions to meet their needs. I completely agree with you. There are important factors unknown at the design stage of development interventions. When little is empirically and theoretically known about a community, little may be done and achieved. Ideally, we need to known the unknowns to design proper interventions and serve better the target communities. But it is unfortunate that it does not work all the time like that, it is not linear, more so in the pandemic-stricken era. We base on what we know to do something. In that process, we learn something new (i.e. evidence) which is helpful to redefine our design and implementation. The complexity of our times, worsened by COVID-19, has pushed all evaluators to rethink their evaluation designs and methods. It would be an understatement to point out that we all know the implications of social (I personally prefer physical) distancing. Imagine an intervention designed through face-to-face results chain as its underlying assumption to achieve the desired change! Without rethinking its Theory of Change (ToC), the logic underlying such an intervention may not hold water. This scenario may apply and rightly prove we need time-evolving ToC. In my view and professional practice, my answer is in the affirmative. We need time-evolving, evidence-informed ToC. We use assumptions because we do not have evidence, right?
Keeping the ToC intact throughout the life of a project assumes most of its underlying assumptions and logical chain are known in advance and remain constant. This is rarely the case. I believe the change of the ToC does not harm but instead it maximizes what we learn to do better and benefit communities. Let’s consider this scenario: assume X outputs lead to Y outcomes. Later on one discovers that A and B factors are also, and more significantly, contributing to Y than their initial assumptions on X outputs. Not taking into account A and B factors would undermine the logic of the intervention; it undermines our ability to measure outcomes. I have not used outcome mapping in practice but the topic under discussion is a great reminder for its usefulness. Few development practitioners would believe flawed ‘change’ pathways. Instead, I guess, many would believe the story of the failure of the ToC (by the way I hate using words fail and failure). Development practitioners’ lack of appetite to accommodate other factors in the time-evolving ToC when evidence is available are possibly the cause of such failure. In the end, evaluation may come up with positive and/or negative results which are counterintuitive, or which one cannot be linked to any component of the intervention. It sounds strange, I guess, simply because there are pieces of evidence which emerged and were not incorporated in the logic of intervention.
I guess I am one of those interested in understanding complexity and its ramifications in ToC and development evaluation. I am eagerly learning how Big Data can and will shed light on the usually complex development picture, breaking the linearity silos. As we increasingly need a mix of methods to understand and measure impact of or change resulting from development interventions, the same applies to the ToC. Linear, the ToC may eventually betray the context in which an intervention takes place. Multilinear or curvilinear and time-evolving, the ToC is more likely to represent the real but changing picture of the local communities.
I would like to end with a quotation:
“Twenty-first century policymakers in the UK face a daunting array of challenges: an ageing society, the promises and threats for employment and wealth creation from artificial intelligence, obesity and public health, climate change and the need to sustain our natural environment, and many more. What these kinds of policy [and development intervention] challenges have in common is complexity.” Source: Magenta Book 2020
All evolves in a complex context which needs to be acknowledged as such and accommodated into our development interventions.
Once again, thank you John and colleagues for bringing and discussing this important topic.
Stay well and safe.
Jean Providence