I recently participated in the Outcome Mapping Leaning Community (OMLS) Winter School, which was co-hosted by the Belgium Development Agency (ENABEL).
I had two related questions before attending this workshop and learning exchange on users of outcome mapping and outcome harvesting:
1. What are the similarities and differences between the outcome-based approaches and other conventional approaches such as result-based and log-frame?
2. Can we use outcome-based approaches in evaluating complex development programmes?
I found that the outcome-based approaches are not very different from the other approaches at the concept level. However, there are some differences at the methodology level. I think this is because outcome-based approaches are adaptive approaches while conventional theory-based approaches are based-on deductive reasoning. More specifically:
The concept is familiar
The following three principles are particularly relevant and often used in evaluating development programmes:
1. Actor-focused approach.
2. Focusing on behavior changes.
3. Systems thinking.
Outcome-approaches categorize stakeholders in a way that is similar to the conventional systems approach – i.e., those in the sphere of control, the sphere of influence and the sphere of concern. It’s just that outcome-based approaches use specific terms. The stakeholders are called social actors. The social actors who interact directly with the programme and anticipate opportunities for change/influence are called boundary partners. The outcome-approach observes the boundary partners’ interactions within and outside of the spheres of control and influence. The interactions are called systems.
Some differences in application
Besides language, there are some key differences. The outcome-based approach focuses on the contribution of the whole system to all changes (both anticipated and unanticipated) that influence the vision (highest level objective). On the other hand, conventional approaches tend to focus on the attribution of the project beneficiaries to the anticipated changes within the context pre-scoped in the Theory-of-Change.
Outcome-based approaches focus on social actors’ observable behavior and apparent attitude changes. The outcome is defined as an observable and significant change performed by a social actor, which has been influenced by an intervention. Conventional approaches tend to require more tangible and quantifiable results, which are not necessarily the behavior/attitude changes, to meet donor requirements.
The outcome-based approach is adaptive in harvesting outcomes. It acknowledges that the interactions among social actors evolve over time, and the change in relationships is considered as an outcome. The conventional approach anticipates changes – i.e. Theory-of-Change pre-identifies anticipated, cascading changes.
Can we use outcome-based approaches for complex evaluations?
The answer depends on the case. Having said so, below are my generalized, initial thoughts, which may be a good starting point for further thinking.
The key principles of the outcome-based approaches are highly relevant in designing the evaluation. There are ways to integrate these principles into the evaluation design. For instance, using actor-centered mapping in the Theory-of-Change would be useful for evaluating the impact and sustainability of the interventions because it allows the assessment of the behavior changes and ownership of the changes.
Outcome mapping of complex programmes will be unmanageably complicated. So you might think it would make more sense to use conventional deductive reasoning-based approaches from the efficiency point of view. However, the retrospective approach of outcome-harvesting would be useful in understanding what has happened and how different outputs and outcomes were linked. It helps to narrow down the scope of investigation on complex subject matters to situations where outcomes were observed. For instance, outcome-harvesting at the beginning of the evaluation exercise would guide the framework to prioritize what to investigate further. So, it is worth trying to mix the theory-based, inductive-reasoning approach and practical, outcome harvesting to focus on interactions within the complex systems.
Outcome harvesting uses progress makers in monitoring and allows for their adjustment during the life span of the project. This flexibility makes it difficult to undertake comparable assessments. Introduction of a comparable baseline and indicators would address this issue. On the other hand, the flexibility would be an advantage in conducting stand-alone case studies.
I think outcome harvesting can be a useful tool for project M&E. Outcome harvesting draws contribution and significance from the description of change. We can monitor the progress comprehensively by acknowledging the dynamic and unanticipated nature of the system changes. I think involving the boundary partners in writing outcome journals to document outcomes would be useful. ‘Boundary partners’ are agents of change in the systems. So, their greater awareness of the system’s contribution to the outcomes would make the changing process more intentional.
In summary…
Outcome-based approaches can be scoped or incorporated into other evaluation approaches. It is the evaluator’s responsibility to choose the most suitable methodology to each evaluation. Appropriate scoping requires understanding the concepts of different approaches and pros and cons of their methodologies. Outcome-based approaches are practical and evolving approaches.
If you are interested, you can find more systematic explanations and check terminologies of outcome mapping and outcome harvesting at the following sites:
https://www.betterevaluation.org/en/plan/approach/outcome_harvesting
https://www.betterevaluation.org/en/plan/approach/outcome_mapping
You can also join an informal and open membership network on OMLC (outcomemapping.ca/join) to share information and experiences.