Thank you so much for the contributions that you made on the topic 'how to measure the impact of M&E work' that I posted for discussion in the forum.
I have made a compilation of the key issues that I received out of the discussions, attached to this email.
I will appreciate further guidance on how to take the subject forward.
The following are some of the key highlights:
I. Nothing is impossible and impact evaluation of evaluations or a systematic review of M&E can be done.
II. The impact of M&E and indicators for this “Impact" is much related to utility/ use of the outputs of M&E.
III. Measuring the impact of M&E outcomes can be in terms of the outcome influencing decision-making, influencing project or programme or policy change/redesign, influencing more resource allocation, informing improvements in production, productivity, food security and service delivery in general, clarifying problems or challenges or issues.
IV. It may not always be possible to measure the contribution at the impact level, but going down to outcome and output level would be easier. To give example of indicators: the number of downloads of an evaluation report or the number of reaches/ views, % of M&E reports shared on websites of related organisations.
V. Perform some sort of counterfactual analysis
VI. (probably using systematic reviews/mixed methods/triangulation) to determine whether having M&E systems in place transformed into some sort of policy change/ improvement or not, no matter what sector/ indicators you are taking into the consideration. The scale and breadth of such analysis would be dependent upon various factors (including but not limited to resources, expertise, priority areas, and number of interventions to account for).
VII. Monitoring data collected at a project level can easily be aggregated at the sectoral level, getting the sectoral plan to feed back into the national strategic plan.
VIII. The extent to which M&E actions, recommendations and decisions are implemented can be used to approximate the impact, so that if zero of those are implemented then we approximate the impact of M&E to be zero, if 50% are implemented then we approximate the impact to 50 % etc.
IX. The extent to which the purpose of M&E is realized can also be used, for example, better understanding of the gaps, challenges, emerging needs, changes in situations, etc. The extent of the achievement can be rated and levels of achievement determined that can be approximated to the impacts created in that context.
X. Adopting outcome harvesting (mixed with another approach like contribution tracing) one would be able to trace the impact of M&E system. The choice of outcome harvesting is because it does not measure progress towards a predetermined objective, but rather collects evidence of what has changed and then work backward to trace a plausible relationship between the change and an intervention contribution to this change. If government proposes an agricultural bank, this can be traced back to recommendations from some M&E effort or if agricultural insurance policy, this policy change can be traced to (i) The project Intervention; (ii) Project M&E efforts (iii) Environment and what it means for agriculture etc.
RE: How to Measure the Impact of Monitoring and Evaluation Work
Dear Colleagues,
Thank you so much for the contributions that you made on the topic 'how to measure the impact of M&E work' that I posted for discussion in the forum.
I have made a compilation of the key issues that I received out of the discussions, attached to this email.
I will appreciate further guidance on how to take the subject forward.
The following are some of the key highlights:
I. Nothing is impossible and impact evaluation of evaluations or a systematic review of M&E can be done.
II. The impact of M&E and indicators for this “Impact" is much related to utility/ use of the outputs of M&E.
III. Measuring the impact of M&E outcomes can be in terms of the outcome influencing decision-making, influencing project or programme or policy change/redesign, influencing more resource allocation, informing improvements in production, productivity, food security and service delivery in general, clarifying problems or challenges or issues.
IV. It may not always be possible to measure the contribution at the impact level, but going down to outcome and output level would be easier. To give example of indicators: the number of downloads of an evaluation report or the number of reaches/ views, % of M&E reports shared on websites of related organisations.
V. Perform some sort of counterfactual analysis
VI. (probably using systematic reviews/mixed methods/triangulation) to determine whether having M&E systems in place transformed into some sort of policy change/ improvement or not, no matter what sector/ indicators you are taking into the consideration. The scale and breadth of such analysis would be dependent upon various factors (including but not limited to resources, expertise, priority areas, and number of interventions to account for).
VII. Monitoring data collected at a project level can easily be aggregated at the sectoral level, getting the sectoral plan to feed back into the national strategic plan.
VIII. The extent to which M&E actions, recommendations and decisions are implemented can be used to approximate the impact, so that if zero of those are implemented then we approximate the impact of M&E to be zero, if 50% are implemented then we approximate the impact to 50 % etc.
IX. The extent to which the purpose of M&E is realized can also be used, for example, better understanding of the gaps, challenges, emerging needs, changes in situations, etc. The extent of the achievement can be rated and levels of achievement determined that can be approximated to the impacts created in that context.
X. Adopting outcome harvesting (mixed with another approach like contribution tracing) one would be able to trace the impact of M&E system. The choice of outcome harvesting is because it does not measure progress towards a predetermined objective, but rather collects evidence of what has changed and then work backward to trace a plausible relationship between the change and an intervention contribution to this change. If government proposes an agricultural bank, this can be traced back to recommendations from some M&E effort or if agricultural insurance policy, this policy change can be traced to (i) The project Intervention; (ii) Project M&E efforts (iii) Environment and what it means for agriculture etc.
Grateful
Abubakar Muhammad Moki
Uganda