How to Measure the Impact of Monitoring and Evaluation Work

Dear Colleagues,

Thank you so much for the contributions that you made on the topic 'how to measure the impact of M&E work' that I posted for discussion in the forum.

I have made a compilation of the key issues that I received out of the discussions, attached to this email.

I will appreciate further guidance on how to take the subject forward.

The following are some of the key highlights:

I.             Nothing is impossible and impact evaluation of evaluations or a systematic review of M&E can be done.

II.            The impact of M&E and indicators for this “Impact" is much related to utility/ use of the outputs of M&E.

III.           Measuring the impact of M&E outcomes can be in terms of the outcome influencing decision-making, influencing project or programme or policy change/redesign, influencing more resource allocation, informing improvements in production, productivity, food security and service delivery in general, clarifying problems or challenges or issues.

IV.          It may not always be possible to measure the contribution at the impact level, but going down to outcome and output level would be easier. To give example of indicators: the number of downloads of an evaluation report or the number of reaches/ views, % of M&E reports shared on websites of related organisations.

V.            Perform some sort of counterfactual analysis

VI.          (probably using systematic reviews/mixed methods/triangulation) to determine whether having M&E systems in place transformed into some sort of policy change/ improvement or not, no matter what sector/ indicators you are taking into the consideration. The scale and breadth of such analysis would be dependent upon various factors (including but not limited to resources, expertise, priority areas, and number of interventions to account for).

VII.         Monitoring data collected at a project level can easily be aggregated at the sectoral level, getting the sectoral plan to feed back into the national strategic plan.

VIII.        The extent to which M&E actions, recommendations and decisions are implemented can be used to approximate the impact, so that if zero of those are implemented then we approximate the impact of M&E to be zero, if 50% are implemented then we approximate the impact to 50 %  etc.

IX.           The extent to which the purpose of M&E is realized can also be used, for example, better understanding of the gaps, challenges, emerging needs, changes in situations, etc.  The extent of the achievement can be rated and levels of achievement determined that can be approximated to the impacts created in that context.

X.            Adopting outcome harvesting (mixed with another approach like contribution tracing) one would be able to trace the impact of M&E system. The choice of outcome harvesting is because it does not measure progress towards a predetermined objective, but rather collects evidence of what has changed and then work backward to trace a plausible relationship between the change and an intervention contribution to this change. If government proposes an agricultural bank, this can be traced back to recommendations from some M&E effort or if agricultural insurance policy, this policy change can be traced to (i) The project Intervention;  (ii) Project M&E efforts (iii) Environment and what it means for agriculture etc.

Grateful

Abubakar Muhammad Moki

Uganda

Dear Samuel

Thanks for sharing this useful information from the ICED newsletter.

There seems to be a lot of evidence on how to establish a robust M&E system for better policy and practice.

There is need for evidence in a similar manner, on how best to determine whether the robust M&E system that has been established actually betters policy and practice.

That is how the discussion on how to measure the impact of M&E system outcomes on policy and practice comes in. The idea is that we should be able to establish the changes that M&E outcomes have contributed to. This contribution to changes of M&E could then be approximated to be its impact.

There is also need for clarity on what a robust M&E system is. The robustness of M&E system could be out of what the M&E outcomes do such as degree at which it can influence or inform better policy and practice.

Grateful

Abubakar Muhammad Moki

Uganda

Dear all,

This ICED newsletter below could be provide some relevant information to the on-going discussion submitted by Abubakar.

Best regards

Samuel

Dear Isha Miranda

Thanks for the good contribution.

However, my interest is to know how to measure the impact or effectiveness of M&E outcomes. In other words, how to measure the influence that M&E evidence or outcomes have generated. How do we do that?

I believe, we should also be able to assess whether M&E outcomes are meeting their intended purposes such as to what extend do M&E outcomes influence: (i) decision and policy making, (ii) redirecting policy or project or program implementation, (iii) influencing termination of policy or project or program implementation that is not meeting its intended outcomes, (iv) influencing planning, budgeting etc.

So that M&E outcomes can be seen to have impact or not.

Grateful

Abubakar Muhammad Moki

Uganda

Dear Abubakar Muhammad Moki,

I would like to share the Sri Lankan experience. In Sri Lanka the Evaluation Policy reached the legislation after 16 years and was launched in Sep 19th 2018.  The Policy framework is now under construction.  I have been through this process all these years and now are a member of the National Policy Framework Technical committee.

First and foremost, it is important to create a culture of evaluation and key activities can fall under three broad areas:  

1. Enabling environment,
2. Institutional Capacity,
3. Individual capacity in the country at all levels from national to sub national government.

If the country does not have a Policy on Evaluation it is necessary to develop it in order to create the legally bonded “must” culture. However, policy alone is not enough to create a culture, which needs the following activities:

1. Enabling environment, creating a demand before the supply

• Do not wait for the whole government but can do it organization by organization.
• If the government has a central agency who already does monitoring you could add or transform the mandate to Monitoring and Evaluation.
• Setting up national policy combined with sub-national.
• All the international donor funded intervention (INGO’s, Finance organizations etc): at the time of the contracting government should/could ask for a joint evaluation activity (i.e. evaluation team will comprise with Government and funding organization members) it is very much a lesson learning exercise that will help in creating ownership too.
• Set up a “must” activities and embed the evaluation into that with inclusiveness in every government own program/intervention/project either nationally or sub nationally.
• Lobbying the legislation either upper house/lower house/senate on policy on evaluation through members who are champion of good governance. Even few members can have a large impact.
• Create more awareness by utilizing your VOPEs and other Professional Associations.

2. Institutional Capacity: policy and frameworks take a long time to materialize. Therefore creating awareness on “what is evaluation” and “how” will benefit to their work and knowledge among the government institution and can make a tremendous impact. Take this message and promote the upcoming policy and framework even it is not yet ready.

Target institutional capacity for:

• Government and Sub –National institutes

• Academia – Setting up workshops and other possible certificate or diploma programmes by practitioners of the field of evaluation.
• Institutes like Project Management, Government Administration training institutes, universities etc.
• Private sector institutes under Project Management etc.

3. Individual capacity: this is important to create the culture.

However, it is better that above centres enroll all the members of the working world to adopt as part of their job as learning and implementing evaluation “Must”

Challenges:

• Resources and Practitioners in the country also need international support in order to upgrade the skills and methods. i.e. evaluation is not a uniform effort it’s change all the time, methods, approaches norms and standards etc.
• Make it important and vital also how to create demanding culture,
• Evaluation is very costly and how do you show the authorities that it is Value for Money.
• Raise awareness to all via language they understand. i.e. legislation, individuals and institutional level too.

If you need any clarification or any other assistance, please ask me I am happy to help.

Dear all,

Thanks for your contributions and please see my responses below:

Dear Dr Émile Houngbo

My issue is how do I know that M&E done has added value to an organisation.

How do we measure the output and outcomes of M&E that has been carried out or done?  And not what M&E measures. M&E should not be an activity whose performance cannot be assessed.  What can be  assessed and who has the yardsticks?

Dear Zahid Shabbir,

Thanks so much for all your useful contributions. However, I have a view that the impact of M&E can be in terms of the following:

1. decisions it has influenced

2. problems or challenges or issues it has clarified

3. interventions it has redirected

4. etc.

We can recast the above and add on the list

So that an M&E system that does the above well, we say it has high impact and one that does none of the above we say has no impact and hence no value for money or value addition or contribution for its work.

Dear Mustapha Malki,

My interest is how to assess the performance of the M&E system in terms of the changes it has influenced. If for example, the purpose of M&E is for evidence informed decision making.

If an M&E output influences decision making, then that should be its impact.  If it does not influence decision-making, then it has zero impact.

Dear Dr. Tarek Sheta,

To measure the impact of M&E work we should look at:

(i)  The extent to which M&E actions, recommendations and decisions are implemented. So that if zero of those are implemented then we approximate the impact of M&E to be zero. If 50% are implemented then we approximate the impact to 50 %  etc.

(ii) The extent to which the purpose of M&E is realized for example, better understanding of the gaps, challenges, emerging needs, changes in situations, etc.  The extent of the achievement can be rated and levels of achievement determined that can be approximated to the impacts created in that context.

Let’s then refine those two above and add on the list

Dear Reagan Ronald Ojok,  

My idea is how to measure the impact of the M&E outcome in terms of the outcome influencing decision-making, influencing project change, influence more resource allocation etc.

This is because many M&E are being done at various levels and no change seems to be seen out of the M&E outcomes because of weak influence of M&E outcomes. 

So how do we measure the M&E outcomes so that we know which ones have made some influence and which ones have not made some influence as an example?

Dear Abubakar,

I think Dr. Emile and Zahid have nicely driven the point home and I would like to agree with the both of them. Other factors constant, the M&E effort coupled with good project managers and a conducive environment is capable of impacting significantly on the project, target community and the policy framework in a particular country.

The contribution of a well-designed M&E system that has been tracking progress and proposing possible realignment (where need be) is reflected in policy change, practice change and behavioral change respectively (depending on the level of change).

In your problem statement (If I may be allowed to call it that), I hear a bit of complexity and uncertainty on the different level of impact (with your example of agriculture) and tracking how they occur (time & place).

I would think that if you adopt Outcome harvesting (mixed with another approach like contribution tracing) you would be able to trace the impact of your M&E system. I chose Outcome harvesting because it does not measure progress towards a predetermined objective, but rather collects evidence of what has changed and then work backward to trace a plausible relationship between the change and an intervention contribution to this change. So if government proposes to set up an agricultural bank, this can be traced back to recommendations from some M&E effort from a particular project, district, ministry or agency.

Secondly, if government  proposes agricultural insurance policy, this policy change can be traced to (i) The project Intervention;  (ii) Project M&E efforts (iii) Environment and what it means for agriculture etc.

Cheers.

Dear Colleagues,

I would love to share few points regarding the subject of measuring the impact of M&E.

1) There should be a distinction and clear line between M&E with regard to their purposes, roles, tools, timing, scope, responsibility, etc. While the monitoring is an ongoing process, the evaluation is one of the main phases of any intervention / project cycle (simply we can imagine the Deming's cycle: "Plan-Do-Check-Act").

2) The overarching purpose of both M&E activities is to "Enhance Performance and Continuous Improvement". This should be achieved through the ongoing (M) and snapshot (E) assessments of the inputs, processes, outputs and impacts of programs and interventions which should lead to better understanding of the gaps, challenges, emerging needs, changes in situations, etc.

3) The main point here about the impact of M&E is the M&E system itself. The system should be robust and well-designed to NOT only measure and detect BUT also to ensure actions, recommendations and decisions of M&E activities to be well-noted and implemented.

It's a close cycle and all outputs of M&E processes should be communicated, implemented, re-assessed and to be integrated into planning and future program designs.

Best regards

Dr. Tarek Sheta
Executive Director
DevSpaces International

Highly-esteemed Abubakr,

Thanks for bringing this important issue on monitoring & evaluation in one of the most important challenges of any M&E system related to its 'social learning' dimension. Besides, it was quite informative to read the contributions submitted within this debate – due to your suggestion – especially the ones of Ronald and Zahid.

The situation you depict is the one that is similar to what you might find in other African countries – I was involved between 2013 and 2015 in a very interesting AfDB initiative entitled "Africa 4 Results" and had a chance to visit some Western and Eastern African countries to face a very similar situation.

I don't have all necessary information to argue anything about your country but I have the feeling that in your case, the building of a National M&E seems to have started from the "harware" part and did not pay attention to the "software" issue. Sometimes I have the weakness to believe that in your case much attention was given on projects and projects monitoring collected data do not fit into national policies. And for this I would join my voice to Ronald and Zahid's contributions.

Having that said, we need to acknowledge that the construction of a national M&E system must start the publishing of a M&E general legal framework that will first will impose upon a Government to have a mid-term strategic plan of "multi-dimensional" development to which is annexed a results framework. This national strategic plan must have been prepared through a "true" participatory approach et be endorsed at end by the Parliament.

At the second level, this national "multi-dimensional" development plan will serve for each sector as a reference framework to establish a mid-term strategic sectoral plan to which is annexed a sectoral results framework. Each strategic sectoral plan must be approved by the Government and should bear a results framework that links the sectoral strategy to the mid-term national development plan.

At this level, any new project or programme will need to have a results framework that will link this project or programme to the sectoral plan. This is the "software" part that I mentioned above.

After that, the "hardware" part of the national M&E system is setup upon a concept note showing the inter-relations between the different levels of the national M&E system; the standard form of M&E unit at the different levels; the data collection procedures and methods; the reporting system and its timing; etc.

With this, one can assume that once monitoring data is collected at a project level can easily be aggregated at the sectoral level, getting the sectoral plan to feed back into the national strategic plan.

In such a situation that you bring in, starting with the "hardware" part, the majority of Government high and line staff might feel that M&E is just and additional "administrative" workload that is imposed from the top and lack of conviction in M&E will be very apparent.

Thinking of disseminating M&E results is highly recommended but talking about M&E "value for money" may just be seen inappropriate as M&E work is a sort of "quality insurance" or "life insurance" for development, and using such a metaphor, one can easily admit that having a "quality insurance" or a "life insurance" has certainly a cost, but omitting to have that insurance will certainly have a "at-least-ten-times" higher cost. This is why I believe the concept "value for money" is not the right concept to a given M&E system. I do not ant to be too much provocative but I feel that this issue of "value for money" is just a "proxy" indicator for a lacking conviction towards M&E work.

Kind regards

Mustapha 

Mustapha Malki, PhD
535 avenue Ampere #5
Laval, QC, Canada

Dear Abubakar,

Thank you very much for your response and queries.

I will go by Dr. Emile's viewpoint, generally it's not actually the monitoring & evaluation system itself which transforms into impact, rather the way the interventions/actions/projects are designed becomes critical to assess the impact of certain interventions.

If I correctly understand your queries, the impact M&E can drive is simply the question of having M&E system in place or vice versa. For this, as far as my understanding and experience is concerned, you need to perform some sort of counterfactual analysis (probably using systematic reviews/mixed methods/triangulation) to determine whether having M&E systems in place transformed into some sort of policy change/ improvement or not, no matter what sector/ indicators you are taking into the consideration. The scale and breadth of such analysis would be dependent upon various factors (including but not limited to resources, expertise, priority areas, number of interventions to account for).

Happy to discuss further in case of any queries.

Regards,

Zahid Shabbir

Joint Director | Program Management Division | AC & MFD |State Bank of Pakistan |

Dear Colleague,

The improvement of the different sectors does not depend only on monitoring and evaluation. It depends both on the quality of the actions developed, their management and then the monitoring and evaluation.

When an action is poorly developed, either because the problems have been poorly identified / formulated or because the actions selected are inappropriate, there is a high risk that the results will be bad: low impact. Even if this step is still successful and the management of the implementation of the actions is not adequate, the risk of failure is still high, only that in this case, the monitoring-evaluation has more power to formulate remedies that, if taken into account, can lead to the right result. Given the persistence of the poor quality of the results that you highlight, it is certainly on the development of actions that we must make serious improvements. If your monitoring and evaluation units have nothing to be ashamed of, that is to say that they are made by people who are really knowledgeable about it, there is no question of bringing in other people who might even damage the work if they are incompetent or partial. Generally, the monitoring-evaluators are not liked and everything is done to prevent you from working well. I presume that these kinds of obstacles must exist in this unit where you work, since it is the presidency, a high place of politics. But, is this still the case in all other sectors? I do not think so. So, above all, review the methodology of developing actions in the country. There must be serious problems at this level.

Thank you.

Dr. Emile Houngbo

Agricultural Economist

Expert in project development and monitoring & evaluation

Dear Zahid Shabbir,

Thank you very much for sharing very useful information.

However, my interest is to know how monitoring being carried out informs better achievements or delivery of the broader outcomes over a period of time.  This is because when, for example, carrying monitoring of headline indicators (somewhat similar to broader outcome over the period of time), the results of the monitoring is expected to inform better realization of the broader outcomes over the period of time.  

So how do we measure the contribution of the monitoring being carried out, towards helping in informing realization of better broader outcomes over the period of time? This is because the monitoring of the broader outcomes is not done for the sake of it but with the aim to inform better realization of the broader outcomes over the period of time.

There should be therefore a way of determining whether monitoring being carried out is contributing towards better realization of the broader outcomes being monitored over time of not.

The same applies to other levels of monitoring.

That is the focus of my assumption.

Kind regards

Abubakar Muhammad Moki

Uganda

Dear Abubakar,

It was nice to see this discussion on the subject. I would like to share our experience in Pakistan as we developed an M&E system for National Financial Inclusion Strategy of Pakistan (NFIS) with the assistance of World Bank Group. I would limit my discussion relevant to your queries.

First of all, we need to put quite a discern between monitoring and evaluation systems, especially the design of monitoring mechanism is very important. Out here, we developed a three tiered architecture for M&E; (i) monitoring of headline indicators (somewhat similar to broader outcome over the period of time), (ii) Activity/Action level monitoring transliterating into programs, policy or regulatory interventions, (iii) bottom tier, a project level monitoring which could be strengthened using blended tools like result based monitoring (RBM), project performance management using Earned Value Management technique (depending upon nature of project through indicators like CPI – Cost Performance Index, SPI – Schedule Performance Index, etc). Similarly, various concepts like value for money (optimizing output through minimal input) could be used to assess possible impact during the design phase of project/intervention.

With regards to evaluation specifically, I think least kind would be participatory evaluation where organization itself could be involved to evaluate the impact of interventions it has undertaken in specific sectors. However, choice remains with the organizations how to go by this endeavor, especially if its novice in the area. In such cases, donors/ multilaterals could be engaged to seek professional assistance (as is the case with many developing economies). I am sure, you must be aware of some sort of project performance reports to cater the information about viability of projects/programs like PPARs – Performance Planning and Revies, ICRs – Investor Confidence Rating, MfDRs – Managing for Development Resources, PfR-Project feasibility Reports, etc which are some of the common reporting mechanisms to assess the development effectiveness used by many multilaterals / donors.

As far as your queries regarding the monitoring of agricultural productivity/ service delivery are concerned, the technological developments have gone by far than required. For instance, you may begin a pilot with Unmanned Aerial Vehicle (UAV) technology in certain area to experiment the productivity results of specific corp/use of fertilizer etc. Similarly, Precision Agriculture is another important technological development which has yielded very good productivity results in case of India. The good thing about these systems is automated data storage and tracking capacities which can greatly assist in monitoring productivity and impact you are looking for can be deduced over the period of time and can witnessed though the ladder of Theory of Change (ToC).

I hope you would find our experience helpful to get fruitful results.

Regards,

Zahid Shabbir
Joint Director | Program Management Division | AC & MFD |State Bank of Pakistan |

Dear Reagan Ronald Ojok,

Thank you very much for your contribution.

It is true M&E is being done at different levels being it at national, district, sub-country etc. 

Ministers, Parliamentarians, District Resident District Commissioners, their Deputies and Assistants technical officers at the Ministries, district Councilors and district technical officials, sub-county Councilors and technical officials etc all are involved in some kind of M&E work.

Resources are being utilized for M&E work at the different levels.  How do we then measure value for money of the M&E at the different levels?  We should for example be to tell that M&E work  being done at the lower levels is creating better impact than M&E being done at the national level because of lower M&E proximity to the service beneficiaries.

So what yardsticks can we use to determine which of those different players has better M&E impact and is able to influence more resources to it due to its greater impact?

I believe if we put our ideas together, we can come up with a framework of indicators that can be used to identify who does M&E better due to its better influence for better impact.  

This is yet to come.

Grateful

Abubakar Muhammad Moki

Greetings Mr. Abubakar,

If I hear you well, the challenge of measuring Monitoring and Evaluation work is just a symptom of a bigger problem with the national M&E system. Your ministry (office of the prime minister) has the mandate to oversee M&E functions in the country. The office of the president also carries out some parallel M&E work, including other government ministries (MoFPED-BMAU) charged with responsibility of carrying out budget M&E.

In such setting how do we make all efforts complement each other? How do we measure the M&E work as government? The answer to these questions might not come at the end of my submission but am glad we are discussing it now. It is very important!

While all the above is happening, there are also some pockets of project based M&E work in other government agencies and ministries (small scale). You rightly stated a possibility of coming up with some indicators of sort, that is a good proposal. However, locating the national M&E system within the broader planning strategy for the country (NDP 2) and linking the system to the different efforts would help in aggregating all M&E efforts. This way, I think you might be able to measure the impact of M&E work, maybe by producing an annual series of Reports on Impact of M&E for government of Uganda "The Annual M&E Outlook".

All the best and feel free to get in touch if need be.

Thanks,

Reagan Ronald Ojok
Uganda Development Bank

Dear Samuel, 

Thank you for your feedback. What you raise is correct and I appreciate.

However, is there a way we can measure the contribution of a good M&E system in informing better agricultural production, productivity, food security and service delivery?  What are those parameters that should be considered and how should they be measured so that one knows the impact or change the M&E system has contributed?

In other words, how can one determine what would have been missed due to an absence of an M&E systems - leaving out its contribution? or how to we carry out contribution analysis of an M&E system towards better agricultural production, productivity, food security and service delivery?

Grateful

Abubakar Muhammad Moki

Dear Naser Qadous

Thank you for the good feedback.

As we all know, M&E is not carried out for the sake of it.  One of the reasons behind carrying out M&E is to provide evidence for informing improvements in agricultural production, productivity, food security and service delivery in general.

The issue now is, how do we determine that M&E carried out has contributed to informing improvements in agricultural production, productivity, food security and service delivery in general? 

How do we determine the contribution of the M&E carried out, so that there is value addition seen out of the investment in carrying out the M&E?

This will help in documenting value addition of each M&E being carried out.

Grateful

Abubakar Muhammad Moki

Uganda

Dear Abubakar,

It is obvious from you discussion that your current M&E system is not functioning as designed. The challenges you mention are not unique to your country; I have found out in my recent work that in many African countries, existing national M&E systems face many challenges including:

1. Difficulty in accessing project and program level data for quick decision-making,
2. Lack of standardization in data generation resulting in data that is of poor quality and unreliable,
3. Delays in submission of periodic evaluation reports by various sector agencies and ministries,
4. Little or no support for real time monitoring and reporting, hence, delays in decision-making and remedial interventions,
5. Existing national M&E system  not providing for adequate citizen participation,
6. Capacity of M&E personnel at all levels.

I also found that some sector ministries and agencies have their own M&E systems which, are specific to their needs and are not linked to the national M&E System therefore making it extremely difficult to compare and analyse national evaluation data.

My proposed solution is for your country to strengthen the existing M&E System to ensure that it is effectively coordinated and efficiently managed by developing a comprehensive, functional and integrated national M&E System that covers all levels of governance (national, district, sub-district), institutions (including sector ministries), projects and programs implementations. This solution must be backed by a legislative instrument/directive to ensure active participation of all stakeholders including Government leaders (Parliamentarians, Ministers and others). Developing a comprehensive M&E system would entail developing a national M&E framework which would define the theory of change, results chain, indicators etc. etc.

Let me know if you need further details

Best regards,

Samuel Boakye

ICED

Dear AbuBakr,

Once I read your Email I felt I have to respond soon. It is a very interesting and very advanced topic "The impact of M&E, and Indicators for this Impact". I think it is much related to utility/ use of the outputs of M&E. Now in most cases sharing of the results of M&E is questionable even at the organizational level with exceptions of some very advanced learning organizations who make their reports available for the public. Again I think that our role in informing policy as consultants is having a good output of our work following the best consulting practices and working  with clients not for them bearing in mind the learning objective of M&E. The M&E people do not have direct authority on policy. This is what evaluation voluntary organizations (VOPEs) are trying to do through awareness and lobbying with Parliamentarians and I think they succeeded in some countries.

For evaluating the role of M&E I think it would not be easy to measure the contribution at the impact level, but going down to outcome and output level would be easier.

To give example of indicators:

• # the number of downloads of an evaluation report or # the number of reaches/ views
• % of M&E reports shared on websites of related organizations

Many others can be mentioned.

For agriculture, the research results in general not only evaluation need to be adapted to the field and be applied research not only scientific. The research in developing countries is weak and needs to be adapted to meet local needs and employed in the field.

At the end I say nothing is impossible and can be done even the Impact evaluation of Evaluations or a systematic Review.

The discussion could be very long

Thanks

Naser Qadous

Palestinian Evaluation Association
Agriculture/ Evaluation Expert