Rapid evaluation to measure the impact of the COVID pandemic in mountainous areas

Dear all,

As a final note to this discussion, below you will find a guidance and a toolkit for rapid evaluations developed by the Department of Planning, Monitoring and Evaluation of South Africa which has started using rapid evaluations as one of the forms of evaluation, with thanks to  Antonio Hercules for sharing these resources with EvalForward.  

• Evaluation Guideline – how to undertake rapid evaluations

• Rapid evaluations Toolkit

Interest in rapid and real time evaluations is increasing and over the next months we will keep sharing experiences on how to overcome the challenges raised and ensure quality and usability of these evaluations.

All the best,

Renata

Hello Community

First of all, I would like to thank you for your contributions by describing your very rich respective experiences and by proposing approaches and tools to ensure that Rapid Evaluation process responds to expectations.

The responses highlighted common problems, namely the time and resources needed for a rapid evaluation of the effects of an intervention or of COVID 19. These resources are often more than planned.

Slow communication between different partners and the availability of archived and analyzed data are also raised as major factors that slow down the implementation of rapid measures.

As for the proposed solutions and tools, Jennifer Mutua advised to pay attention to unexpected factors  when estimating the budget for the evaluation. Carolina Turano suggested ways to improve communication between partners to make the process agile. Elias SEGLA proposed data collection tools and suggested that the rapid evaluation team should be internal with its own organisation chart and modus operandi. Nayeli Almanza's response describes the rapid data collection methodology to measure the impact of COVID 19 on migrant populations and Aurélie Larmoyer gives practical suggestions for individuals and teams to work in order to improve timeliness of protocols and reactions that can be crucial for the intervention.

To summarize, I think that 1) it is necessary to work on the communication time between partners during the development of the approach as well as the feedback and reaction; 2) it is necessary to use modern tools, especially virtual means and mobile phones; 3) the budget issue remains and could require innovative work from the team to optimise the results within the budget allocated.

Finally, I hope that this issue of rapid evaluation will receive the attention it deserves and will be developed especially during this COVID 19 crisis.

Thanking you once again for your answers and shared references, I remain at your disposal for further exchanges on this issue.

Yours sincerely

Malika

[This comment was originally posted in French] 

Dear Malika,

Thank you for raising such an important question. I find it interesting in two respects:

First, because it raises the question of how we can capture the immediate (or medium term) effects of the COVID-19 situation on our realities. Many evaluators are grappling with this question. Some colleagues in the UN system have worked to draw some general directions in this respect. For instance, the recent publication from the ILO office of Evaluation (https://www.ilo.org/wcmsp5/groups/public/---ed_mas/---eval/documents/publication/wcms_757541.pdf) might be of inspiration, as it lists, in annex, typical evaluation questions that match the need for collecting specific information relevant to COVID-19.

I also find your question interesting because it asks about how to make rapid evaluations, which we also had many possible reasons to aim for even prior to the pandemic, and on which therefore there are past experience to build on. And, if our colleague Jennifer is right in underlining that evaluation does not easily lend itself to fast reaction, I think there are ways to expedite processes to cater for the urge of timeliness. I can share the following learning points in respect to what worked when I aimed for conducting evaluations rapidly. First, focus. It makes a difference when someone’s time is entirely to the task, while multitasking takes away the precious focus you need to get to where you want fast. Second, aim for a good enough plan. We often go round in circles to prepare our evaluations, and invest a lot of time in back and forth exchanges over it, a straighter line, it can help to start with a rough scoping and testing and refining your focus and approach as you go along. Third, compensate any cut corners with engaging few select stakeholders with strategic knowledge as sounding board along the way.

Of course, the COVID-19 situation complicates these rules of thumb, in particular when engagement needs to be virtual; and my last piece of advice is to get savvy with modern technologies for engaging by virtual means. As you report, this might last, so might be worth investing in such new competences.

Best, Aurelie

Dear Malika and community,

You have proposed a very interesting topic with an urgent need to obtain steady information as the pandemic evolves.

As for the International Organization for Migration of the United Nations, we have developed and applied a methodology in Mexico and some countries in Central America based on telephonic interviews with key stakeholders in the field. This methodology is called Displacement Tracking Matrix (DTM) and is widely used by IOM since 2004. 

We have identified that by asessing communities representatives, shelter directors and actors from the local governments and social organizations, we can obtain general information about the impact COVID-19 over mobility and needs of the migrant population settled in the borders, as well as in movement across the country.

Particularly, I was leading the development of these studies in Mexico. So far we have 3, but the latest one in english is available here:
https://dtm.iom.int/reports/mexico-emergency-tracking-southern-border-m…

There is some information that synthetizes the methodology approach that you can access here: https://displacement.iom.int/system/tdf/tools/Methodological%20Framewor…

If you have further comments or questions, don't hesitate to contact me.

Greetings,

Nayeli Almanza, MSc

Dear community,

Thanks to Malika for bringing up this issue of rapid evaluation, which shows once again the difficulties we often encounter in the practical implementation of certain theoretical notions.

My point of view on the issue is that of an institutional actor and not that of a consultant. In Benin we have started to conduct rapid evaluations, a new concept to which we have been exposed in South Africa with the "Twende Mbele" Programme (a cooperation programme in evaluation that we have initiated with South Africa and Uganda to strengthen our national monitoring-evaluation systems through the sharing of experience and the development of collaborative tools).

It is within the framework of this programme that we have developed a specific methodological guide on this type of evaluation and have simultaneously undertaken 4 rapid evaluations, 3 of which concern public interventions and the 4th relates to the effects of COVID-19 on the informal sector.

First of all, it must be said that the major difference between rapid and traditional evaluation lies in the constraints of time and limited resources that characterize rapid evaluation. In Benin, for example, a normal evaluation (excluding impact assessments which can take up to 5 years) takes on average 9 months to 1 year, or even longer, due to many factors related to procedures (notably administration, procurement, institutional management especially when there are many stakeholders), and sometimes even due to the data collection and analysis phase, which is often lengthy. Rapid evaluation therefore calls for new processes to reduce the time of the pre- and post-data collection phases. With the adaptation we made in Benin, the overall time for a rapid evaluation was estimated at 12 weeks maximum. This implies a data collection period of a maximum of two weeks to a month, to allow time for initial activities, organising data collection, analysing results, writing a draft report, obtaining observations and finalising the report. However, it is a reality check to see if this is realistic.

In terms of tools, the gap in time and scope can be filled by using rapid methods:

- Collection with groups (rather than individuals), workshops,

- Use of routine data or other evaluations,

- Team work to carry out different steps at the same time (Collection and processing)

Furthermore, we believe that in order to effectively save time, it is preferable for the rapid evaluation to be carried out by an internal team, because this is the only option that does not require a procurement procedure. But it does require that appropriate organizational mechanisms are put in place, for example:

- a project organisation chart for the team,

- the organisation of the weekly working time to be devoted to the evaluation mission and strict delivery deadlines,

- support measures for the team, etc.

In addition, as the evaluation team did not reside in the communities, we identified focal points in the data collection settings to facilitate with community members in order to save time.

Malika was not specific enough to enable us to propose solution approaches adapted to her context. But from the little I have retained, here are a few ideas that I share for the moment. I could provide more factual elements once we have learned from the experience currently underway at the Benin Public Policy Evaluation Office.

Thank you and good luck to all of you.

Elias SEGLA 

[the original contribution is available on the French page]

Dear Malika and community,

Thank you for sharing your challenges in carrying out rapid evaluations. 

I am currently involved in a Real Time Evaluation (RTE) conducted by the FAO Office of Evaluation. Despite the fact that the methodology of this RTE has been designed to provide real-time feedback to FAO teams and partner organisations, it is in fact taking longer than initially anticipated. The reasons were more or less similar to those mentioned by Jennifer, namely bureaucracy processes being slow, recruitment of national staff, travel restrictions, among others. 

Nevertheless, I believe that some adjustments could be made to the ‘regular’ way of conducting evaluations, to make the process more agile. This may eventually contribute towards providing feedback and recommendations more ‘real time’. Some ideas could be to (either/or): 

• Increase the communication flow with the relevant stakeholders /communities involved to provide regular updates on the evaluation process. Each update should also include a short overview of preliminary findings that emerged in that particular stage of the evaluation. 
• Organize periodical workshops to allow discussions on preliminary findings and conclusions with a wider audience and in a participatory manner, and use this opportunity to draw ideas for recommendations together with the audience.
• Create brief communication products to disseminate preliminary findings and recommendations throughout the evaluation process and not just at the end of it. 

Although I am fully aware that some of the challenges related to time and resources involved in evaluation processes are simply difficult to overcome; creating a space for continuous feedback throughout the evaluation process, using different forms as highlighted above, may be one way to allow communities/stakeholders to react in a more timely manner.

I hope this contributes to the discussion.

Kind regards, 

Carolina 

Dear Malika,

Thanks for sharing your experience. Our initiative as shared on this platform [www.evalforward.org/comment/reply/node/118/field_comments_ref/19], involves a joint partnership of the Evaluation Society of Kenya (ESK) and Monitoring & Evaluation Department (MED), funded by the World Bank. So our context is different from yours, as it involves coordination between national and county governments (making things more complex). Nonetheless, like you, our experiences were that indeed the rapid evaluations were not as quick as initially planned (and as the term seems to imply). Much more resources than initially planned were also expended. Some contributing factors to these were:

• Slow start owing to the protocols/bureaucracies in communications between the national and county governments
• Sometimes delayed responses at the technical level (especially where staff are thin on the ground)
• Limited documentation including well analyzed and archived monitoring data
• Generally, M&E Systems are weak. This limits the ability to make quick studies.  Alternative means involving wider scope and additional monies had to be employed, towards redressing data gaps

In view of all these and subject to context, I think, it’s important to take into account these limitations, including potential budget over-runs at the evaluation planning stages.

Kind Regards,

Jennifer