How do we adapt our evaluation approach to the impact of the Covid-19 pandemic?

Thank you, Nick Maunder, for bringing this up. The Covid-19 pandemic put to test every evaluator on how innovative and resilient one can be. My president’s directive to work from home, reached us when we are in the process of data collection in the field. This was an outcome harvesting. We had a replanning meeting over the night and decided to prioritize the focus group interviews (the stories) to avoid being locked down away from our homes. Focus group discussions would have been challenging to collect through skypes or telephone calls. The storytellers (FGD) are mostly community members. They have challenges in accessing bundles, telephones, internet, and most other forms of communication.

The rest of the data from the key informants (substantiating) was collected through skype meeting, telephone calls and WhatsApp calls. It was not easy as put, but eventually, we were happy with the effort we put and the data collected.

Therefore, the evaluators need to continually remind themselves on what the goal of a particular evaluation is. And how best they can gather the data.

Dear Members,

Following up on this timely thread on how to adapt our evaluations in the time of Covid-19, I am sharing the fresh blog from WB colleagues and experts on evaluation methods. The blog provides a decision tree "Making Choices about Evaluation Design in times of COVID-19" and some practical examples. 

Alena

Following up on the issues raised earlier on telephone surveys, I would like to share our recent experience from an ongoing evaluation of a waste management project in Zaatari refugee camp, hosting Syrian refugees in Jordan. The four-year project was funded by the European Union and implemented by FAO. 

Since all FAO evaluation missions have been cancelled, but we had already recruited a team to carry out the evaluation, we decided to do as much as possible of the evaluation remotely. The project aimed to improve livelihoods with increased green job opportunities through the integral utilization of residues of treated wastewater and biosolid in generating renewable energy and compost. As the project had a small number of direct beneficiaries – 33 persons (mainly women) who were employed at the waste segregation unit- and the FAO project team had all their contact details, we decided to undertake phone interviews remotely to seek their opinion of the project.

Here are some lessons: 

• The importance of advance warning: Prior to the interviews, the project team produced a short flyer in Arabic with information on who we were, why we were going to be in touch and the overall purpose of the evaluation. The flyer also included details on the phone number that would call them and who would call. This was printed and distributed by FAO project staff to the 33 beneficiaries working in the waste segregation unit.  We then texted them, introducing ourselves and saying we wanted to talk to them and organise a suitable time to talk.  These exchanges of messages until a suitable time was found to speak, allowed them to agree to being interviewed and ensured there was an informed consent when proceeding with the interview. As the initial messages were sent by WhatsApp, we also had the advantage of seeing if the messages were arriving and being read.
• A flexible methodology: We reached beneficiaries using their language (Arabic) and we soon realised that by contacting over the phone (rather than through meetings during an evaluation mission) we had a lot more flexibility in arranging the interviews at a time that was suitable for the beneficiaries. The majority of the beneficiaries were women, so they were able to decide on the best time for them, among all their household and work chores. Our Arabic speaker evaluator was a man and we soon also learnt that this would not be a problem when speaking to them. Actually our evaluator was pleasantly surprised at how chatty the female beneficiaries were and how social norms that one may have encountered if the meetings were in person, did not have such an overburdening role.  A large number of the evaluation questions were about livelihoods  and we also found that all respondents were very open to describe their financial situation and the economic implications and that the anonymity of the call dissipated the sensitivities of sharing information on income.
• As Covid-19 is a global challenge, we found that the beneficiaries of the project fully understood why were not there in person and the potential risks of meeting up, so they were very understanding and helpful – at times for example offering to trace other people from the team if we were not able to contact them. Some participants even volunteered to take part in group discussions, which we did not take up as we did not want to encourage them meeting up in groups due to the risk of the spread of Covid-19. From the FAO side, the limited engagement of the project team also allowed us for a more direct communication channel with the beneficiaries and less chance of a bias from the project team as we interviewed 20 out of the total 33 persons.

Syrian refugees at Zaatari camp have a very high level of ownership of telephones, so our case might not be applicable in all contexts. As also highlighted in the previous contribution, when shifting from face to face to phone interviews it is important to be aware of the trade-offs. Telephone interviews are also far from perfect, in our case while we got very rich conversation we did miss the body language and other information and aspects from visual observation that help you understand  things better – are they comfortable talking? How do the project team and beneficiaries relate to each other?

Today Evaluation can no longer be a standalone exercise. In the context of this crises it needs to become an ongoing and action-oriented process throughout the intervention (developmental evaluation).

It has to contribute concretely to people’s social, environmental and economic wellbeing as well as to peace locally and regionally as well as globally.

I would like to quote Michael Quin Patton’s recent blog: “All evaluators must now become developmental evaluators, capable of adapting to complex dynamics systems, preparing for the unknown, for uncertainties, turbulence, lack of control, nonlinearities, and for emergence of the unexpected. This is the current context around the world in general and this is the world in which evaluation will exist for the foreseeable future” and would recommend reading it https://bluemarbleeval.org/latest/evaluation-implications-coronavirus-global-health-pandemic-emergency

Isha Wedasinghe Miranda
Independent Evaluator and Programme Management Consultant

Sri Lanka 

Hello dear members!

I thank NICK MAUDER for having launched this reflection among us. Indeed, I fully share this observation on the cessation of ordinary activities carried out, in particular those relating to evaluation exercises. In fact, this is an unforeseen situation, but one which gives rise to reflections and provides the opportunity to reinforce the approaches used until now. I have often had the opportunity to manage evaluations in conflict zones where displacement is not possible due to insecurity. In these cases, I used "telephone" survey methods. This type of survey requires following some principles, among others:

1. The formulation of short and precise questions, to avoid long discussions;

2. A fairly strict time management because the people concerned can be discouraged quickly by lack of direct physical contact;

3. A good orientation of the discussions, stay focused during the interview;

4. The language of the interview is important in that the interviewee must be comfortable with understanding the questions asked, often interpreters are required;

5. Good planning of the interview can often be time consuming because it requires a moment of total availability of the person or group.

However, the method requires more triangulation efforts to verify the data collected. It must be said that this is an alternative methodology but not a replacement for human contact. Indeed, human contact is always the best means of evaluation because it allows the evaluator to make observations and personal observations which can support the appreciation of the facts.

Dear colleagues,

Thank you very much to Nick for starting this very relevant topic and discussion.

I particularly would like to stress the ethical responsibility we carry as evaluators mentioned by Carlos. Some countries might not have major restriction in place yet. In fact, it would be legal for the local team to conduct focus groups and face-to-face interviews. However, it is up to an evaluator to decide whether it is ethical. This might imply that even local consultants would need to conduct data collection through online engagement tools. It had recently happened to my colleague managing an evaluation in Indonesia and Brazil where the team decided to avoid face-to-face data collection by consultants in both countries as they deemed it unethical. 

As so much remains unknown about Covid-19, any decision we make with regards to our current and future evaluations will be based on imperfect data. Science presents different scenarios but some of them suggest that it might take up to 1,5 years for the health situation to stabilise. This health emergency might be a good opportunity to learn how to design a methodology for a credible evaluation at a distance. 

On 1 April, our colleagues from USAID are offering a free webinar "Discussion on Challenges and Strategies for M&E in the Time of COVID-19". Interested members could register here: 

https://www.eventbrite.com/e/discussion-on-challenges-and-strategies-for-me-in-the-time-of-covid-19-registration-100817255124

Best regards,

Alena 

Dear Nick,

Thanks for your very interesting post, and for sharing your experience in grappling with this enormous challenge. I wish to share a couple of thoughts, from my role as evaluator and as a commissioner of evaluations.

In the short-term, I wonder the value that stakeholders may give to evaluations done at distance, and in the medium-term, the threat this pose to evaluation as a profession.

What is the added value of an evaluation that is done from distance? In a couple of ongoing evaluations with large field components we are facing some issues whose consequence we should not underestimate as they might reduce the credibility of the whole exercise (inability to observe first-hand changes, reliance on the evaluand to select who participates and who does not, limits to triangulation with beneficiaries and local partners, etc.) and put our teams in danger of being challenged in case they come up with negative or erroneous if not inaccurate findings.

Then, evaluation as a profession: if we are doing things from distance and without credible triangulation and bottom-up participation, what makes us different from those doing reviews or even performance audit? If we advocate for distance evaluations, and colleagues/partners realize that these can be done cheaply and in a non rigurous manner, we may have issues in the future i) selling evaluation as a distinctive and truly learning tool, and ii) getting adequate evaluation provisions/budgets.

Linked to this, the moral imperative for evaluators of not making harm. In view of all the unknowns that this pandemic is bringing, it is our duty not to put more people at risk, neither local evaluators nor beneficiaries. Trying to postpone evaluations if feasible, at least till it becomes clearer what we could safely do in the field and what we cannot, it will just be a fair and ethical thing to do.

Best regards,

Carlos