In the second of our three-part interview with leaders and managers currently engaged in humanitarian-development evaluations, we asked how the Covid-19 pandemic is affecting the practice of evaluation, and where to draw the Do No Harm line (see first part here).
Dr. Robina Shaheen, Global Head of Monitoring, Evaluation and Learning Services, Action Against Hunger, UK: There is a greater burden of data collection now on respondents. In addition to the time that they need to spend responding to evaluator’s questions, they may also need access to internet and mobile devices. Unless evaluators can cover these costs or respondents are able to bear these themselves we are forced to rely on convenience sampling, accessing and relying on those respondents that have the relevant means to communicate and can potentially bear the costs where required. This is not ideal and also adds biases that affect the overall findings and may potentially not provide an accurate and dynamic representation of the results and processes.
Those unlikely to have access to the required technologies and devices are the poor and most vulnerable. This therefore means their voices go unheard at a time when the impact on these groups will be greater than ever and we need to hear their voice more than ever before to adapt the programmes.
Muriel Visser, Team leader for the evaluation of WFP school feeding contributions to the SDGs: The biggest risk is that evaluations will be deferred or deprioritised during implementation at a time when learning from experience is more important than ever. On Do No Harm, it is important not to ask in-country national evaluators/researchers to take risks with their own and families’ health that international evaluators or the client would not take. It is important to ensure that data collection does not put beneficiaries or other informants at risk. This means that data collection is more likely to take place remotely and/or to be conducted on a one-to-one basis or with very small groups. Evaluations will usually cover a longer period than just the Covid-19 period. There is a risk that the effects of Covid-19 make it more difficult to capture the picture prior to the pandemic because of challenges to availability of key informants but also because Covid-19 may have affected many of the expected outcomes from interventions in a substantial manner.
Gaby Duffy, WFP Senior Evaluation Officer: I do not see any specific threats to independence. Credibility comes from a rigorous process and sound data, so certainly the challenge is to be able to collect enough primary data, and to be transparent about any limitations that emerge during the evaluation. In some cases where remote data collection is not feasible, and secondary data sources are weak, OEV has decided to cancel the evaluation. Do No Harm guides my decisions as an evaluation manager. In cases where field missions may be possible, the question to ask is still is it the right thing to do? Am I putting evaluators, beneficiaries and other stakeholders at risk by having an international team fly in to conduct interviews? If there is a minimum risk at all, then it is clearly not the right thing to do.
Martine Van de Velde, Leader of the team evaluating WFP’s country strategic plan in the Lebanon: The biggest challenge has been ensuring we give a voice to the vulnerable population groups being targeted under the interventions. The evaluation is strategic in nature and we have been able to build on evaluations conducted before the Covid-19 outbreak. These earlier evaluations had a strong emphasis on beneficiary consultations which allowed us to consider primary survey data and transcripts from focus group discussions. Initially we were considering bringing in-country based evaluators onto the team and proceed with limited in-country consultations. However, from a Do No Harm perspective, exposing beneficiaries and national colleagues to unknown and preventable risks was unacceptable.
Marie-Hélène Adrien, Team leader of the evaluation of WFP Policy on South-South and Triangular Cooperation Policy: The threats amount to limited country observation and potential for limited access to communities with no access to internet for remote calls. Even if borders reopen, countries may require a quarantine for visitors and this will add to the costs of the evaluation. Do No Harm is ensuring Covid-19 does not put at risk the health of the evaluator; WFP staff; WFP partners and stakeholders.