Launching a new series on Evaluation in times of Covid-19, this is the first of a three-part interview bringing together reflections from leaders and managers currently engaged in humanitarian-development evaluations on how the pandemic is affecting the practice of evaluation.
Dr. Robina Shaheen, Global Head of Monitoring, Evaluation and Learning Services, Action Against Hunger, UK: The biggest effect has been on our ability to propose methodologies that require face to face in-country data collection. This would have been undertaken either by international staff and/or in-country data collection teams. Prior to Covid-19 face to face access to in-country respondents provided greater direct control and the ability to oversee and ensure quality control. The current situation has forced us to think about how we use remote means to both access respondents and collect data without risk to evaluators and respondents. Whilst we have been using mobile devices to collect data in the past, the additional challenge now is remotely accessing respondents and sites. This means exploring audio and visual technologies through which interview, focus group discussion and observational data can be collected remotely. However, whilst evaluators may have access to the relevant devices and technologies, the greater challenge is respondent’s access to these and the related costs. For example, telecommunication networks are not always reliable and mobile use and internet is expensive.
Muriel Visser, team leader for the evaluation of WFP school feeding contributions to the SDGs: Travel restrictions have meant that evaluations have to move to remote meetings/interviews and remote methods of team working. This is less disruptive for mixed-methods evaluations, where it is possible to adjust the balance between methods as well as basic ways of working, and for strategic evaluations, where data collection does not depend so heavily on on-the-ground surveys, face to face interviews and visits to project locations. Challenges in such situations are much greater.
There are some positive, as well as negative, effects of remote communication. Firstly, inception “missions” tend to drag on much longer when done virtually, because the pressure to fit meetings in a specific and usually short time-frame during the presence of evaluators in country is removed.
Secondly, when an evaluation team cannot physically meet for a workshop or other form of preparation for the evaluation, it may make sense to spread the team workshop over a number of virtual sessions, rather than concentrate all the collaboration into one or two days. This offers the advantage of having time for reflection in between shorter meetings and also reduces the fatigue that we have found sets in during remote meetings. Thirdly, reaching informants particularly at country and field office levels, as well as technology is a challenge at times, so more time has to be factored in to deal with these challenges. The role of the evaluation focal point from the client is even more important, as is taking time to identify the right informants during the inception phase.
Gaby Duffy, WFP Senior Evaluation Officer: The greatest impact has been on the field missions to collect primary data. This has forced our evaluators, to find alternative ways of gathering the perspectives of stakeholders, mostly using remote data collection tools. It also implies a more robust use and analysis of secondary data. These decisions are made together with field colleagues and the evaluators, who so far are willing to adapt to the fluctuating situations. As the pandemic evolves, the plan to conduct evaluations is reviewed, to see if field missions are indeed feasible and safe. So it requires a huge amount of flexibility.
Martine Van de Velde, leader of the team evaluating WFP’s country strategic plan in the Lebanon: Travel restrictions have precluded in-country missions to consult stakeholders at face-to-face meetings and prevented evaluators observing intervention results on the ground. To proceed with evaluation, we have needed to quickly adapt to a remote approach, based on virtual consultations with stakeholders and a greater emphasis on the review of documentary evidence. WFP staff and the evaluators have been required to consult closely and agree quickly on approaches for working remotely. For both the evaluators and the stakeholders, there are concerns about the future because of the pandemic and balancing working from home with family demands. Dealing with connectivity issues has been an additional challenge on occasion.
Marie-Hélène Adrien, team leader of the evaluation of WFP Policy on South-South and Triangular Cooperation Policy: Changes can be highlighted in four main areas. Firstly, methodology, there is no ability to travel for inception missions at HQ, nor for pilot countries to test our data collection tools. Secondly, timing, there is uncertainty about the timing and the methodology for in-person data collection. Thirdly, country sampling, some country offices are unable to host an evaluation mission. And fourthly, costs, the revision of proposals to reflect all Covid-19 related implications
How have we adapted? Firstly, we conduct a remote inception phase; secondly, we build decision check-points in the calendar of the evaluation to revisit the methodology; and thirdly, we extend the timeline for the execution of the evaluation – timeframe invariably takes longer. What can be achieved in a four-day in-person country visit was spread over 2 weeks.
Evaluation in times of Covid-19 is a space for evaluation leaders, managers and experts to share thoughts on the possible implications of the pandemic on the practice of evaluation