Claudia [user:field_middlename] Martinez Mansell

Claudia Martinez Mansell

Evaluation Manager
FAO
Spain

My contributions

    • Following up on the issues raised earlier on telephone surveys, I would like to share our recent experience from an ongoing evaluation of a waste management project in Zaatari refugee camp, hosting Syrian refugees in Jordan. The four-year project was funded by the European Union and implemented by FAO. 

      Since all FAO evaluation missions have been cancelled, but we had already recruited a team to carry out the evaluation, we decided to do as much as possible of the evaluation remotely. The project aimed to improve livelihoods with increased green job opportunities through the integral utilization of residues of treated wastewater and biosolid in generating renewable energy and compost. As the project had a small number of direct beneficiaries – 33 persons (mainly women) who were employed at the waste segregation unit- and the FAO project team had all their contact details, we decided to undertake phone interviews remotely to seek their opinion of the project.

      Here are some lessons: 

      • The importance of advance warning: Prior to the interviews, the project team produced a short flyer in Arabic with information on who we were, why we were going to be in touch and the overall purpose of the evaluation. The flyer also included details on the phone number that would call them and who would call. This was printed and distributed by FAO project staff to the 33 beneficiaries working in the waste segregation unit.  We then texted them, introducing ourselves and saying we wanted to talk to them and organise a suitable time to talk.  These exchanges of messages until a suitable time was found to speak, allowed them to agree to being interviewed and ensured there was an informed consent when proceeding with the interview. As the initial messages were sent by WhatsApp, we also had the advantage of seeing if the messages were arriving and being read.
      • A flexible methodology: We reached beneficiaries using their language (Arabic) and we soon realised that by contacting over the phone (rather than through meetings during an evaluation mission) we had a lot more flexibility in arranging the interviews at a time that was suitable for the beneficiaries. The majority of the beneficiaries were women, so they were able to decide on the best time for them, among all their household and work chores. Our Arabic speaker evaluator was a man and we soon also learnt that this would not be a problem when speaking to them. Actually our evaluator was pleasantly surprised at how chatty the female beneficiaries were and how social norms that one may have encountered if the meetings were in person, did not have such an overburdening role.  A large number of the evaluation questions were about livelihoods  and we also found that all respondents were very open to describe their financial situation and the economic implications and that the anonymity of the call dissipated the sensitivities of sharing information on income.
      • As Covid-19 is a global challenge, we found that the beneficiaries of the project fully understood why were not there in person and the potential risks of meeting up, so they were very understanding and helpful – at times for example offering to trace other people from the team if we were not able to contact them. Some participants even volunteered to take part in group discussions, which we did not take up as we did not want to encourage them meeting up in groups due to the risk of the spread of Covid-19. From the FAO side, the limited engagement of the project team also allowed us for a more direct communication channel with the beneficiaries and less chance of a bias from the project team as we interviewed 20 out of the total 33 persons.

      Syrian refugees at Zaatari camp have a very high level of ownership of telephones, so our case might not be applicable in all contexts. As also highlighted in the previous contribution, when shifting from face to face to phone interviews it is important to be aware of the trade-offs. Telephone interviews are also far from perfect, in our case while we got very rich conversation we did miss the body language and other information and aspects from visual observation that help you understand  things better – are they comfortable talking? How do the project team and beneficiaries relate to each other?