RE: Challenges of evaluation | Eval Forward

Dear Hynda,

You have raised a very important question, which affects the quality of evaluation work. Evaluations of development programmes in the broadly defined areas of rural development, agriculture and food security are inherently complex. The assessments of results in these areas are affected by a multiplicity of biophysical, economic, and social systems and factors. There are different types of constraints and challenges in evaluation work that depend mostly on the context of the programmes or policy work being evaluated.  For example, accurate and timely assessments of potential impact and development change may be affected by the remote location of project sites, social stratification of rural communities, time required to produce productivity gains, adoption capacities of local communities, and many other factors.

Evaluators often encounter issues with availability of baseline data, or information on the prevailing conditions of the development situation at the start of the projects or programmes addressing food security and agriculture development. This issue could be addressed by reconstructing baselines, for example, using ‘recall’ technique, i.e. requesting key beneficiaries or stakeholders to recollect information about these conditions in the past.

Security situation in the country may also have a huge impact on the access to data and methods we chose for evaluation. The choice of evaluators could also be highly limited, as not all may have necessary clearance to visit high-risk areas, or experience in working in similar situations. 

Accessibility of project sites may also be restricted or banned. To address these constraints, local consultants with access to restricted zones may provide support in data collection, and potential alternative evaluation methods could be also considered. In recent FAO’s evaluation of the large irrigation rehabilitation programme in Afghanistan, evaluation team faced a constraint of accessing some of the programme sites. The team opted for alternative method by using the open-source data from Google to assess the potential impact of the programme on the livelihoods in those specific sites. Google Earth maps were utilized to measure the expansion of the irrigated area and the vegetative cover along different sections of the rehabilitated canals. The methodology for measuring these areas was also using preliminary information from enumerators in the field who had access to the restricted zones, and were engaged in supporting collection of necessary data and information for the evaluation (e.g. the GPS coordinates of the irrigated areas in the vicinity of the irrigation canals). Then this information was analyzed based on historic data available from Google Earth on before- and after-project conditions and the changes based on vegetative cover at different periods during a year.

These are just a few highlights of the constraints and challenges that evaluators may encounter in their work and an example of possible ways to address those. The range of such constraints is quite broad, and we encourage all members of this community to share their experiences in addressing different types of constraints and limitations.

Kind regards,

Serdar Bayryyev,

Evaluation Officer

Food and Agriculture Organization (FAO)