FAO Office of Evaluation is currently concluding its evaluation of FAO’s contribution to Sustainable Development Goal (SDG) 6, “ensuring availability and sustainable management of water and sanitation for all”.
This is the third SDG evaluation our office has conducted (see Olivier Cossée’s experience of evaluating FAO’s contribution to SDG 2 here). Experience told us it would be a monumental task but, needless to say, that proved an understatement. Here are some of the things we learned.
Lesson 1: Explore fully the potential links between the SDG being evaluated and other SDGs before deciding on the scope of the evaluation.
Determining what to evaluate in an SDG evaluation is not always clear cut, as the SDGs are often interconnected. There needs to be flexibility in the allocation of evaluation resources (budget and duration), but it is also important to recognize that hard choices on scope will need to be made.
SDG 6 has six technical targets, two means-of-implementation targets and eleven global indicators. The technical areas span a range of thematic areas from the quantity and quality of water supply, water pollution and water resources management to water use efficiency, transboundary water, ecosystems, sanitation and hygiene. A portfolio analysis showed that FAO worked in all SDG 6 target areas to varying degrees. Logically, therefore, our evaluation set out to assess the extent to which FAO had contributed effectively to these areas. This, in itself, was a mammoth exercise. We ended up conducting 10 thematic studies, including thematic areas linked to the various SDG 6 targets and cross-cutting issues such as climate change, gender equality and the principle of leaving no one behind.
Furthermore, as a key input to agricultural production, water is directly related to many other thematic areas of FAO’s work, such as crop and animal production, aquaculture and fisheries. Crops and livestock account for 70 percent of global water withdrawals and play a major role in water pollution. It was, therefore, crucial to examine the footprint of FAO’s entire water-related portfolio. To cover this aspect in depth, the scope of the evaluation would have to more than double in size. Alas, we realized this in the inception phase, when we had already used up a third of our timeline and no longer had sufficient funding to recruit more experts in other thematic areas.
What did we do? We adjusted the scope of the evaluation. We continued to focus on FAO’s work in the SDG 6 thematic areas, but also conducted country case studies that enabled us to examine FAO’s entire portfolio of work at national level for its potential positive or negative effect on water.
If we could re-do things, we would allocate more resources to covering the potential positive or negative effects on water of FAO’s non-water activities and recruit a wider array of experts in areas such as forestry, fishery and ecosystems. We now know that for any future SDG-related evaluations, it would be wise to spend more time exploring an SDG’s interconnectedness with other goals before determining the scope of the evaluation.
Lesson 2: The SDGs can provide a useful, comprehensive analytical framework for thematic evaluations.
In thematic evaluations, the SDGs are often used as means of linking organizational contributions to high-level impact. They are infrequently used as an analytical tool or framework for thematic evaluations. For example, if our evaluation were of FAO’s work on water resources management, we would probably only look at its water-related work and assess the key results achieved, including how these contributed to the SDG targets. It would be a one-way, straightforward analysis. Our focus would probably lie more in the areas where most of FAO’s work occurred and less in areas of little work.
This being an SDG evaluation, however, we were compelled to look at FAO’s work through an SDG 6 lens, with its various interconnected target areas. We found that FAO worked extensively in areas related to SDG 6.4 (water use efficiency), 6.5 (integrated water resources management) and 6.6 (ecosystems), for example, but little on SDG 6.3 (water quality and pollution). The latter raised a red flag because of the interconnectedness of agricultural activities and the use (or overuse) and pollution of water resources. This enabled us to question potential trade-offs, where achievements in one area could jeopardize others.
Another lesson we are taking away, therefore, is that applying an SDG lens to a thematic evaluation can enrich our analysis and correct our tunnel vision.
Lesson 3: The SDGs are more than the sum of their targets and indicators.
In our SDG 6 evaluation, we looked at how FAO supported its Member Nations in different SDG target areas. If this had been a project or programme evaluation, we would have explored the use of output and outcome indicator data to measure results and contributions. However, the status of SDG indicators at country level is not a valid measure of FAO’s contribution. Moreover, focusing solely on the areas monitored by these indicators would narrow our vision. During our inception phase, we realized that focusing too much on the indicators could make us lose sight of other important aspects of the targets not captured by them. There was also a risk that we might ignore the interconnectedness of different aspects of water resources management and links to other SDGs. In the end, our evaluation focused on whether the spirit of the SDG and its targets were well reflected in the FAO’s work at national level.
What we learned is that we must pay attention to the intention behind the SDGs and their targets.
Conclusion
The evaluation report of FAO’s contribution to SDG 6 is being finalized and will be posted here. A video presentation of the evaluation findings, conclusions and recommendations can be found here. Overall, it found that FAO had undertaken a great deal of good work in the SDG 6 target areas, but with significant gaps and a general sense of fragmentation and poor coherence. Even though water is central to FAO’s mandate, its visibility in the Organization’s strategic documents and in field projects was surprisingly low. The evaluation recommended that FAO build on its existing strengths and move towards the coherent and strategic recognition of the central role of water resources management in FAO’s actions in agriculture and food production. It specifically recommended that FAO develop more explicit approaches to water governance and to water quality and pollution.