SDG evaluations: How to eat an elephant

I thought I might share some of the lessons we learned from our Evaluation of FAO’s contribution to SDG 2 – Zero Hunger.[1]

I was recently discussing the challenges of evaluating SDG support with Ian Goldman from CLEAR and Dirk Troskie from the Western Cape Government Department of Agriculture, South Africa. They seemed somewhat surprised that we had embarked on such an endeavour, as the causal links between the 2030 Agenda and action at country level are tenuous and hard to pinpoint.

Nations assess their progress against SDG targets through Voluntary National Reviews presented to the High-level Political Forum on Sustainable Development. This system of reporting and monitoring, which involves UN agencies such as FAO for the indicators each agency is ‘custodian’ for, is meant to keep pressure on signatories and hold them accountable. However, FAO cannot force a country to do something it doesn’t want to do. It’s all about ‘soft’ power ‒ persuasion, inspiration and influence, things that appear intangible. So, how do you assess that?

The first thing we did was to clarify the expected causal pathways. For SDG2, we came up with not one, but two theories of change. The first focused on the roles and expectations of an organization such as FAO regarding the SDGs (theory of change No.1 in the figures below). We then developed a theory of change for SDG 2 itself (theory of change No.2), regrouping the targets into three levels, each with two targets: support for food production (targets 2.A and 2.5), food production (targets 2.3 and 2.4), and food consumption (targets 2.1 and 2.2), with markets (targets 2.B and 2.C) linking the three.

Theory of Change No.1 - FAO support for the SDGs (link here)

Theory of change No. 2 ‒ a simplified representation of SDG 2 (link here)

Arranging the targets in this way, we noticed a tension between the left- and right-hand sides of theory of change No.2. On the left-hand side, targets 2.1, 2.3 and 2.A are all about a traditional view of agriculture, oriented towards incomes and the production of enough staple food. On the right-hand side, targets 2.2, 2.4 and 2.5 call for a transition to more sustainable and equitable agriculture and food systems. This SDG 2 theory of change therefore could be summed up as follows: Countries need to reorient and expand their agricultural support systems (research, extension, etc.), markets and policies to promote evolution towards more diverse, more sustainable and more equitable agriculture, maintaining biodiversity rather than eroding it and producing diverse and nutritious food at a lower environmental cost.

SDG 2 calls for a decisive shift away from the “productivist” philosophy of the Green Revolution of the 1960s and 1970s, which created many of the problems we face today for instance by promoting the widespread use of pesticides and chemical fertilizers. This aligns well with the underlying principles of the 2030 Agenda ‒ in particular the emphasis on environmental sustainability; social and economic inclusion (to “leave no one behind”); innovation; and acting at scale.

This preliminary theoretical work gave us a series of dimensions embedded in the 2030 framework, against which to assess FAO support for SDG 2 and its evolution over time, namely, an analytical framework grounded in the 2030 agenda as designed.

Then we broke down our vast scope into 14 thematic studies (see links below) and 15 country case studies. In selecting these, we chose to focus on what seemed, at first glance, to be good practices that FAO could learn from and disseminate. This was a pragmatic decision based on the formative nature of the evaluation. There would have been little sense in visiting countries where FAO’s assistance had been negligible or unremarkable.

But there was a risk involved: that we might paint an overly positive picture of FAO’s performance at country level. We offset this by developing a standard methodology and rating scales for the dimensions mentioned above, such as the degree of attention FAO programmes paid to “leaving no one behind” and to environmental sustainability. We then triangulated the results with a broader sample of prior evaluations at country level.

What did we find? FAO had helped design the SDGs and singularly SDG 2, which historically stems from the Zero Hunger Challenge. However, FAO’s performance was markedly better in global communication and advocacy than it was at country level, where operational weaknesses limited its contribution to SDG 2. Progress towards more sustainable food systems is extremely slow and in fact, while global food and nutrition security was progressing prior to 2015, it has been regressing since the SDGs came into effect and further undermined by the COVID-19 pandemic. Clearly, the SDGs are not a magic bullet.

The SDG 2 theory of change (No.2) highlighted the importance of trade, markets and consumer taste as drivers of the transformation towards more sustainable agriculture, and our case studies confirmed this. There are many ways to support SDG 2, one of them being through the food that we all choose to buy and consume ‒ and the message that consumer choices sends to food producers around the world.

I would like to emphasize the following methodological lessons:

• Despite the precautions we took, there was a lot of heterogeneity in the studies. Having a large number of case studies helped counter this problem. Each of the country and thematic case studies led to its own set of recommendations for improvement, in addition to the recommendations of the overall evaluation, as a way of putting the material to good use.

• The positive bias inherent in the search for “good practices” was not as strong as we had feared, as it was balanced by an opposing force ‒ optimism. If you ask an organization such as FAO to nominate good practices, you will get a lot of candidates, not all of which will actually be good. Some of those will, in fact, be average or poor. Consequently, our sample of case studies allowed us to bring in not just the good news, but quite a lot of bad news as well.

• Another lesson learned was the danger of complexity. We spent a lot of time untangling the connections between SDG 2 and other goals. This work – conducted with the help of the  Centre for Development & Environment (CDE) of the University of Bern ‒ was useful in identifying certain trade-offs between agriculture and other goals such as water pollution and habitat destruction, but difficult to communicate because too complex. We thus created a simplified version (theory of change No2 below) where the connections to other SDGs are only suggested, rather than mapped.

The Western Cape Department of Agriculture did something quite similar: tasked to evaluate how the Department had contributed to SDG 2 and related national and local development goals, they analysed the question through sound theory, identified 14 key intervention points, and set up to monitor and evaluate through a well-suited, semi-standard methodology.

They came up with a nice summary for it: How do you eat an elephant? You just cut it into pieces.

(no elephants were harmed during this evaluation)