How are development projects affecting the environment and how do we evaluate this impact?

I wanted to also support  and commend Yosi for raising this important topic. Recently I attended the 4th Conference on Evaluating Environment and Development in Washington, D.C. and one message stood out in particular to me - if we do not evaluate an intervention's effects on natural systems (positive or negative) and not only human systems, our evaluative conclusions will be incomplete and invalid.

Dear Yosi, 

Thanks for raising this important topic!

Agri-food systems generate over a third of global greenhouse gas emissions: it is therefore crucial to find win-win solutions promoting resilience across climate, environment, and development simultaneously.

Evaluations play a crucial role in this context: since 2010 IFAD’s Evaluation Policy provided for all project performance evaluations to address how projects support Environment and Natural Resources Management (ENRM) and climate change adaptation (CCA). The Evaluation Manual includes guidance for rating performance on these two criteria, along with OECD DAC criteria and others. These specific ratings and assessments are subject to the same quality assurance process as all IFAD’s evaluations, and the performance on these criteria is reported to the governing bodies for learning and accountability.  

However, while incorporating such considerations is the first step, ensuring that sufficient know-how is available in each evaluation to credibly assess the ENRM and CCA effects is quite another matter. 

In the last 10 years, 90% of IFAD projects evaluated were rated moderately or satisfactory in their support to ENRM and CCA. However, only 30% were found to do no net harm to the ecosystem. This points to the need for a close examination of the methods being used to rate ENRM and CCA by the IFAD Evaluation Office. 

A recent thematic evaluation on IFAD support for smallholder farmers to adapt to climate change, reviewed how IFAD’s agricultural projects interacted with the surrounding ecosystems. Methodologically, no precedence existed to assessing this human-ecosystem nexus. The Evaluation Team developed a rubric approach to assess the consequences of IFAD projects on selected dimensions of ecosystem – such as, water quality and management and soil health.  Of the 20 case studies conducted, covering 14% of IFAD’s portfolio of projects engaged in Climate Change Adaptation, only six were ‘doing no harm’ or better, as shown in the figure below. 

Dear colleagues,

thank you for engaging in this discussion and for your useful feedback.

Thanks Adeleke for sharing how your CSA project measured environmental impacts – I have previously come across ‘hectares of land under improved CSA practices’ as a potential indicator and I see you measure not only sustainable land management but also ‘reduction of agricultural land
expansion at the expense of forest lands’ which is useful to know. It will be interesting to see the future impact of some of the outputs related to sensitizing the community on climate risk issues and introducing RR actions and strategies.

Thanks Steven for drawing our attention to your new eval framework which attempts to ‘mainstream’ climate into the evaluation cycle - I was not aware of this and it does indeed seem to be a useful tool in assessing the weight given to climate considerations during evaluation and what lessons
can be gleaned from this and applied to project design.

Thanks Lal, your discussion on the 3 groups of indicators of adverse environmental consequences. In the past, the indicators I have come across mostly focus on the physical environment and largely ignore those related to fauna and flora. Hence, your framework is very useful as it takes into
account the nuances and interdependencies between the environment and the flora and fauna in a specific area. Thank you also for the sobering reminder that a magical list of widely applicable indicators does not exist and instead, these indicators should be developed based on contextual
information and field data.

Thanks Silva your illustration is spot on – indeed, I find my risks and assumptions sections are too concerned with the project results and what hinders’ their achievements rather than how the project results are affecting the environment. This requires a new change in mindset – thank
you for the reminder.

Asante Daniel, to your first point, I have found that environmental and social safeguards are sometimes used at the project design phase to ensure the project meets a certain threshold (to avoid, minimize, mitigate adverse impacts) to qualify for approval. However this process is at times
completely disconnected from the evaluation process, however these two phases should be connected in a feedback loop. You make a lot of observations about the interaction between those doing the measuring and the communities in which this process is taking place – I will continue to
ponder these!

And my thanks also to you Hadera for providing such an indepth description of project level output indicators that can logically point to outcome and impact level indicators on mitigation and adaptation - I will keep these in my indicator reservoir. It will be interesting to see whether in the
future, these type of CSA projects will be designed (and evaluated) in a way that allows us to make concrete estimations of impact level indicators such as reduced emissions based on project level data.

Best regards,

Yosi

My contribution to the ongoing discussion:

Thank you Yosi for raising this important and contemporary discussion topic.

In my experience in development project evaluation and project preparation works (projects related to agriculture/animal farming and natural resource management), indicators are inbuilt in the development projects. The indicators can be direct indicators or proxy indicators.  During development project evaluation, evaluators are provided with “TOR” to evaluate the success of the project.  In doing the evaluation works, indicators inbuilt in the project document and their respective means of verification are core base for the evaluation work. The type/nature of the development project (say is the project climate smart/environment friendly) determines the level of contribution of the project to climate change mitigation and adaptation. 

Thus at the stage of development project evaluation, I believe evaluators mainly depend on the indicators inbuilt in the project. So the question would be: was due attention given to  prepare and implement  climate smart projects,  in the face of the alarming  climate change?; and what indicators are used/can be used to measure  the  level of the projects’ contribution to climate change  mitigation and adaptation?   

Since the last 10-15 years or so, development partners including financial institutions at all levels (national, regional and at international levels) are better aware and have better understanding of the alarming climate change. They understand climate change is real, although, it is true that the level of awareness/understanding varies. 

To this end, I believe substantial efforts have been made by relevant development parties including FAO, among others, to identify, make available and promote/scale up climate smart agricultural/animal farming/fishing practices to mitigate and adapt climate change. These days it is becoming mandatory in most cases (when situation allows) to design and implement climate smart development projects. That is projects, which contribute to climate change mitigation and adaptation.  Having said so:  how can then the climate change mitigation and adaptation contribution levels of climate smart development projects be measured?   What type of indicators can be/have been used? Direct indicators or proxy indicators?

My experience: I will mention one of the development project evaluation works I was involved in, in the evaluation of the “National Livestock Development Project implanted in Ethiopia, which was thoughtfully designed to be climate smart/environment friendly”.  Among others, this project has employed different climate smart practices with the aim to overcome/decrease animal feed shortage. The understanding during the preparation of the project was to reduce the emission of the major Greenhouse Gases (Carbon dioxide, Nitrous oxide and Methane), by employing climate smart practices, while also meeting the objective of the project. Under the animal feed improvement component of the project, the activates carried-out include:

• Back yard forage development (perennial grass and legumes);

Indicator(s): number of farmers growing forage in their back yard; rate (%) of adoption; yield of forage  produced;

• Establishment  of pasture including perennial grasses and legumes: and planting fodder trees;

Indicator(s): Size of the established pasture; yield of forage produced; and number of communities, which established pasture; rate (%) of adoption; 

• Pasture rehabilitation by over sowing with legume plants;

Indicator(s): area /size of pasture rehabilitated; yield of pasture produced; 

• Under sowing crop lands with legume plants for forage use;

Indicator (s): area/size of crop land under sowed; number of farmers that under sow their cropland; yield of forage produced from the land; rate (%) of adoption;

• Alley cropping that is planting fodder trees within crop lands.

Indicator(s): No of fodder trees planted; number of farmers who planted fodder trees in their crop land; rate (%) of adoption; and yield of forage produced.

The above indicators are proxy indicators (indirect indicators), which are assumed to be related to direct impacts, in this case to the climate change mitigation and adaptation contribution levels of the animal feed improvement practices. All the animal feed improvement practices employed by the project are believed to contribute to climate change mitigation and adaption. To mentions some, they:

• reduce emission of carbon dioxide from soils because of soil plant cover;
• remove carbon dioxide from the atmosphere through the process of plant photosynthesis;
• enrich soil organic matter and are reservoir of carbon in the soil; 
• Improve the year-round availability/ production of animal feed/forage and as such minimizes overgrazing (which fuels carbon emission). 
• Improve soil fertility of crop lands, and hence improve crop production due to the inclusion of legume plants to the crop land. This in turn contributes to enhance food security and reduces natural resource degradation caused due to scarcity of food.

Dear Yosi,

Hi and, as Silva said, thanks for prompting responses to such an important issue. 

It's not the first instance of some organisations not following through  - either by monitoring or evaluation - on the criteria used to approve a project by their boards or programme committees. Indeed, and a generic point here, interest in the project at this level typically wanes following approval. Few boards receive feedback as to the consequences of their decisions, for good or ill.   

Moving on, and as an attempt to answer both your questions - on methods and indicators - simultaneously, in capturing and measuring climate change and enviromental impact, my reflections are about how:

1. Environmental (and social) safeguards should strive to be more ambitious; that is, by its rating certain metrics helping to predicting and then be used to 'proving' more than the project does no harm. What's wrong with having a balanced set of metrics being about doing good? Would this not help the latter - the doing good - being picked up in a ToC and/or a Logframe, less so an indicator more a result itself, so ensuring they are not forgotten? To what extent climate is integrated into evaluation is at least partially determined by to what extent it is an integral objective and central to the project's design. Or is this too obvious a point to make?
2. In agriculture related projects, there is limited interest in Indigenous knowledge systems on the culture of farming and conservation. So much so, that one gets the impression that "climate smart" agriculture is necessarily associated with the introduction of outside practice and technology. And I am not being naiive in stating this. Communities and individual farming households vary in the knowledge they have and farmers farm for many different reasons. 
3. How do communities and farmers "monitor and evaluate" their environment, including their farms. Would not they have valid practices and signs (read indicators) to M and E the impacts and to communicate and adapt accordingly. Local cultures, territorial governance systems, sustainable livelihood traditions and the experience of sacredness are all valid issues in this regard.
4. Such climate change and environmental projects need methods and indicators that reflect quiet, slow, long-term and  flexible support (Ken Wilson - https://news.mongabay.com/2022/02/journeying-in-biocultural-diversity-and-conservation-philanthropy-qa-with-ken-wilson/ As Ken further says:  “They are necessarily messy, meaningful, and organic; they do not thrive when we press upon our Indigenous partners the stereotypes of perfection". 

5. Finally, when we talk about  improvements or changes in the environment/climate change, as well as contributions to improved mitigation and adaptation, we sometimes forget two things. First, to be more inquisitive about how those who make the improvements and/or changes respond to the support projects offer, and the assumptions the project designers made about who will respond, how and why. These are often overlooked. We leap too quickly into "measuring" the consequences of these responses as dictated to by needy logframes and impatience. Second, as Silva pointed out, the need to understand how such improvements are inter-dependent on, not mutually exclusive to, farming and broader ecosystems  - meant with its original meaning.  The success of one is dependent on an improvement in the other. Take Resilience as another example. Are we saying that a household's ability to becoming more resilient is largely dependent on their acceptance of what any one project offers? In other words, capacities among households in becoming resilient is independent of what is going on and/or what is being provided with support from outside. I certainly hope not.

    

Apologies for the lengthy reply, yet I hope some of the above helps,

Shukrani nyingi na bahati nzuri 

Daniel       

Hello Yosi, thanks for this very important question! 

I am collecting some tips on including environmental issues in evaluation. This is one of them. Hopefully, I will share more.

See, a "thinking environment" is a mindset. 

The moment we take a more ecosystemic perspective, we will immediately realize the limitations of our approaches. 

But we also discover that simple things - as an extra question - can go a long way.  :-)

The Coastal Community in Pakistan is most affected by Climate change 

Dear members!

Evaluation of the Environmental impact of Development Projects

As far as one is aware, this is a long overdue topic that has sadly been neglected with disastrous consequences. Two of the best known examples of this are:

• Under the Soviet regime, damming Amu-Darya river to procure water for the cultivation of cotton and cereals led to the Aral Sea disaster that resulted in a dramatic reduction of that body of water and the salination and dessertification of a vast area which previously supported a large pastoral community for centuries.

• Directed by democracies of a sort, ‘Green Revolution’ implemented in Mexico and Pakistan among others, resulted in salination of semi-arid but previously arable areas; this was owing to the introduction of chemical-intensive cultivation of wheat as practiced in the USA where the climatic conditions were not comparable. Ironically enough, the innovator of the procedure, ‘Father of the Green Revolution’ received Nobel prize!

It would be irresponsible to overlook the resistance the commercial interests involved in projects did and shall continue to exert against the incorporation of environmental impact in the evaluation of every action connected with development. The outcry against the Maipo Valley development project in Chile should be a salutary example here.

After this somewhat gloomy preamble, it is necessary to unify two aspects of environmental impact that are unjustifiably taken separately, viz., environmental impact and climate change. Climate change has two origins; first, periodic changes in the configuration of the solar system relative to earth and variations in solar activity which are beyond human intervention. Secondly, impact of human activities on the environment which have an adverse effect on the availability of ecosystem services.

These ecosystem services are:

• A salubrious climate; whether the climate of an area is salubrious or not is determined by those who are indegenous to the locale. Having said that, it is indisputable that the world’s glaciers have been retracting for more than 70 years and average local temperatures have risen.

• Soil fertility; requires the equilibrium between the use and the return of soil nutrients.

• Availability of potable water; depends on the facility with which water cycle takes place.

• An adequate presence of pollinators and other beneficial animals.

• An adequate presence of indigenous flora and fauna which prevents the predominance of undesirable local or foreign plant and animal species. Failure here entails an increase in weeds and the increased use of biocides whose consequences are highly undesirable.

This list is not claimed to be exhaustive, and its purpose is to identify some of the reasons for including the evaluation of environmental impact of every development project. Here, one runs into two difficulties:

• What constitutes measurable indicators of environmental impact?

• Does what constitutes measurable indicators of environmental impact is the same for every project?

Obviously, measuring the reductions in local ecosystem services such as temperature, rainfall, or the degree of soil salination etc.,  after the completion of a project would be only of academic interest to the local inhabitants. Therefore, an evaluation of the environmental impact of a project should be always undertaken before its implementation. Having thus established the reason for pre-project evaluation of environmental impact,  it is necessary to identify the potential indicators of negative environmental impact.

These fall into the following distinct but logically linked categories:

• Equilibrium between the use and return of ecosystem services provided by a given physical environment.

• This equilibrium depends on two logically inseperable factors:

•  Stability of the physical features of an area;  these include its geography, geology, atmosphere, water ways, lakes, hills mountains, roads, buildings etc.

• Equilibrium between its flora and fauna; this has two dimensions viz., its biodiversity and the optimal population of each of the species in the area. No species is exempt from this requirement.

Other things being equal, the stability of the physical features of an area depends on the equilibrium between the flora and fauna of that area. For instance, the consequences of denudation of hill sides by human agency results in silting up of rivers and floods. Moreover, the resultant loss of green cover brings about a drastic change in the mechanism of heat exchange between the ground and space. Deprived of its green cover, ground absorbs more heat from the sun which is released back slowly, thereby increasing the local temperature. Further, loss of water from the heated ground interferes with the local water cycle, not to mention the generation of dust rendering the area susceptible to wind erosion.

In this instance, fauna i.e., man displays no equilibrium between himself and the flora, for his action is not sustainable. A sustainable action uses an ecosystem service, here it is timber, which is not returned through selected harvesting and replanting. It would be tedious to list many more of such greed/ignorance driven actions, but the reader may easily identify a vast number of them. For the present purpose, it would be sufficient to identify some of the critical indicators of adverse environmental impact.

In order to pre-empt certain trivial objections, it is necessary to point out that natural disasters like volcanic erruptions, earthquakes, storms and tidal waves do cause serious environmental damage, but earth’s resilience enables it to recover from them; unfortunately, this resilience is much undermined by thoughtless human activities and excessive population.

Another challenge is to establish justifiable base-lines for a set of suitable indicators. This is because some important basic knowledge is simply not available, and the competence required to generate them on an ad hoc basis is lacking in every country irrespective of its standard of education or wealth. This will become clear as one proceeds.

To sum up the critical points that have emerged, the stability of the purely physical environment of a given area depends to a large measure, on the equilibrium between the flora and fauna endemic to it. Of course, the latter often may undermine it to a certain degree, for example, tree roots loosening hillsides and animal burrows collapsing riverbanks. But, the resilience of the whole is often sufficient to mitigate the ill effects of such events.

Meanwhile, human interventions often bring about destabilisation of the physical environment of such magnitude, it becomes impossible to mitigate its adverse effects on the environment and making good the resultant loss of ecosystem services of an area. Consider the effects connected with the Aral Sea disaster. Thus, indicators of adverse environmental consequences may be placed in two categories:

• Human interventions that directly impact only on the stability of the purely physical environment. It ought to be noted however, that they may affect the areas biosphere as well in an indirect way. This separation is made simply for the sake of clarity. Eg. Emission of green-house gases by factories, vehicles etc.

• Actions which destabilise the physical environment and the equilibrium between it and its flora and fauna. Eg. Deforestation of hillsides and erection of buildings that adversely interferes with the heat exchange between the ground and space.

• Undertakings that disturbs the equilibrium between the flora and fauna of an area. Eg. Use of biocides and the introduction of foreign species.

Before one can get to specific indicators, it is essential to acquire certain basic information on the existing physical environment and its biosphere. Unfortunately, neither of these are easy to come by in spite of the much vaunted capacity of miraculous latest technology with a dash of AI stirred into it. Indeed, the current physical geography and geology may be ascertained by such means, but the difficulty is that their current status is often too degraded by previous activities hence its use as a base-line may be misleading. Data on previous average rainfall, temperature and atmospheric constituents is not always available. Thus, one is compelled to resort to an ad hoc standard here.

As for reliable surveys of previous flora and fauna of an area, matter is even more problematic. Such information is often patchy or non-existent. Even making a survey of present fauna and flora of an area seems to be extremely difficult. One has consulted many graduates of biology in affluent and less affluent countries only to find that while they knew much about plant and animal genetics and molecular biology, they failed to identify even the commonest species extent in their own area.

Therefore, one is left with only one alternative to overcome these challenges. It involves using what is known to bring about the above three types of change as the indicators of actions that entail adverse environmental impact. This proposal may not be the best, but, under the circumstances, it may go some way to avert another Aral Sea disaster or the aftermath of another green revolution that is certain to turn mud brown.

As noted previously, what is indicated is a stringent pre-project evaluation before it is too late. It would consist of the following steps, expanded or skipped over with reference to their relevance to a given development project:

• Determine which of the three above categories of adverse environmental change the proposed project is likely to entail. For example, a factory or a power plant on a barren ground would imply an increased emission of green-house gases. Here, the evaluator may request the project planners to consider a more benign alternative.

• Infra-structural improvements inevitably involve destabilisation of the physical environment and loss of flora and fauna. Mitigation of this  would require two-fold strategy; first, seek the least harmful alternative, for instance, in transport, give priority to water, rail and road transport. Secondly, planting indigenous trees/shrubs/bushes by the banksrailways and roads, and their nurturing and follow-up would somewhat mitigate the negative impact on the environment. Depending on the terrain, it may repay to dig trenches parallel to its contour lines to counteract the resultant loss of water retention in the area.

• Agriculture and attempts at environmental regeneration do frequently disturb the equilibrium between the local flora and fauna. In agriculture, this is unavoidable, but recommendation of agro-ecological methods and multi-culture go some way to lessen the negative consequences of present traditional mode of food production. In regenerating degraded environment, reject the introduction of foreign species which has already done more harm than good. Rapid growth is not a viable way of environmental regeneration. It is vital that the new plantations are nurtured for at least 5 years and then followed up for another 5 years.

It may be objected that no specific set of indicators has been proposed here. This is quite true, but the fact remains that it is impossible to set forth a universally applicable set of relevant indicators. Here, the evaluator has been offered a few guidelines; ascertain which category of adverse effects on environment a project may bring about. To do this effectively, the evaluator must study both the project proposal and the area where it is to be implemented. On considering the target area, the evaluator may need the support of local expertise, which may frequently be anecdotal. Beware of highly qualified expert opinion from sources that have never been to the target area.

After this, the evaluator would be able to identify what adverse effects the proposed project could have on the environment and propose some adequate means of their mitigation. He would have no choice in the matter, for in addition to the three types of adverse change, variations in the physical environment of target areas is legion. One’s success here solely depends on one’s analytical ability honed by experience.

Best wishes!

Lal Manavado.

How have your evaluation methods captured the impact of development projects on the environment or climate change? 

This is a timely question because although the importance of integrating climate considerations into development programs is increasingly recognized, how such programs account for climate change often remains overlooked.

A good starting point for meaningfully capturing the impact of development programs on climate change in evaluation methods is to first ‘mainstream’ or integrate climate considerations throughout the evaluation. In a 2021 paper published in Global Food Security (https://doi.org/10.1016/j.gfs.2021.100509) we shared a framework with guiding questions for different evaluation components:

Evaluation scope 

a. Does the introduction of the evaluation acknowledge a climate change issue(s)? 
b. Does the evaluation include an objective/question/criterion specific to the assessment of climate change adaptation, mitigation, and/or impacts? 

Evaluation approach 

a. Is climate change adaptation, mitigation, and/or impacts mentioned in the evaluation theory, methodology, methods, and/or analysis? 

Evaluation results 

a. Does the findings section provide information on climate change adaptation, mitigation, and/or impacts? 

b. Does the conclusion provide information on climate change adaptation, mitigation, and/or impacts? 

c. Are there specific recommendations to address climate change adaptation, mitigation, and/or impacts?

What indicators have you found to be most effective in measuring improvements or changes in the environment/climate change, as well as contributions to improved mitigation and adaptation? Emission levels? Resilience measures? Climate finance raised? Insurance products made available? Or others?

In the above-mentioned study, we also applied the framework to examine evaluations of UN agencies working in food and agriculture (e.g. FAO, WFP, IFAD, UNICEF, UNEP, UNDP) and found many different approaches and indicators used. For example, IFAD defined a new adaptation criterion in an updated evaluation manual (2016) as: “The contribution of the project to reducing the negative impacts of climate change through dedicated adaptation or risk reduction measures”. IFAD also offered core questions to guide the evaluation such as: “To what extent did the program demonstrate awareness and analysis of current (climate) risks?”. 

It is important to note that climate mainstreaming in program planning and evaluation is not keeping up with the urgent need for climate action. In a paper currently in press in WIREs Climate Change titled “Greener through gender: What climate mainstreaming can learn from gender mainstreaming” (doi: 10.1002/wcc.887), we leverage lessons from gender mainstreaming to accelerate progress in climate mainstreaming, drawing on a review of mainstreaming practices from the UN agencies mentioned above (stay tuned!).

Steven 

Great discussion topic Yosi.
I'd like to share some thoughts based on my involvement with the regional BOAD funded project "Promoting Climate Smart Agriculture in West Africa".
The project seeks to strengthen the resilience of people to the adverse effects of climate change and to increase production while contributing to mitigation through carbon sequestration. This project has environmental benefits that include: (i) sustainable land management and reduction of agricultural land expansion at the expense of forest lands; (ii) contribution to the mitigation of GHG emissions through carbon sequestration; (iii) improving the capacity of actors to implement climate resilient practices; etc.

The project's results framework includes relevant indicators (all of them are effective), and here are few of them:

• Percentage of the target population by means of resilient livelihoods to climate change suffered
• Rate of improvement of yields to support food security and improve the living conditions of beneficiaries
• Type of income sources for households generated under climate change scenario
• Number of beneficiaries (F/M) informed about climate risk issues through the actions of meteorological services
• Number and type of risk reduction actions or strategies introduced at local level
• Level of technical capacity of regional, national and local institutions to promote climate resilient best practices in an CSA approach
• Number of community plans or policies improved or implemented that incorporate the CSA approach.

To read more about the project: https://www.adaptation-fund.org/project/promoting-climate-smart-agriculture-west-africa-benin-burkina-faso-ghana-niger-togo/