Are you considering social and environmental criteria in your evaluations?
Dear EvalForward evaluators,
I believe we are all aware of the fact that development activities may have unanticipated environmental and social effects and of the interactions between social and environmental factors. Major agencies and the UN in particular are considering how to integrate environmental and social impacts in their evaluations, and how to ensure they are included when the ToRs does not mention them. These can be:
- Environmental impacts: Climate change, droughts, flooding or footprints of projects, environmental risks and hazards.
- Social impacts: Disability, indigenous people, vulnerability, poverty, resilience-building, inclusion, and social cohesion.
For example, in its efforts to promote the New Way of Working under the so called “Nexus approach”, the UN Country Team in South Sudan paid greatest attention to evaluation of both environmental and social factors of its core contribution to country beneficiaries. This included: stronger support to mitigate flooding and droughts; enhancing household food production and strengthening capacities to absorb and adapt to shocks; the fight against gender-based violence; delivering cost-effective, high-impact essential health services to the most vulnerable targets that have experienced the strongest effects of conflict in the country and assisting in assessing and monitoring the accessibility of health services and service readiness; assisting with the recovery of economy through a comprehensive approach to rebuilding trust and re-establishing access to basic services; restoring productive capacities and nurturing effective partnerships; support the safe, voluntary and dignified return of displaced people to allow them to rebuild their lives and return to productive activities (2019-2021 UNCF core contribution in South Sudan).
- Are you considering such aspects in your evaluations?
- What are the challenges you have in doing so and how do you think we should address them as a community?
I hope to hear your experiences and ideas!
Serge Eric Yakeu DjiamCredentialed Evaluator / Evaluateur Qualifié
It is awesome to get your comments and inputs on this interesting topic. I echo some colleagues to agree that the contextual factor should always guide the consideration of these two key development working areas both in the design and the evaluation process.
I am happy to hear from Ms. Rathner that the UN working group is about to formalise the UNEG wide guidelines on this important matter.
More contribution and thoughts are welcomed.
Season's greetings and happy new year to all and to your beloved families.
Serge Eric Yakeu Djiam, Ir., M.Sc., CE
Credentialed Evaluator / Evaluateur AccréditéInternational Evaluation Expert & Visiting Professor Policy Evaluation Research & Rural Development Co-Chair, EvalIndigenous (EvalPartners Network) Vice-President, International Development Evaluation Association (IDEAS)
Martina RathnerPrincipal Evaluation SpecialistUNESCO
Dear EvalForward members,
The dedicated UNEG Environmental and Social Impact (ESI) working group conducted a stocktake and initial assessment of existing agency guidance on incorporating environmental considerations into evaluation practice. This showed that some UN agencies’ policies and guidance are already integrating agency specific environmental and social impact considerations. In a next step the working group envisages formalizing UNEG wide guidelines. The initial assessment can be found here: http://www.uneval.org/document/detail/2951
Malika BounfourPresident Association Ayur pour le Développement de la femme Rurale
What we measure reflects our values and what matters to us. These are notes from the article below I came back to share with you as a complement to my previous message. Also, the article suggests that impacts on subgroups need to be reflected in the measurements. This is in accordance with recommendations of our meeting participants
Have a good day
Malika BounfourPresident Association Ayur pour le Développement de la femme Rurale
Hello everyone and thank you for this subject and for sharing your thoughts and approaches.
This is parallel to the discussions held last week by the francophone network for evaluation at the FIFE2021.
That said, I also believe that it is more an issue of méthodologies to bring out these aspects. Organisations usually set their policies to take into account environment and gender in programming and evaluation.
I also would like to share with you that as side event to the Mountain International Day, we organized a meeting around "the olive tree, the Mountain and the environment ". Main and first recommendation by participants was to consider multi-level social and landscape planning and analysis when talking about rural setting. This is because mountain conditions or results (almost) disappear when considering only the rural facet for development or evaluation.
Thus, one approach is to consider intersectional analysis for evaluation.
Steven LamPhD candidate University of Guelph
Hi Serge and all,
Yes, I try to integrate these themes into all evaluations. Clients are often very open to learning about ‘for whom’ their programs work. This information helps them know whether their program supports different groups of people.
In terms of environment, there tends to be a bit of hesitancy at first, as the link between program activities and environmental implications can be fuzzy. It could be that there are no implications. But asking about the environment provides a starting point for discussion.
As Silva noted, there have been many efforts to promote the measurement of social impacts. The UN system typically does this by using a human rights/gender equality lens (see UNEG Ethical Guidelines 2008 and UN-SWAP 2006). Many UN agencies also outline this need in their evaluation policies.
Similarly, there are many guidelines for mainstreaming environmental and climate change considerations into programs and policies (UNDP did a stocktake in 2010). UN agencies typically speak to this theme in evaluation guidance documents.
While it would help if TORs asked and budgeted for questions around social and environmental impacts of programming, I agree with Silva that we should advocate for them if these elements are not there.
A challenge I initially faced was, “well, how do we do this?” I’m currently finishing up my dissertation focused on answering this question. Examining previous evaluations of food security programs, I’m finding lots of evidence showing us how, why, and in what context we should integrate these themes.
We should engage with methodological developments from the literature and try them out. Ask questions such as: how do different groups experience this program? And how has climate change affected people's experiences? Share your process and learnings.
Evaluations could play in promoting equity and environmental sustainability, and we must. Steven
Abubakar Muhammad MokiCommissioner Policy Development and Capacity Building Office of the President-Cabinet Secretariat
1. Social and environmental criteria includes items on gender and equity, gender based violence, community resources use and management, partnerships and sustainability.
2. Our National Evaluation Standards for evaluation apply in areas of social and environment assessment. Copy attached.
3. Challenges are more in limited capacity to conduct social and environmental assessment, and limited available of data for use in social and environmental assessment.
4. The Uganda Evaluation Association promotes knowledge sharing on the matter.
Regarding the TORs, another practice is as follows :
1. The client determines the ToR based on what the client and his or her stakeholders require
2. The client looks for evaluators who can understand and execute the TOR
3. The evaluator who thinks can understand and execute the TOR indicates expression of interest to execute the TOR
4. When the evaluator is provided with chance to execute the TOR he or she writes an inception report showing what he or she has understood and how he or she will execute the TOR
5. If the client is convinced with the inception report that the evaluator understood the TOR and can execute the TOR then evaluator is given the assignment
6. The evaluator then executes the TOR based on the inception report agreement
7. There is no room for the evaluator to maneuver the TOR but understand it and demonstrate the understanding in the inception report.
Christine Kataike AbongPrincipal Data Analyst Local Government Finance Commission
Silva I do agree totally with you on providing a comment on TORs, in most cases if expectations are not well articulated by both parties, acceptance of evaluation reports may have issues. By implication, the commissioners' inception report should be thoroughly discussed to clear all misunderstandings of the assignment.
Richard TinsleyProfessor Emeritus Colorado State University
Silva, I can only endorse your concern that an evaluation as to include evaluating the TOR. I would say that is the most important part of an evaluation as only when you evaluate the TOR can you provide guidance for future projects to better serve the beneficiaries. Too often evaluations, particularly when internally, become propaganda tool to promote the TOR and the project. This can then become a major disservice to the intended beneficiaries as it will reinforce TOR and projects that are basically failures and provide no benefits. I think this is the case with the 40 years of reliance on Producer organization to funnel assistance to smallholder farmers. A close look will show these are really a scandal, attracting only a very small percent of the potential participants, and even then the members divert most of their business to other traders. Please review the following webpages:
Said Hassan OsmanAGRICULTURE OFFICER Agro-Coastal Research and development Organization
If I go though the proposed topic relating to the environment, it is very important to remind and make the environment progressive. The social environment is collectively all of the things that human have overlaid on the top surface on the earth.
The key environmental challenges are several and are related to deforestation, land degradation, increasing aridity and overgrazing, water scarcity, waste disposal, climate change and ecosystem services. And in my assessments these challenges cause negative trends to poverty, health, economy and ecological and human resilience.
If all the aspects I have mentioned above won't be maintained, the existence of human life will be highly at risk in the future.
Dr Said Hassan - Agronomist
Silva FerrettiFreelance consultant
Isha, you mention that "We, as evaluators, are obliged to execute the TORs duly"
My take is that, as evaluators, we should also question the TORs and negotiate them!
One of the main contribution we can offer is to propose alternatives ways to look at change, beyond the "cut and paste" TORs that are offered to us.
Some organizations and evaluation managers are actually quite open to this.
Others are not.... and, if it is the case, well... their problem.
I certainly would not want to work on an evaluation that I feel is missing the point from the start. :-)
See, as cyclo-activists say about car drivers... "you are not IN traffic, you ARE traffic".
As consultants, we do have a duty to resist TORs which we know are constraining learning and quality of work.
Another point... I was surprised by how the question was presented to us.
The question says "Major agencies and the UN in particular are considering how to integrate environmental and social impacts in their evaluations"
"Are considering"? Now... environmental concerns are (unfortunately) relatively new... but social ones, are they really?
We had all sort of cross cutting themes for ages (gender, disability and the like...).
I am really scared by how the "triple nexus" (a glorified take of the relief / development continuum - discussed for the past 2 decades) and "social impacts" are presented as if they were a new thing, requiring starting with a blank slate.
It would be healthier to highlight that these concerns are not at all new, otherwise we just risk going around in circles.
Best to all
Isha MirandaVisiting Lecturer and Independent Evaluator Independent Consultant
The issues are as follows:
Lack of horizontal assessment: the most common problem in TORs is a lack of horizontal assessment. There are various gaps in the evaluation process that could be filled but, instead, TORs often request to focus on their issue vertically rather than horizontally.
We, as evaluators, are obliged to execute the TORs duly and this causes:
- Lack in oversight: we should be able to use the important external elements that may have influenced the intervention to our advantage.
- Gap in observation tools, such as behavioral tools for stakeholders
- Gap in focus group discussions: i.e., lack of time and preparation
- Gaps in external factors to the programme.
There is also often inadequate understanding of the community, sites, or the intervention by the evaluation team: most assessors come into the issue with preconceived notions based on their previous experiences. This is one of the most serious errors we make. I strongly think that comparable programs may face different obstacles, methodologies, beneficiary profiles, and behaviors, and that time variables should be considered. We need to think about new difficulties. Do not use the same team over and again.
Here are some of the social and environmental aspects that should be included in evaluation:
Livelihoods need to be assessed based on the subject and local context, f.i crop production: evaluation needs to look into government or any other organization collaboration and cohesion, capacity (f.i. government logistics and services to the community: health facilities, education, agriculture centers for advocacy, product collections beneficiaries' capacity, collaborative capacity development) and new knowledge on climate changes, hazards mitigation, government subsidies (advises, fertilizers, seeds, technology etc.)
Contribution of stakeholders: Assessing community knowledge and actions related to their livelihoods. Gaps in consistency, technical knowledge, logistics (localize and new crop development technologies), product knowledge, market information, price variables, and market middleman contributions. In addition, there are local political interferences and impacts.
Stakeholders (Beneficiaries behaviors) - Consistency in production fields: health aspects (wellbeing risks, health deterioration due to epidemic, infectious illnesses, family and external abuses, education levels (formal, informal, and subjective), family nutritional level (adult and children).
Financial Management - Poverty: Reduction Traps, wins and losses, women have greater access to microfinance inside and between families. Credit should be targeted at low-income households, particularly women.
Nayeli AlmanzaM&E, Sustainable Landscapes Specialist Sr. Rainforest Alliance
In my organization, we are facing a challenge to integrate both aspects into our programmes' evaluations, because traditionally those areas were approached separatelly. We have found that we cannot separate the social impact from an internal or an external evaluation focused on the results of our interventions, including dimensions never considered before. For example, in the final stage of an Activity focused on producers strenghtening on better agricultural practices we are designing the terms of reference for an impact evaluation that will aim to include perspectives of the participation migrant work force in agricultural activities.
In my experience, the main challenges we are facing are related to some lack of experience conveying the social aspects within the rural approaches we traditionally face. We are looking for internal peers assessment in the first stage and later will look for external assessment with experts from sister organizations.
I think more reflection and work has to be done over the integration of both perspectives, but it is also a great oportunity area to collaborate with our peers.
Thank you for the space and the post, best regards