The pervasive power of western evaluation culture: how and in what ways do you wrestle with ensuring evaluation is culturally appropriate and beneficial to those who legitimise development aid?

I just joined the Evalforward platform and found the contributions on this thread interesting, relatable (its a small world after all), and absolutely thought-provoking. Glad to hear and looking forward to the upcoming webinar to listen to more of the unpacking on this enriching topic. 

Dear all,

My thanks to all of you who spared time to contribute to the discussion. I hope you found it interesting to read about the insights and experiences of others. The discussion will now be closed, but given the number of rich and varying responses, EvalForward, in collaboration with EvalIndigenous have decided to set up a webinar event on Monday 24^th October at 14.00 (Rome time). On their behalf, I would greatly appreciate, if you have capacity, to participate and invite others in your own networks along as well.

John Ndjovu will make a presentation to provoke, what we hope, will an exciting opportunity to share and learn more about this extremely important issue.

With thanks in advance, and thanks again for contributing. We look forward to seeing you all there, so to say!

Daniel

Mr Mustapha Malki, PhD

Warm and respectful greetings to Mustapha Malki. I consider your contribution on this issue to be excellent and appropriate.

The various tools for analysing and understanding the target population and the resulting development actions must take into account the circumstances of the individuals as representatives of their culture.

Evaluation is the fundamental tool that collects and internalizes knowledge and experiences, presents a diagnosis, socially characterises a population and carries out a systematic analysis of the information. 

The structure of this practice allows it to generate an inner vision of the target population, based on the inherent contribution of nature itself and the contributions of the different tools applied.

Sincerely, Pedronel Lobaton Bedon. Consultant.

The discussion on cultural issues and evaluation, put forward on the EvalForward platform by Daniel Ticehurst, reminds me of an anecdote told in some African countries: the story of a hole on a main road, 3 km away, which was repaired every year and which reopened every year during the rainy season and therefore caused many serious accidents. And although the hospital was just 3 km away, the lack of ambulances made it even more difficult to deal with the problem. So the government decided to close the hole with concrete so that it would never open again... and open another hole right next to the hospital to solve... the problem of lack of ambulances.

By starting my contribution with this anecdote, I want to be a bit provocative in saying that trying to address cultural issues at the time of the evaluation is in fact a rather late debate in the socio-economic development process, especially if one agrees that any development action must be inherently participatory, and that the involvement of beneficiaries must be concretised quite early in two important initial stages of a development process (elaboration of the development problem, and identification of development/change objectives). Such a strategy is essential if beneficiaries are to be actively involved in the later stages of the development process (implementation/realization, monitoring, evaluation, hand-over, etc.).

Reading some of the contributions, many of which remained rather technical and methodological, I felt that in the early 1980s, when Robert Chambers' book "Rural Development' Putting The Last First" was published, many of the cultural aspects in development processes had been discussed from the outset. This identifies the six major prejudices (or biases) hindering the contact of outsiders with rural "reality" in general, and with deepest rural poverty, in particular.

The discussion on cultural aspects in the context of an evaluation is certainly interesting, but it cannot be denied that it remains a rather reductionist debate presenting the evaluation of a development action as if it were an independent island, whereas it is just one stage in the process of that development action and, moreover, intervenes at a moment when that process is quite advanced in time and space, and has perhaps reached a point of no return in its evolution.

Among the contributions on this theme, there are 3-4 that have - in my humble opinion - identified the problem of the frequent lack of consideration of cultural aspects not only during the evaluation, but also during the whole process of the development action: these are, according to the temporal order of the contributions.

(1) Njovu Tembo Njovu does not comment, rather he makes a case for the philosophy of development in the world being quite dominated by Western ethnocentrism; this remains an undeniable truth and I must say that I largely agree with the content of this argument. He states that "the global evaluation system is dominated by notions of investigation emanating from the Western, patriarchal, white-privileged Global North... and that national evaluation systems are controlled by donors". This refers to the seriousness with which the monitoring and evaluation of development actions are taken into consideration in Southern countries, which remain highly dependent on Western countries for financial resources to be allocated to monitoring and evaluation activities. And the iron law of Western countries in terms of development is that if they open budgets for Southern countries for development actions and related monitoring and evaluation activities, then part of these budgets must be used in the form of human resources from these donor countries. And then we fall back into the vicious circle of the expatriate who will formulate the development action with his Western blinders having only very ephemeral knowledge of the socio-cultural aspects of the environment in which the development action in question will evolve, and then what about our present debate on the cultural aspects and evaluation. It is therefore necessary to work, as Silva Ferretti mentions, to develop the necessary tools that can enable expatriates to integrate local knowledge into their development actions, giving more consideration to the fact that the beneficiaries can be both learners and teachers at the same time (dixit Njovu). This would lead to the decolonisation of development work in general, and evaluations in particular, as Ventura Mufume suggests, and avoid perpetuating "the culture of white supremacy" (dixit Silva Ferretti).

(2) Eriasafu Lubowa believes that "the challenge of cultural sensitivity would be partly resolved in the design phase by a thorough participatory stakeholder analysis, undertaken during the development of the results framework and indicators". For this person, the active participation of the main stakeholders of a development action, especially the beneficiaries, in the design phase, and in the monitoring of the implementation, should minimise/counteract the problems of cultural sensitivity that would arise during the evaluation. Thus, this statement adds weight to my argument that it is somewhat late to address cultural aspects at the time of evaluation; they certainly need to be addressed at the very beginning of the development action process, i.e. at the design stage.

(3) Ram Chandra Khanal states that "For various reasons, cultural issues are less represented in the evaluation design and subsequent phases. When designing an evaluation, most survey and observation methods and tools do not take into account context, space and time, and are mainly focused on outcomes and their associated indicators." This is the dilemma of evaluation, which quite often remains within the results framework of the development action, elaborated years before by a team of expatriates who did not have enough knowledge about the cultural and social environment in which the development action would evolve (example of a World Bank project on animal traction for land ploughing in Africa). And as Silva Ferretti says so well, if we accept that evaluation means "results, indicators", we may be killing the possibility of a cultural appropriation of the evaluation, and perhaps even of the development action as a whole, from the outset. This is especially true as many practitioners equate evaluation with the documentation of results and indicators, which can distract them from other avenues of analysis.

So, in the end, the evaluation stage alone cannot take responsibility for addressing the cultural aspects of the environment in which a development action is carried out, although it may be able to do so within the strict heuristic framework that an action research activity such as an evaluation can offer. And so the real discourse on these aspects needs to be planted in the design phase of that development action by allocating more space and active role to the beneficiaries and reducing the white supremacy of expatriates who can in no way hold the "absolute truth".

Mustapha Malki, PhD

Specialist in monitoring and evaluation

Freelance consultant

[Originally posted in French]

I fully agree with Ram's contribution. As opposed to the traditional concept and methodology of evaluation as a tool that collects and categorises information and obtains results, the evaluator must understand the local context and culture. This may lead to different findings and conclusions.

In other words, the evaluation must structure and shape the different situations it captures and analyses. And, automatically, the universe determines and consolidates the spirit of life of any culture in the world.

[Original contribution in Spanish]

Hi Safieh and Eriasafu,

Yes, I always do want to capture the involvement of women and different ethnicities/castes in all aspects of projects. So indeed, I do visit houses and other locations, not only official meetings, and try to establish what is happening behind the scenes. But usually when decisions are made in larger meetings in mixed communities  it is the powerful in a community (and they are usually not the women or ethnic minorities) that control decision-making if there is no one present to facilitate participation of everyone.

That said, when doing a large scale evaluation there isn’t much scope for going very deep (as compared to a small project). We can’t visit all households in every community. In practice, I find that opinions vary from person to person or community to community also, so participation may not produce a similar view from all. If evaluating a project with national level funding and perhaps not local level staff or advisors, we rely on Government staff and the results framework for the indicators. Generally it is Government who decides on the priorities and approaches, and they may have different opinions from the local community or individuals.

There is also the question of whether it is reasonable to ‘interfere’ if local cultural practices are actually detrimental to women’s health or safety. To give a long example, but one that demonstrates the challenges - I work with a long term project in Nepal and the menstruation taboos in remote villages can be quite extreme, ranging from exclusion in chhaupadi huts, not being allowed to touch taps or use toilets, or eat nutritious food during menstruation or after childbirth. Bearing in mind the UN Right to sanitation and water, and the Nepali Constitution, these practices are not acceptable in law in Nepal, yet they continue. Hence, while some in the villages are opposed to interventions for change on the basis that these are traditional practices, we feel that it is justified to intervene to change practices. Something similar could be said about FGM practices in other countries.  If I came in as an evaluator and spoke with some members of the community (including older women) they would probably say that this is the local practice and is perfectly acceptable. But others, particularly the young women, may object strongly. So whose voice is considered? (it becomes a bit like the old adage of a blind person describing an elephant, depending on which part they have felt!). And by forcing women to go for open defecation rather than use their toilet, the overall sanitation and hygiene of the community is impacted, yet people may not even see the connection. So it is quite challenging.

In the case of less extreme issues – such as countries where local women have traditionally not participated in working life outside of the home… if the indicators are anticipating more participation and there isn’t, then we end up having to score the project poorly on gender equality and women’s empowerment. This might be a question of targets being set too high, out of synch with local culture, but then the question remains of whether it is ok to try to make a change? There are often these days expectations that development project will achieve transformative change for women or ethnic minorities, people with disabilities or other disadvantaged groups. But that can be difficult to see if women are not participating in the meetings or training, leading activities or receiving benefits. How do we measure any potential change?

Anyway – enough rambling! I don’t have the solutions here, but I am aware that as evaluators we are stuck between communities/projects and the financiers, and trying to make an acceptable and fair assessment!

Good luck to all others with these conundrums! Pam

Hi,

This is indeed a great conversation. To me, evaluation must be more than culture sensitive. It should be culture driven. Any evaluation in a given context should follow the way of thinking of the culture within which the evaluand belongs to. 

Having experts from the community or involving the community members while using an « external » framework doesn’t make the evaluation culture sensitive.

We as evaluators have to learn how the culture generate and use knowledge. 

At 3DLab, we have started consulting with knowledgeable people in our community to develop, I should say, to uncover the traditional frameworks of knowledge generation.

I believe there is still more to do.

Rgds

Hi Ram,

This is a great experience to learn from. I agree with you that having a local expert who knows the local culture context is critical to ensuring culture-sensitive evaluation.

Regards, 

Eriasafu

Dear all,

Thanks for sharing the illuminating experience. I have a minor input based on my experience.

The value of culture in evaluation is less discussed and practiced discourse among the development researchers, professionals, academics and funders. Due to various reasons, cultural issues are less represented in the evaluation design and subsequent phases. While designing an undertaking evaluation, most of the inquiry and observation methods/tools do not consider the context -space and time and are mainly focused on the results and their associated indicators. This is more important when dealing with socio-developmental issues. The recent approach of using the Theory of Change, in principle, covers the wider spectrum of the context but an understanding of people and their practices (=culture) have not been an important part of the analysis. For example – in a group of people (let’s take the example of the ‘hill Bramin’ in Nepal) women do not generally say their husband’s name, do not shake hands (they have a different way of greetings when they meet people) and may not speak with a man frankly from outside. In addition, family roles of men and women are also determined by the social systems/culture on which they are used for generations which may be strange for people from the West. Some communities worship their god before they initiate project-related activities or complete the project tasks. There are cases where development interventions are designed without considering the cultural aspects (such as demolishing temples or sacred or religious places of certain communities to construct a road that directly affects their culture). These are some examples. In this case, an evaluator without understanding the local context and culture – s/he might understand different ways and overall evaluation findings might be different. I feel having a local expert (who knows the culture) and a respectful conversation with the communities are two strategies I have used in my evaluations.

Have a lovely weekend.

Best regards, 

Ram

Hi Safieh,

I was just sharing some thoughts using the term "critical challenges" to express my point but I don’t have a link. 

Regards, 

Eriasafu

Hi Pam,

I wonder how/if you went about getting the womenfolk decision-making captured in this evaluation process, which is what I imagine you were trying to measure and convey?

With that said I am not sure if decisions are made simply by men who are going outdoors for meetings or if the actual discussion-making process is more likely done by the women (too?) prior to the meeting conducted outdoors.

Eriasafu, could you please send a link about critical challenges that you has been done and you think might be helpful?

Thank you all so much for this rich discussion,

Safieh.

Greetings form Mexico to the Evalforward community! 

In rural development, conservation and forest management projects, we have faced the challenge of giving the appropriate dimension to the worldview of indigenous peoples. In Mexico there are more than 68 native peoples and, as in other countries, they are mostly present in rural areas.

In the different projects I have been involved in, we have tried to implement Free Prior and Informed Consent schemes with indigenous peoples throughout the management cycle of rural development projects. We have developed simple tools that have allowed us to propose projects, report on their development and appropriation, carry out process and impact evaluations, as well as the closure and exit strategy of our presence in the territories.

In particular, in GEF projects, this is a requirement to be fulfilled whenever the projects have an impact on the territories of indigenous peoples. I would like to share with you a very useful course on this subject at: https://www.fao.org/in-action/capacitacion-politicas-publicas/cursos/ver/fr/c/1397095/ (in Spanish) 

In addition, in the organisation where I work, we have the Dedicated Mechanism Specifically for Indigenous Peoples and Local Communities (Dedicated Mechanism) project, which is a special project of the Forest Investment Programme. This programme supports the efforts of developing countries to Reduce Emissions from Deforestation and Forest Degradation (REDD+) through the full and effective participation of indigenous peoples and local communities in activities that contribute to REDD+ by promoting sustainable productive sub-projects that foster economic development and the exchange of knowledge to strengthen their capacities in the sustainable use of their territories. In this project, monitoring schemes have been developed for the systematisation of information and evaluation to verify the progress of activities in the territories. The development of the tools was very sensitive to the inclusion and context of the participants, so there are some manuals with guidelines and formats that allow the collection of information, even in places with limited internet access. I am sharing the link to the library in case you find it useful: https://www.mde-mexico.org/biblioteca/ (in Spanish) 

[Contribution translated from Spanish]

If we accept that evaluation means "results, indicators" we might have killed the possibility of cultural appropriation from the start.

"Evaluation" means different things for different people. Making it equate to  "documenting results and indicators" undermines many other alternatives.

As in feminist evaluation (which is not only about "gender" but about rethinking approaches to make them intersectionally inclusive), we should question what an evaluation is for, what ways of seeing change it embraces. Beyond results there are processes, principles, worldviews. The moment you are discussing with local actors: "what matters for you in looking at change?" you are already working to make it culturally appropriate. If it is just about "defining indicators", sorry but this is a non-starter.

In fact, such a situation is a critical challenge in evaluation. I think the cultural sensitivity challenge would be partly resolved at the design phase by doing a thorough participatory stakeholder analysis, inducing during the framing of the results framework and indicators. Thus, full participation of the key stakeholders in design and monitoring should minimize or counteract cultural sensitivity concerns that would come up during evaluation.  I also think that to counteract such issues, prior to the evaluation itself, evaluability assessments also looking into cultural issues would reveal the possibility of such concerns happening.

Regards, 

Eriasafu

Hi,

This is not so much a ‘how to’ but something I struggle with at times, when doing an evaluation and trying to track gender outputs/outcomes. How far do we say that issues of gender equality and women’s empowerment should be applied in a project, and therefore to evaluate it accordingly?

I was accused once by a local evaluation team member of not respecting local cultural norms. The accusation was that in that culture, women don’t participate in meetings, work outside the home, etc. and only men were the likely stakeholders - and by asking questions about this, including from that team member (in what I felt was a respectful manner), I was not being culturally appropriate.

Personally I feel that if the project stakeholders have agreed activities should be done in a certain way, and especially if there are national/local gender strategies or commitments, then it is fair to discuss lack of compliance in the evaluation. Otherwise we are ignoring commitments to improve the status of women and just supporting ‘business as usual’, even if it isn’t local cultural practice. Naturally we need to consult with project staff or local sources to ensure we understand the issues and aren’t blundering into a highly sensitive issue, or ignoring a local method of Gender Equality and Women’s Empowerment (GEWE). It might be that the local stakeholders haven’t been fully consulted or informed about the need to involve women (or ethnic minorities or castes, or whatever the specific sensitivity is), and that is another issue for the evaluation. But I don’t feel that as evaluators, we should simply ignore the way things are done.

Best wishes, Pam

Thank you very much Ana Urgoiti for highlighting the EvalIndigenous guidelines that I use in my work.

As a founding member of the global EvalIndigenous network (https://evalpartners.org/evalnetworks/evalindigenous/), I am helping indigenous communities and evaluators across the world to have their voices heard in the social and economic development processes and get their human rights respected. I focus on the bottom-up approach of working more with people at the grassroot level to empower them with evaluation skills. 

For democracy to be sustained, peace to be enjoyed across the world and the successful implementation of the SDG, there is need for an all-inclusive approach in evaluation. I and my fellow indigenous evaluators are increasingly advocating to other evaluators and development institutions that indigenous communities can be both learners and teachers. We work with indigenous communities that are on the margins of the global evaluation system.

No one should be left behind in the development and evaluation processes. High net worth communities or global north do not have a monopoly on knowledge and solutions to global challenges. The global north has the highest consumers of global products and contributors to climate change. Many of the challenges being faced by indigenous communities around the world can be placed squarely on the global north. The evaluation system is dominated by Global North-western-patriarchal-white-privileged notions of inquiry (as evidenced, for example, in ‘Evaluation Roots’). It has for a long time been complacent with the failed democracy and paradigm of development that has brought the planet to the brink of catastrophe. We are a global village, and the majority is fighting for an end to the systematic racism and other -isms of leading global developmental systems. We need change in the way we practice evaluation. Indigenous knowledge gave us the principles of relating to each other and with our environment. Much of the SDGs are found in the indigenous communities’ way of life. The African indigenous way of life is based on the Ubuntu philosophy that teaches us to love each other (I exist because you exist) and our planet.

Here in Zambia, I have noted that there are many communities where they are development interventions by NGOs and the government, but the locals or beneficiaries are not involved in the designs, implementations, and evaluations of these interventions. We have had slavery, colonialism and now political independence but the majority of people still leave in the era of slavery. The Royal Establishments and elites (including evaluators) seem complacent in the paradigm of under development. Democracy has been captured by a few and evaluators are part of the justification system. May I remind those who were hailing South Africa as a champion of M&E in Africa that it was much like the veneer of Christianity. Indeed, there was an evaluation department in the Office of the President. However, did that change Jacob Zuma from treating South Africa and his cronies that captured the government businesses from treating the country like a personal chiefdom? Across, in Zimbabwe with the highest number of educated people in Africa was brought to economic ruins by the best educated President in the world, Robert Mugabe. Here in Zambia, we set up a Ministry of National Development with an evaluation wing. The ministry was scrapped in late 2021 and evaluation shoved to the dark corner of the Ministry of Finance.

The evaluators dash in for a few days wearing the local wrappers (chitenge), to shake hands with the elders and nod politely in what they believe is a cultural response way and tick their evaluation checklists. Even Africans that take such field visits tend to behave like sociologists from the land of ice and where the sun does not go to sleep for six months. They let project officers parade to them beneficiaries that were selected by the officers to talk well about the projects. They talk with civil servants about good governance and implementing this or that global agenda and are taken to places where they can take selfies to show that they were in Zambia and raced a cheetah but have no time to walk out of the airconditioned offices and cars to the villages to ask what the ordinary citizen thinks about all those indicators on good governance and implementation of the SDGs by their government. Therefore, we tend to have evaluations and researches that are very much detached from the realities of the majority of people in the communities. The reports are graphically and statistically impressive but worth less than a dry cob of shelled maize or village toilet tissue to an ordinary citizen.

We have indigenous evaluation systems. However, the current evaluation practices totally ignore them. Those who are aware about them and would want them used have no power to insist on their use. The development of the national evaluation system is donor driven. Though the 2005 Paris Declaration on Aid effectiveness has a principle of transferring ownership to the locals, not much has changed in the past two decades. African states and the African Union are too reliant on external funding for M&E. Believing it’s a ritual to be practiced for show to external institutions that there is good governance. They are not champions for the Made-in-Africa Evaluation Methodologies. VOPEs on the African continent talk about the local methodologies in conference and academic spaces but cannot defend to use them in practice. They mainly struggle with the challenge of sustaining their daily operations. African VOPEs do not survive on mobilising resources for their operations from members or local philanthropists. They cannot even afford to ensure that evaluators conduct evaluations of high quality and protect the interests of all stakeholders.

Therefore, EvalIndigenous has prepared a short guide to help Indigenous and tribal communities that may not be able to effectively participate in the developmental and evaluation processes in their communities. It is not my document but a product of EvalIndigenous. These are questions that they may want to ask institutions and evaluators that come to work with them. These ten questions can be found on the below links: 

https://evalpartners.org/evalnetworks/evalindigenous/

https://evalpartners.org/sites/default/files/10Qs4evaluatorsA4.pdf

BACKGROUND OF EVALINDIGENOUS

EvalIndigenous is a multi-stakeholder partnership formed as part of EvalPartners and built on the foundation of the knowledge and expertise of indigenous peoples around the world. EvalIndigenous seeks to bring awareness to, include, and celebrate the cultural traditions and values, languages, legal/political governance practices, and ways of life of Indigenous peoples wherever they live. Our focus is to ensure that policies and evaluation practices for Indigenous peoples are based on equity, fairness and justice. The United Nations Declaration on the Rights of Indigenous Peoples (2007) frames our work, moving from an evidence-based focus towards a shared global understanding of good practice for Indigenous peoples and our rights within the field of evaluation. Part 1 of Article 31 of UNDRIP states,

Indigenous peoples have the right to maintain, control, protect and develop their cultural heritage, traditional knowledge, and traditional cultural expressions, as well as the manifestations of their sciences, technologies and cultures... They also have the right to maintain, control, protect and develop their intellectual property over such cultural heritage, traditional knowledge, and traditional cultural expressions.

 WHY EVALINDIGENOUS

The programs implemented in Indigenous communities are evaluated using transient evaluators. They are unfamiliar with the cultural realities, protocols and community contexts.  It is now time for Indigenous peoples to reclaim their evaluative frameworks and processes in order to strengthen and make evaluation more authentic in both Indigenous and nonindigenous communities. It is time to recognise what has been done to bring both worldviews together, to provide an authentic approach to cultural responsiveness.

John T. Njovu

Atlas Project Coordinator - Africa and Asia 
Member - Global Indigenous Evaluation Task-force (EvalIndigenous)
Member - UNEG - EvalPartners' 2021 Global Forum Organising Taskforce 
Honorary Member - Zambia Monitoring and Evaluation Association (ZaMEA)
Life Member - The International Development Evaluation Association (IDEAS) 
Former Treasurer - African Evaluation Association (AfrEA)
Former Chairperson - The Governance Monitoring and Evaluation Committee of the Government of Zambia

Hello...

again I am not really adding here a practical lesson, sorry...

but I just found quoted this recent paper by USAID, which might be of interest to the people following this thread.

https://usaidlearninglab.org/resources/report-integrating-local-knowledge-development-practice

USAID’s Agency Knowledge Management and Organizational Learning (KMOL) function in the Bureau for Policy, Planning and Learning, Office of Learning, Evaluation and Research (PPL/LER) facilitated conversations with development practitioners to learn how development organizations are integrating local knowledge into their programs. The report explores three aspects of this topic: Leveraging Best Practices, Addressing Challenges, and Achieving Best Outcomes.

As an African evaluator, born and raised in a country that went through colonial dominance for 500 years, not only do I support the decolonization of evaluations, the Made in Africa evaluation movement and other initiatives but I also contend with and try to correct a culturally intriguing “sin” that most evaluations commit without knowing.

When designing questionnaires to capture respondents’ demographies, evaluations do often ask whether or not the person belongs to an “indigenous” group. Unfortunately in the African countries where Portuguese colonialism flourished for over half-a- century namely, Mozambique, Angola, Cape Verde, Guinea Bissau and Sao Tome & Principe, the colonial system used a divide and rule approach, among other strategies and words such as “indigenous”, “assimilados” and others, were employed to legitimize a cultural caste system where the native black population was sub-divided into those categories where the “indigenous” was equivalent to “uncivilized” and the assimilado, as who acquired the Portuguese citizenship through a long process of acculturation, subservience and brainwashing, was closer to the system than the non-assimilados. Assimilates were thus used to tax their fellow blacks, using coercive measures including physical abuse, teaching, and playing religious roles.

As the world today grapples with social problems such as racism, it is worth bringing to everyone’s awareness that the employment of the word “indigenous”, in evaluations, in Portuguese speaking African countries resuscitates the legacy of dehumanizing class system the Africans have gradually been burying for generations.  Alternatively, evaluations would do less harm adopting categories such as "minority group" and others.

Sincerely,

Ventura Mufume

Freelance evaluation consultant

Mozambique

Dear all,

I had this message drafted a few days ago, was not able to send it, but I think the moment – after our colleague ask for concrete experiences – is very appropriate to share. So here are some lessons from personal experience.

These I have learned during 4 intensive months of work in northern Uganda with refugee communities (mostly from South Sudan) to develop and administer a survey – a few years ago, while working as an independent consultant.

[For a bit of context, this process involved the community in all steps of the survey from design/piloting, translation to 4 languages (by community members), selection and training of non-professional enumerators (community members), application of survey and feedback/participatory analysis.]

1. You also have a culture, yours is also ‘a culture’. At the eyes of the other, you are the strange. I particularly ‘discovered’ myself as Latin-American during these months in Uganda (note: Brazilians don’t really identify with the Latin-American stereotypes and neither with the lable of ‘latino’- even if we are seen as such and in reality share so much in terms of culture with all other Latin-Americans. Please also notice that, even if Latin-American and having lived most of my life in Brasil, I am a white middle-class woman, that had access to higher level education and whose culture is very close to western/European – this is where I speak from, and how I am perceived).   

2. Be prepared to recognize you made a mistake and act in case something happens. In one situation I felt the need to go at the houses of each of my team members (around 12 in total) and have an individual conversation. This was after one very difficult meeting, in the middle of a lot of stress and time pressure. It was all sorted out, but took a lot of energy to make sure all was kept on track and the trust (built over weeks of work and intensive dedication) was not broken.  

3. Be open (be curious!), be patient, and always respectful. Have some reality checks. I had talks with my driver that helped me to understand the culture in which I was immersed. And if necessary, take a day or two off to breathe in the middle of culturally difficult situations – and talk to experienced colleagues. Better to step off for a couple days than having to fix things later.  

4. And a lesson from something that went very well: be mindful and respectful with dress codes. Women were open to receive me in their houses and talk to me because I dressed respectfully – they literally told me that they appreciated that I did not wear pants, but longer skirts and modest blouse. (I believe I was able through this and other attitudes to show respect and build trust. After one Focus Group Discussion, women sang for me and ‘baptized me’ with a name in their language.)  

I hope this adds a bit of concreteness to the discussion J

Kind regards

Emilia

Thank you to all who have contributed to the discussion. Many of you point to the importance of culturally appropriate behaviours, and these are associated with compelling reasons. Some provide telling examples of western culture and how some of their institutions remain stuck despite being aware of the consequences in needing to change. However, too few reveal specific instances of experiences in how, either as commissioners or evaluators, they have sought to be culturally appropriate and/or how they have not, and with what consequence.

Therefore, we would welcome any ‘personal’ experiences that respond more explicitly to the question: what lessons or experiences – successes, challenges, failures - have you had in trying to ensure evaluations adequately prioritise indigenous knowledge, values and practices?

Many thanks. 

This is an interesting topic. When we understand culture and philosophy of each other. You will get this primary goal. In fact, there are always similarity and differences in this topic. If you respect this difference. You will find proper cultural evaluation and deliver/get maximum mutual benefits for both sides. This could be the best way to change our thoughts.

Greetings!

Purpose of this note is to provide a frame of reference, which would facilitate an evaluator/monitoring expert to take into account the relevant cultural elements that ought to be incorporated into the evaluating or monitoring processes.

However, this is not as straightforward as it may seem, for evaluation can be divided into three distinct levels on whom cultural factors may have very different types of impact.

Let us begin with the top most level where the goal of a plan/project/programme is determined. Here, assuming the utility of achieving that goal is demonstrable, one is concerned with the question whether that goal is culturally neutral or not. As an extreme example, one may cite primary education for all children, but in some cultures, its inclusion of girls may lead to unforeseen results.

At the second level, one faces the challenge whether the strategic means used to achieve one's goal are culturally acceptable. For example, a decision to farm out the execution of a project to a highly technically advanced source may attain its objective for the moment, but the local beneficiaries may not be able to maintain it in the long run owing not only to the differences in technical competence and resources, but also because of divergent work ethics.

Finally, the actual operational methodology used in the field may contain culturally objectionable elements. Sometimes, it may not be possible to avoid them except by resorting to some other and less efficient methods.

Best wishes!

Lal Manavado.

Dear Naser,

thank you so much for your good example and clarification, actually the examples you talk about are exactly my views/ my concerns and it is just a consideration to initiate any step for evaluation.

Let me put my comments as follows: I was a country Research Collector when I made a data collection for determination of baseline study and the realities observed  are:

(1). Design the tool and test before starting any activities;

(2). Use of local staff who understand the local context and we just make a translation of the tools in our local language with purpose to avoid any misunderstanding/bias;

(3). We have collected a lot of information and targeted many groups of population to participate in the research;

(4). We have a serious issue to be considered in our research  and we must understand the contact of our respondents vis a vis for example meeting with women violated and sexual workers as are part of our research.

Regards,

My expression and vision in terms of lived experience within the spirit of coexistence of the indigenous communities is the following: we can respectfully say that the evaluation is culturally appropriate if it manages to generate a presentation as a result of assuming, agreeing on the lessons that are extracted and at the same time are embodied as experiences of having assumed the different challenges and if finally ends up prioritizing and endorsing in a justified and concerted manner with the environment and the cultural context and their pillars of knowledge, their respective values and lessons learned as indigenous practices.

Dears

In my opinion any evaluation not considering the  context in general, not only cultural, will not be an evaluation in essence. 

From our experience in women empowerment interventions timing of any data collection event should be agreed with women otherwise no one shows up. We invite more people than needed to achieve the target of participants. in the latest experience we recruited external evaluation expert for the Women Empowerment project. He thought of the livelihood assessment tool, where there are questions like: how many days did you sleep without food? When was the last time you ate meat? ...etc. He could not fill any of the questionnaires, therefore we decided to cancel this section.  Why? As they are poor this was very sensitive for them. 

Another story: Trying to understand a success story a PR officer visited and interviewed a beneficiary, the second day she thought of a photo at specific time, so she called the beneficiary and said "Please can you ask one of your kids to take a photo for...", But the lady/ beneficiary was old single with no family but supporting the extended family. This also might happen if you ask about a father or a mother or husband to an orphan or widow. 

So  to respect culture, values :  

1-  Design tool

2- Test the tool    

3- Use local staff who know the context

4- Explain to the evaluation team any sensitive issues related to specific communities or specific people

5- Collect as much data as possible before interviewing people.

Regards

Naser Qadous

Palestine

I just accessed an interesting article / website, highlighting characteristics of a white supremacy culture.

Evaluators do risk to - willingly or unwillingly - to embrace them.

(and the sector really pushes us to do so).

So... these are not lessons or experiences.

But a useful checklist to break the issue down and harvest practices.

The article is on:

https://www.whitesupremacyculture.info/

And I found it mentioned here

https://aidnography.blogspot.com/2022/09/development-ict4d-digital-communication-academia-link-review-455.html

Dear Ana,

Many thanks for responding, for sharing John’s 10 questions and his email address. 
 

I wonder: has anyone else come across them or questions similar to them? And, if so, have you been asked them? If not, have you asked them of yourself in designing an evaluation? 
 

It seems to me, responses to them could usefully inform the design of evaluation and/or help teams adequately prepare. That is, rather than waiting for community members to ask them on ‘arrival’, so to say.

Does not doing so run the risk of potentially de-railing the process and wasting community members’ time?
 

What do you or others think?  

Many thanks again Ana and will connect with John to find out more.

With best wishes,

Daniel 

Hi Daniel and Pedronel,

I remember an intervention from John T. Njovu member of EvalIndigenous working in Zambia around "10 Questions Indigenous Communities Should Ask Evaluators", based on how  Indigenous knowledge has often been marginalised by colonisation or by development and in the idea that just as evaluators are asking Indigenous peoples to share their knowledge, evaluators should also be prepared to be both learners and teachers – sharing knowledge about evaluation with communities and enabling people to be part of the evaluation so they learn by doing

Here you have them, I really liked that approach! 

1. Who do you know in this community? Do the evaluators have any relationships with people in your community? Have they come with someone who can guide them in behaving respectfully? Is there someone in their team from your community?

2. What do you know about this community? Have the evaluators done ‘homework’ to get to know your community’s cultural, historical, political context? Do they understand your worldview and how you live? Do they know what it means to be Indigenous?

3. Where are you from? Are the evaluators willing to introduce themselves and share about themselves? Do they have an understanding of how their own background might differ from and be similar to the communities?

4. Do you speak our language? How will the evaluators communicate and understand you? Do they know your language or are they reliant on interpreters? What language will the evaluation be conducted in and the evaluation report be written in?

5. What do you know about the history of the initiative? Do the evaluators know how the initiative came to be in your community and whether or not the community needed or wanted it? Do they know about the decision-making behind the initiative? If not, are they curious?

6. What relationship will you have with us during this evaluation? Are the evaluators interested in working alongside community advisors? Will they be spending time in the community, both formally (for the evaluation) and informally (getting to know the community)?

7. Will people in our community get work in the evaluation? Will funding for the evaluation be spent locally, employing community members to help collect evaluation information? How will the community members be compensated for the time spent on this evaluation?

8. Will we have a say in the design of the evaluation? Has the evaluation design already been decided, or will the community be able to have input into the evaluation design, the methods used, the people who are talked to, and the way information is collected?

9. Who will be analysing the evaluation findings and writing the report? Will the evaluators collaborate with the community to analyse and report on the evaluation findings? Will community members be involved in dissemination activities (e.g., conferences, funder meetings)?

10.How will you support our use of the evaluation findings? Will the evaluation findings be available to the community? Will the community be supported to use the findings to improve the initiative and/or to advocate for change (including more funding)

If someone ever try this, I am sure he would love to receive feedback!

Here you have his contact: njovujt@gmail.com

Ana 

Dear Pedronel,

Hi and thank you for responding. And I completely agree with how evaluations and evaluators are challenged in the way you describe. Failing to overcome these risks an exclusion of more diverse streams of knowledge and local ways of making change can be especially hampered by a fixation on a pre-ordained finishing line rather than flowing with a generative process at the speed of seasons. 

What are the challenges you mention in relation to learning about and prioritising indigenous knowledge?; and how do you think these can be overcome?

Best wishes and thank you again,

Daniel 

Evaluations and evaluators face the challenge of trying to understand and adequately prioritise indigenous knowledge, values and practices. This will enable them to achieve successful results.  

Indigenous people protect and conserve 80% of global biodiversity. Their sustainable management of biodiversity is fundamental to food security, nutrition, health and development.   

Indigenous principles and values are based on a DEEP AND HARMONIC RELATIONSHIP WITH NATURE. Indigenous people advocate caring for Mother Earth –our home– as it provides the guidelines on good living and everything required to sustain the spirit of life and coexistence. They also promote THE CONNECTION WITH THE UNIVERSE, THE RESPECT FOR ALL LIVING BEINGS, THE IMPORTANCE OF ANCESTRAL KNOWLEDGE, THE RESPECT FOR THE ELDERLY AND CHILDREN and, finally, THE COMMUNAL SENSE OF LIFE.

[Translated from Spanish]