RE: The pervasive power of western evaluation culture: how and in what ways do you wrestle with ensuring evaluation is culturally appropriate and beneficial to those who legitimise development aid? | Eval Forward
Yes, I always do want to capture the involvement of women and different ethnicities/castes in all aspects of projects. So indeed, I do visit houses and other locations, not only official meetings, and try to establish what is happening behind the scenes. But usually when decisions are made in larger meetings in mixed communities it is the powerful in a community (and they are usually not the women or ethnic minorities) that control decision-making if there is no one present to facilitate participation of everyone.
That said, when doing a large scale evaluation there isn’t much scope for going very deep (as compared to a small project). We can’t visit all households in every community. In practice, I find that opinions vary from person to person or community to community also, so participation may not produce a similar view from all. If evaluating a project with national level funding and perhaps not local level staff or advisors, we rely on Government staff and the results framework for the indicators. Generally it is Government who decides on the priorities and approaches, and they may have different opinions from the local community or individuals.
There is also the question of whether it is reasonable to ‘interfere’ if local cultural practices are actually detrimental to women’s health or safety. To give a long example, but one that demonstrates the challenges - I work with a long term project in Nepal and the menstruation taboos in remote villages can be quite extreme, ranging from exclusion in chhaupadi huts, not being allowed to touch taps or use toilets, or eat nutritious food during menstruation or after childbirth. Bearing in mind the UN Right to sanitation and water, and the Nepali Constitution, these practices are not acceptable in law in Nepal, yet they continue. Hence, while some in the villages are opposed to interventions for change on the basis that these are traditional practices, we feel that it is justified to intervene to change practices. Something similar could be said about FGM practices in other countries. If I came in as an evaluator and spoke with some members of the community (including older women) they would probably say that this is the local practice and is perfectly acceptable. But others, particularly the young women, may object strongly. So whose voice is considered? (it becomes a bit like the old adage of a blind person describing an elephant, depending on which part they have felt!). And by forcing women to go for open defecation rather than use their toilet, the overall sanitation and hygiene of the community is impacted, yet people may not even see the connection. So it is quite challenging.
In the case of less extreme issues – such as countries where local women have traditionally not participated in working life outside of the home… if the indicators are anticipating more participation and there isn’t, then we end up having to score the project poorly on gender equality and women’s empowerment. This might be a question of targets being set too high, out of synch with local culture, but then the question remains of whether it is ok to try to make a change? There are often these days expectations that development project will achieve transformative change for women or ethnic minorities, people with disabilities or other disadvantaged groups. But that can be difficult to see if women are not participating in the meetings or training, leading activities or receiving benefits. How do we measure any potential change?
Anyway – enough rambling! I don’t have the solutions here, but I am aware that as evaluators we are stuck between communities/projects and the financiers, and trying to make an acceptable and fair assessment!
Good luck to all others with these conundrums! Pam
RE: The pervasive power of western evaluation culture: how and in what ways do you wrestle with ensuring evaluation is culturally appropriate and beneficial to those who legitimise development aid?
Hi Safieh and Eriasafu,
Yes, I always do want to capture the involvement of women and different ethnicities/castes in all aspects of projects. So indeed, I do visit houses and other locations, not only official meetings, and try to establish what is happening behind the scenes. But usually when decisions are made in larger meetings in mixed communities it is the powerful in a community (and they are usually not the women or ethnic minorities) that control decision-making if there is no one present to facilitate participation of everyone.
That said, when doing a large scale evaluation there isn’t much scope for going very deep (as compared to a small project). We can’t visit all households in every community. In practice, I find that opinions vary from person to person or community to community also, so participation may not produce a similar view from all. If evaluating a project with national level funding and perhaps not local level staff or advisors, we rely on Government staff and the results framework for the indicators. Generally it is Government who decides on the priorities and approaches, and they may have different opinions from the local community or individuals.
There is also the question of whether it is reasonable to ‘interfere’ if local cultural practices are actually detrimental to women’s health or safety. To give a long example, but one that demonstrates the challenges - I work with a long term project in Nepal and the menstruation taboos in remote villages can be quite extreme, ranging from exclusion in chhaupadi huts, not being allowed to touch taps or use toilets, or eat nutritious food during menstruation or after childbirth. Bearing in mind the UN Right to sanitation and water, and the Nepali Constitution, these practices are not acceptable in law in Nepal, yet they continue. Hence, while some in the villages are opposed to interventions for change on the basis that these are traditional practices, we feel that it is justified to intervene to change practices. Something similar could be said about FGM practices in other countries. If I came in as an evaluator and spoke with some members of the community (including older women) they would probably say that this is the local practice and is perfectly acceptable. But others, particularly the young women, may object strongly. So whose voice is considered? (it becomes a bit like the old adage of a blind person describing an elephant, depending on which part they have felt!). And by forcing women to go for open defecation rather than use their toilet, the overall sanitation and hygiene of the community is impacted, yet people may not even see the connection. So it is quite challenging.
In the case of less extreme issues – such as countries where local women have traditionally not participated in working life outside of the home… if the indicators are anticipating more participation and there isn’t, then we end up having to score the project poorly on gender equality and women’s empowerment. This might be a question of targets being set too high, out of synch with local culture, but then the question remains of whether it is ok to try to make a change? There are often these days expectations that development project will achieve transformative change for women or ethnic minorities, people with disabilities or other disadvantaged groups. But that can be difficult to see if women are not participating in the meetings or training, leading activities or receiving benefits. How do we measure any potential change?
Anyway – enough rambling! I don’t have the solutions here, but I am aware that as evaluators we are stuck between communities/projects and the financiers, and trying to make an acceptable and fair assessment!
Good luck to all others with these conundrums! Pam