Disability inclusion in evaluation

Thanks dear Judith for sharing , human rights give priority to disabled people in any circumstance.

Regards.

Norbert TCHOUAFFE, PhD
Pan-African institute for Development, Cameroon
Author of Tchouaffé's theory of change 
Latest book: Science-Policy-Data interface for responding to COVID-19 in Cameroon

Thank you very much for the well consolidate key points from the discussion, which are stepping stones for future improvement measures. Thanks to all who have contributed to the discussion. God bless you all!

With  regards,

Hadera

Very interesting thread, thank you to all who contributed. 

Dear EvalForward members,

Thank you indeed for your participation in the just concluded discussion on disability inclusion in evaluation.  A special thanks to members who shared insights, made comments, or provided useful materials for further reading on the topic and to the EvalForward team for a great coordination always.

From the contributions and discussions, it is evident that the concept of inclusion of people with disability and other marginalized groups in evaluations is important, but still new or rather is given low attention.

Several limitations to inclusion exist, for example, lack of appropriate communication strategy, lack of inclusive tools and appropriate approaches, and limited awareness creation are some of the barriers to widening the inclusion gap. Evaluators ought to bring everyone on board, it is an opportunity to be heard, People with disability should be entitled to every benefit in support of human diversity in society and in the development process, most importantly inclusion needs to go beyond supporting people with disabilities to involving them in the actual practice of evaluations. Disability comes in so many forms and needs different approaches, including in evaluation, requiring capacity building of project/program teams and M&E staff, to better support inclusion of people with disabilities in evaluations.

Very Best,

Judith

Dear Members/Colleagues,

My EvalForward contribution,

First of all, humanity does not exempt people with physical disabilities from contributing to development programmes or projects, irrespective of who they are in the society. If you are an asset you are a great contributor to advancement of knowledge in human development. People with disability should be entitled to every benefit in support of human diversity in a society. In other words, disability inclusion and practices in evaluation should also be encouraged, given priority and value in the society. Why? Because policies, evaluation are made by mental knowledge, not physical power which gives inclusiveness. Meanwhile;

1. Evaluation is not a physical exertion of energy, but mental exertion of energy which includes disability inclusiveness contribution to the evaluation development world.

2. Evaluation an be strategically practiced with the assistance of technological gadgets on their parts.

3. Physically challenged people can come up with their invention that will help them in evaluation system, such as technological inventions and policies that will strengthen their inclusiveness.

4. According to UN support for their participation, if the needs are given to support their physical strength and energy, people with disabilities are as good as every other human in driving the research outcome of evaluation implementation.

Thank you.

Esosa.

Dear Judith,

Thanks for tabling this important topic.

Just wanted to share some resources that you may find interesting to address some of the questions you raise, which we all try to grapple with the best we can these days. This is a recording of an exchange session that the Office of Evaluation of WFP organized last May, to discuss practical strategies to address some of the challenges you raise, related to making evaluations inclusive.

Sway (office.com)

Hope this brings useful insights.

Best,

Aurelie

I think this is an important conversation, the issue of disability inclusion and by extension other marginalized groups in evaluation. No doubt that inclusion will bring everyone on board and an opportunity to be heard. 

I think one of the barriers to disability inclusion and others such as the LGBTIQ is the lack of appropriate communication strategy. In most societies in Africa for example, LGBTIQs do not accept being called gay, lesbian, bisexual, etc. same as a person with disability would not accept being disabled for fear of being stigmatized. Using languages that are not offensive in drafting our tools would impact their inclusion and participation. Spending more time identifying, and strategizing an appropriate method including defining the right language would ensure active participation and inclusion of everyone in the evaluation. Reflecting on the issue of visualization, could be an important way to draft the tool, without asking questions like, are you a disabled person, you are a gay? Alternatively, which of these (emoji) appropriately describes you?

In short, to enhance disability inclusions, we need take a deeper reflection to understand the contexts in which we are conducting the evaluations. 

My comment to the ongoing discussion

1: Are evaluations respecting inclusivity? 
(Do you consider the concept of inclusive evaluation always respected during design and actual evaluations of projects and programs)?

• I do not consider the concept of inclusive evaluation always respected during design and- actual evaluations of projects and programs. Reasons include:

(i) Inclusivity is not mostly mainstreamed within the projects/programs to be evaluated right from their inception and preparation stage;
(ii) Evaluation ToRs rarely include/attach importance of the need for inclusivity, due to : (a) to less understanding/awareness of the diverse group of people’s disabilities; and (b) budget and time limitation, given the often short time and budget allocated to projects/programs to be evaluated, this is regardless of the size of the project/programs.

For example, there are cases that five and more year projects, with large geographic coverage are planned to be evaluated within say 10 days or so, with a lot of rush. This happens both for mid and end term evaluations. In such cases let alone to rightly include relevant people with disabilities (which include their care takers and or responsible organizations), even adequate participatory discussions
and decision taking with people without disabilities are compromised.

2: What hinders full involvement of people with disabilities in projects and program evaluations?
Please share personal experiences.

• Lack/inadequate care takers/ representative organizations which voice for people with disability to fully involve in evaluations;
• Lack/inadequate tools for ease of communication with evaluators;
• Lack/inadequate awareness of people with disabilities of their right to be involved in evaluations.

3: How could Evaluators mitigate the risks associated with absence of inclusive evaluations specifically for people living with disabilities in our communities?

• Improve their understanding of the importance of the inclusion of people living with disabilities in evaluations;
• Be committed to be voice for the voice less/ the people with disabilities;
• Sustained advocacy with this regard and sustained informed move to the designing of guidelines for the inclusion of people living with disabilities in evaluations.

Dear Umi,

Thank you indeed for sharing these useful materials on disability inclusion in M&E, the Capacity Building modules on disability inclusion will definitely contribute to the existing body of knowledge and the gaps that we look to close in this area.

Encouraging EvalForward colleagues to make reference.

Looking forward to more contributions to this topic.

Thank you,

Judith

Dear all,

Sending you our recent work to develop training Modules on Disability Inclusion in Monitoring and Evaluation (M&E) for the United Nations M&E Staff/Focal Point in Indonesia. This module is developed by MONEV Studio and the Washington Group and has been delivered to all UN M&E staff/focal points in Indonesia in December last year. Enjoy the read https://indonesia.un.org/en/234468-training-modules-capacity-building-disability-inclusion-united-nations-monitoring-and
 

Best wishes,

Umi Hanik 

I am certainly no expert on this topic, but it is something that I have struggled with (and I look at it here as both an implementer and as an external evaluator). The difficulty I find is that inclusion of people with disabilities is only one issue among many in implementation and evaluation. For instance, bilaterals or the UN are starting to talk about disability, but we still don’t see much evidence of inclusion in practice (other than perhaps individuals receiving support in some form of income generation). Even with the interest to work more on the topic, there isn’t much inclusion to actually evaluate! It is early days as yet and there are so many topics that project teams are asked to look at, including caste, ethnicity, gender/sex, youth/age, poverty and remoteness, in addition to the actual project thematic topic itself (eg. agriculture, water supply, etc).

In our Finnish (and EU) funded projects in Nepal, it was feasible to take some actions on implementation (and staff training) as we had the hands-on team in place at local level – but I still question how big a contribution we could make. I feel like it is often easier to work on the topic as an NGO, where you are more able to give individual attention, rather than a big project working at scale.

The greatest difficulty in addressing this, in my opinion, is that disability comes in so many forms and needs different approaches, including in evaluation. Generally speaking, we can bring together groups that are differentiated by sex; or caste; or ethnicity – and ask for quotas, targeted activities, focus group discussions, etc. Of course, not everyone in that group of women, for instance, will have the same interests, but they will have the opportunity to participate. But an activity that suits a blind person may not suit someone with a mental disability. And I have often found that people with physical disabilities – especially following an injury – often don’t identify themselves as being a person with a disability (PWD). I have also heard that there is even conflict within groups sometimes – with for instance, Brahmin caste PWDs being upset about being included in a group with Dalit PWDs, though I haven’t seen this personally. I recently was involved in a GALS training process in Tanzania, and we required participation by people with disabilities as well as religious leaders, youth groups, entrepreneurs, etc. But it was clear that while the people with physical disabilities could participate well, those with mental disabilities struggled.

There are also practical barriers to inclusive evaluations. Generally, the budget for evaluations is not large and I really doubt that many development partners will be keen to pay more for projects that are not specifically disability focused. While I agree that there are steps that can be taken to be more inclusive, they are potentially a lot more expensive in terms of time and money. And as I am sure all evaluators can recognise, we often carry out flying visits during evaluations (even getting into more remote areas away from the road is difficult), and may struggle to get a representative selection of the community for focus groups, etc. It may not be possible in the time available to visit the homes of PWD, nor for them to physically reach the meeting area. Access for evaluators or staff with disabilities is also problematic in rural areas. We had some experiences in Nepal, for instance, taking a blind interpreter to the field, but it was pretty challenging, due to the difficulties with access. The young woman got terribly car sick on the winding roads, exacerbated by not being able to look forwards on the road, etc.  – and needed a lot of help trekking uphill. More like a good example to us and the community, rather than something that we could replicate easily. And we had to say no, on one occasion, to a potential wheelchair bound evaluator in the mountains, as it simply isn’t feasible to get out of the car in the rough ground. Again, if you aim to represent all sectors within the evaluation team, then having women and men, and a spread of caste/ethnicity, as well as a PWD, along with the required thematic expertise and language skills, is virtually impossible. We also can’t assume that an evaluator from a specific group will necessarily be more sensitive to the issues of that group. Obviously we should, however, ensure that the evaluators in a team discuss potential disability issues and have an open mindset.

Sounds a bit defeatist, I realise. We can do some things to promote inclusion in evaluation. Use of online methods can assist us to reach people in remote areas – but this requires that they have access to a smart phone or laptop and expertise or assistance. And while this works for individual meetings – or several with their own connection – it doesn’t work for focus groups in a community setting. Invite participation of everyone in meetings, and enquire who isn’t participating, and who in the community may have a disability. Ensuring that if the disaggregation of data has extended to disability, then we report on this. If it hasn’t, then it is a recommendation for the project team. Encourage the project to provide sensitization/training on issues of disability for their staff (simple exercises like getting them to use crutches or a wheelchair are a great way for them to really feel the issues, rather than theoretically understanding them).

I am interested to hear more ideas from others for making evaluations more inclusive – beyond the obvious “ensure there is sufficient budget”.

Good luck to everyone on this challenging issue!

Yes, it's true that every place is different, but on the program I'm involved in as Monitoring and Evaluation Manager, the active participation of the disabled and their inclusion is very important, because their opinions count a lot if we're talking about an inclusive evaluation. Even if it often requires a great deal of awareness-raising on their part to ensure their active participation, it's quite normal to explain to them the importance of an inclusive evaluation, which is above all centred on them for a good evaluation and to measure the extent to which the project's objectives have been achieved.

[Translated from French]