Addressing disability inclusion through evaluations in agriculture and rural development

image
©FAO

Addressing disability inclusion through evaluations in agriculture and rural development

Dear EvalForward members,

I would like to hear from you regarding your experience in addressing disability inclusion through evaluations.

The 2019 United Nations Disability Inclusion Strategy (UNDIS) provides the foundation for sustainable and transformative progress on disability inclusion through all pillars of the work of the United Nations. The Strategy includes a system-wide policy, an accountability framework and other implementation modalities. This Strategy requires the evaluation offices of United Nations agencies and organisations to report annually on the extent to which disability inclusion has been addressed through their evaluations. For some UN entities, this is the first time that disability inclusion is being addressed in a systematic manner through programming and evaluation, and this requires a rethinking of evaluation approaches and methodologies.

Persons with disabilities, as defined by the 2008 Convention on the Rights of Persons with Disabilities, include those who have long-term physical, mental, intellectual or sensory impairments which in interaction with various barriers may hinder their full and effective participation in society on an equal basis with others. Disability inclusion therefore implies the meaningful participation of persons with disabilities in all their diversity, the promotion of their rights and the consideration of disability-related perspectives, in compliance with the Convention on the Rights of Persons with Disabilities. Recent studies suggest that at present there are approximately 1 billion persons with disabilities in the world (about 15 per cent of the global population), of whom 80 per cent (800 million) live in developing countries (see for example the UN Flagship Report on Disability and Development). While analysis of the incidence, distribution and trends in disability is limited by a lack of high-quality data, the available studies indicate that there is a positive correlation between poverty and disability, at both the individual and the household level, and that disability is generally associated with multidimensional poverty. This correlation is even more pronounced in rural areas.

Despite the significant challenges faced by persons with disabilities living in rural areas, there is comparatively less experience in addressing disability inclusion through evaluations related to agriculture, rural development and food security. I would therefore like to tap into the experiences and expertise of the Eval Forward Network:

  • What has been your experience in addressing disability inclusion through evaluations?
  • What best practices can you suggest for ensuring the inclusion of persons with disabilities in our evaluations?
  • What challenges do you foresee in ensuring the inclusion of persons with disabilities in our stakeholder consultations and evaluation outreach going forward?
  • Can you point to published evaluations that have explicitly addressed issues of disability inclusion, to help us compile a repository of best practice examples?

I look forward to hearing from you as we open this important discussion.

Kind regards,

Eoghan Molloy

Evaluation Specialist
Independent Office of Evaluation (IOE)

International Fund for Agricultural Development (IFAD)

Dear Eval Forward colleagues;

A very interesting topic indeed, many thanks to Eoghan for bringing it up.

I would like to add my contribution from Rwanda's experience with a specific project evaluation from agriculture sector in Rwanda, where people with disabilities have been included from project design, to implementation and evaluation.

In the agriculture sector specifically, social inclusion is a critical aspect and disabilities is a cross cutting issue in all delivery sectors in Rwanda according to the Social Protection Policy, and this means in all projects and programs cycle as well as all development interventions.

The 5-year Agriculture Transformation Strategy for Rwanda (PSTA 4) recognizes the importance of addressing the needs of all actors in the agriculture sector and enabling farmers and agribusinesses to realize their full potential. The PSTA 4 promotes the inclusion of people with disabilities into the agriculture sector, through measures such as adaptive technology and labour-saving technologies. Furthermore, the PSTA 4 addresses HIV/AIDS through improved food and nutrition security, labour-saving technologies as affected persons may have reduced physical capabilities

Similarly, the social protection sector (in line with the Social protection policy) is responsible for conducting needs assessment of poor and vulnerable households (Women headed HH, People With Disability, HIV Positive Heads of HH, Child Headed HH…) and working with the Agriculture sector (especially Ministry of Agriculture and Rwanda Agriculture Board, agriculture-focused CSOs and private sector organizations) to ensure social protection and inclusivity of beneficiaries are prioritized within the agriculture sector’s programs and projects.

A specific example of Project Evaluation in which People with disabilities have been included is;

The “Rwanda Private Sector Driven Agricultural Growth (PSDAG) Project" funded by USAID has been implemented in Rwanda with one of its objectives being a crosscutting objective on Social inclusion of gender, youth (defined as ages 18-35), and People with Disabilities (PWD).

Under the project Target group; PSDAG supports government agencies involved in investment promotion, PSDAG also strengthens private sector capacity and facilitates expanded investments for existing and new private sector entities (local and international businesses located in Rwanda).

PSDAG supports private entities which promote inclusive growth which demonstrate potential to benefit smallholder farmers, women’s economic empowerment, women’s leadership, and engaging youth and persons with disabilities (PWDs) and PSDAG has actively been involved in 15 districts in Rwanda.

The project had a performance evaluation in 2018 and made sure the process was socially inclusive for example an extract from the methodology section says;

 “Between May 22, 2018 and June 4, 2018, the evaluation team conducted a total of 16 FGDs with 106 respondents out of a planned 120, or 88.4 percent. These were comprised of smallholder farmers belonging to activity-supported local farmer cooperatives, including representatives of other groups, including women, youth, and People with Disabilities (PWD). One major factor considered during the evaluation was to ensure a conducive environment for participation of all respondents specifically PWD.

Under the recommendation section, the evaluation team had this to say on social inclusion among others;

Social Inclusion: In key informant interviews and focus group discussions, innovations or other approaches suggested by respondents to enhance the meaningful engagement of targeted beneficiary groups included among others:

·         Promoting digital technologies, especially to improve engagement of youth; and Promoting access to existing financial services opportunities for all.

·         USAID/Rwanda, and other donors should consider promoting pilot projects deploying these inclusive approaches with cooperatives seeking to increase the engagement of women, youth, and People with Disabilities.

While some progress has been made, the gaps are still many, most projects are considering people with disabilities at design stage, but very limited involvement in decision making and evaluation processes, som PWD have movement limitations to areas where for instance focus group discussions are being conducted, they have limited information on project status due to their limited participation in the projects and hence are left out during evaluations.

There is a need for more efforts among project managers and evaluators to support full participation of PWD in project evaluations, ensuring that we are leaving no one behind in the process.

A pleasant weekend,

Judith

Greeting!

We shall continue from where we left off in our previous contribution, which is given below. Avoiding all arcane expression typically in use, let us point out a few important facts that are important to real disabled people with reference to agricultural pursuits:

   • Certain disabilities like blindness or severe visual impairment will exclude people from engaging in agricultural work in any meaningful way.

   • This will also apply to people with severe to limited mobility owing to leg injury or Mysthenia gravis, etc.

   • Certain mental dysfunctions may preclude the sufferers from participation in agricultural pursuits.

   • Thus, the opportunities available to disabled people in agriculture are limited owing to certain physical or mental handicaps from which they suffer. This happens to be the situation in real life.

   • However, deaf people and those who suffer from other disabilities  may be able to fruitfully engage in agriculture provided that what they plan to do do not impose an undue strain on them.

   • What those people may usefully do depends on the real life conditions that exist in the area they live. We cannot suggest any meaningful generic improvements unless we have area specific information.

We hope project planners and evaluators begin right there and start from the beginning rather than basing their work on theoretical or academic papers. After all, we are trying to ameliorate the lives of real people, arn't we?

Best wishes!

Lal Manavado.

Genta Konci

Genta Konci

Independent Office of EvaluationUNDP

What has been your experience in addressing disability inclusion through evaluations?

The UNDP Independent Evaluation Office (IEO) conducted a corporate evaluation disability-inclusiveness in UNDP in 2016.  The evaluation looked both at the inclusion of persons with disabilities in programming, as well as how welcoming the organisation is to persons with disabilities within its workforce, enabling operational as well as programming considerations.

This evaluation also played a key role in supporting UNDP to address gaps in meeting disability inclusiveness standards and became an advocacy tool to guide and promote disability-inclusiveness across UNDP. The evaluation team included two PWDs, and the lead evaluator had extensive expertise in evaluating human rights in development. Evaluation questions covered four aspects of disability inclusion (accessibility, accountability, participation, and non-discrimination). Also, evaluation stakeholder mapping and data collection methods actively involved persons with disabilities and their representative organizations in all the country case studies, regional and headquarter consultations.

The timing was fortuitous, and the team’s efforts to engage with associations and persons with disabilities – including within the evaluation team, were noteworthy. The disabilities inclusive development evaluation made an impact – influencing the development of a new UNDP disabilities strategy and the recent UN-wide strategy. 

What best practices can you suggest for ensuring the inclusion of persons with disabilities in our evaluations?

Conducting a thematic evaluation on Disability Inclusiveness evaluation proved to be very important guiding piece for UNDP work and UN-wide strategy on disability. As the same time, it provided the Independent Evaluation Office (IEO) with further insights on the methodologies used to assess disability as well increased awareness and understanding of the topic.

IEO is supporting the implementation of UN Disability Inclusiveness Strategy and has reported on the status of disability in evaluation, which will serve as a baseline to measure progress towards disability inclusion in evaluation. IEO has committed to include in the current IEO Evaluation Guideline, guidance on how to address disability inclusion; to further include disability in the evaluation process; and to conduct a meta-analysis of evaluation findings, conclusions and recommendations related to disability inclusion.

What challenges do you foresee in ensuring the inclusion of persons with disabilities in our stakeholder consultations and evaluation outreach going forward?

At the time of UNDP IEO Disability Inclusiveness evaluation, the team faced challenges in identifying and mapping UNDP work on disability-inclusive development, both at programmatic and internal/institutional levels. Disability-inclusive development was not a distinctive area of work that is highlighted within the UNDP Strategic Plan, and country offices have been under no obligation to report on the extent of their support and activities in this area. Consequently, while many UNDP projects self-identify as paying attention to ‘vulnerable groups’, few have made clear their relevance to persons with disabilities, and even fewer identify a specific budget for this work. Another challenge were the incomplete data records of prior project work. With the fairly rapid turnover of personnel in many country offices, there were sizeable gaps in institutional memory. As a consequence, the global portfolio set out in this evaluation may not fully and accurately account for every disability related project that UNDP has carried out.

Can you point to published evaluations that have explicitly addressed issues of disability inclusion, to help us compile a repository of best practice examples?

UNDP Disability Inclusiveness Evaluation: http://web.undp.org/evaluation/evaluations/thematic/disability.shtml

Greeting!

Speaking in general terms about evaluation, there does not seem to be a wide-spread agreement on what exactly one intends to evaluated. Avoiding the use of any jargon, we would have thought that what really matters is to evaluate whether a project/programme has succeeded in improving the quality of life of the target group it was designed to benefit. This is not the same as the successful physical conclusion of any development effort. For instance, in an earlier discussion successful completion of a modern motor way and a bridge to join an island and the main land had been cited as vain efforts.

We have pointed out that meeting the following criteria is essential if a project is to benefit its target group:

1. It is sustainable by the target group with respect to the physical resources available to it.

2. The competence required to run and maintain it are within the available skill set of the target group.

3. The project makes a significant contribution to improve the education in its real sense, health, nutrition, security in its broadest sense, procreation and what we have termed non-material needs. This last includes aesthetic enjoyment i.e., literature, music, etc., participation in games and sports, etc. Improvements in any one or more of them will increase one's quality of life.

After these introductory remarks, it is clear what we need to find out is how and to what extent agricultural pursuits may contribute to the quality of life of the disabled people. We hope that work on this area will succeed in determining the what is possible to achieve in clear and concise terms.

Best wishes!

Lal Manavado.

Åsne Kalland Aarstad

Åsne Kalland Aarstad

Senior Adviser, Evaluation DepartmentNorad

The last evaluation on disability inclusion commissioned by the Evaluation Department in Norad was launched in 2012 and can be accessed here: https://norad.no/om-bistand/publikasjon/2012/mainstreaming-disability-in-the-new-development-paradigm-evaluation-of-norwegian-support-to-promote-the-rights-of-persons-with-disabilities/

In the preparation for an upcoming evaluation on disability-inclusion, the Evaluation Department has consulted widely with stakeholders, including a number of Norwegian-based DPOs. One issue that was raised multiple times during our consultation concerns the need for clarity around terms and definitions, ensuring a common language, and common methods of measurement, ensuring reliable and comparable data.

The Evaluation Department is currently commissioning a mapping of Norway’s’ disability-inclusion efforts from 2010 and onwards. The mapping will take point of departure in the recently introduced OECD-DAC policy marker on inclusion and empowerment of persons with disabilities (http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=DCD/DAC/STAT/RD(2019)1/RD1&docLanguage=En )

Best,

Åsne

 

Dear Eval Forward members,

Thank you for the very useful insights and examples that have so far been provided in response to my earlier email regarding disability inclusion in evaluations relating to agriculture and rural development.

We have had some very interesting inputs, such as the capturing of individual responses from persons with disabilities through the household survey conducted for the Inter-Agency Humanitarian Evaluation of the Drought Response in Ethiopia 2015 – 2018, as shared by Amleset Haile. Other contributions, such as  that from Bassirou Diagne, raised the point that the inclusion of persons with disabilities could be captured under analyses relating to ‘vulnerable groups’. However, underscoring the complexity of the topic, Mohammad Lardi shared the example of a UNFPA multi-country study on young persons with disabilities, which gave a thorough analysis of the intersecting vulnerabilities and marginalities faced by persons with disabilities (e.g. HIV status, gender, distance from urban areas, poverty, age, etc.) and made the important point that every disability is different and that it is somewhat artificial to “lump everyone under the same heading”

As Pamela White has noted, people might tend to have different and subjective understandings of what ‘disability’ entails, and therefore they might not so readily self-identify as having a disability, for a host of reasons. Similarly, Lal Manavado pointed out that persons with disabilities who live in rural areas are perhaps even more likely to be excluded or misrepresented, and extra care and consideration is therefore needed to ensure that evaluations truly capture the views of persons with disabilities living in rural areas.

From that point of view, I am curious as to how the IAHE in Ethiopia, or indeed other evaluations have identified persons with disabilities in their sampling, and what is the accepted best practise for doing so.

Thank again to those of you who have provided links to external guidance (e.g. the Washington Group on Disability Statistics). If there is similar guidance that could be of use, please feel free to share with the Eval Forward network.

And do please continue to share any other interesting examples of how persons with disabilities have been included through your evaluations.

Looking forward to continuing this discussion.

Kind regards,

Eoghan

Dear Eoghan, 

Thank you for raising this issue. Greetings from Ethiopia. I was a team member of the Inter-Agency Humanitarian Evaluation of the Drought Response in Ethiopia 2015 - 2018 evaluation and was the one fully responsible for the drought-affected community survey conducted for the UN response in Ethiopia (2015-2018).  As part of the survey, we purposely included various types of community members including disabled people, and also wanted to know their reflection towards responses provided and how it was perceived by the different groups.

Herewith please find the report attached and focus on the process, methodology, and questions itself and the whole approach towards the assessment. I would be very happy to share in detail my/our experience in this evaluation if you find it useful. Please let me know, I would be very happy to engage.

Best regards,

Amleset

 

Dear Colleagues 

I would like to share with you my input on the issue of  disabilities. My experience with this specifically is a bit linked to managing implementation of evaluation. In Morocco, I was working with UNFPA in Country office as national assistant Representative when Head Quarter from UNFPA New York  sought my contribution on managing evaluation Sexual Reproductive Health among youth in morocco in July 2018.  The project titled as “young persons with disabilities”:  global study on ending gender-based violence, and realising sexual and reproductive health and rights”  was  highlighting promising steps being taken in Ecuador, Morocco, Mozambique, and Spain.

The study identified multiple areas in which CSOs, national policymakers, regional monitoring bodies, and international organizations are making progress in reducing and eliminating discrimination against young persons with disabilities. This report seeks to further that important work and to contribute to the global movement to leave no one behind.

The specificity is that the standard protocol has been conceptualized from an academic team and we have been trying to implement it on the ground. It was  time consuming in coordination and advocacy to get evaluation acceptable by national partners. It is a complicated issue, for a combination of reasons possibly  the limit of stakeholder’s roles  and population severely disabled  are not all represented, and health care for everyone is a problem in developing countries. Some are out of schools in rural areas. There is also level of shame and not wanting to admit that there is a problem for people with disabilities especially for sexual and reproductive health.  

Hopefully, we have in Morocco data, even disaggregating data already by sex, location. And we found some national institutions are doing targeted household surveys to try to find out the functional disabilities and how they might be served by being involved in the social and economic activities

The other issue is that every disability is different – you can’t just lump everyone under the same heading.

I have also met with organised groups of people with disabilities in (Agadir) in south of Morocco – they have good ideas, good projects and have playing a great role in COVID-19 helping targeting PWD  especially those who are more in need by advocacy in all level they obtain aid and distrusted, success stories exist and need evaluation.

Another issue is category of elders among them a high rate of disability that need mix of strategies. We haven’t done so much on disability within health in Morocco.  

I think that under COVID 19 many innovations revolutionize our approaches to reach unreachable people with disabilities, evaluation can help us to document success stories.

As I have participated in the evaluation mentioned above and are attaching here a short summary on finding 

 

YOUNG PERSONS WITH DISABILITIES:  July 2018

Global study on ending gender-based violence, and realizing sexual and reproductive health and rights

MOROCCO

Morocco is broadly representative of Arab states and of lower-middle income countries more generally. Morocco ranks 123 out of the 188 listed on the UNDP Human Development Index.296 It represents both a region and level of economic development wherein many states are developing new institutions to promote their citizens’ human rights but also face challenges. These new developments include laws promoting the equality of persons with disabilities, but also specific considerations, notably the stipulation that state policies and practices do not conflict with the provisions of the Islamic Shariah law, which can have an effect on the policies and strategies implemented regarding SRHR and GBV prevention and response services. Morocco, however, was chosen for this study because of the progress it has made promoting the rights of young persons with disabilities, including enjoyment of and access to SRHR and GBV prevention and response services.

Research shows that the national prevalence rate of people with disabilities in Morocco is 6.8 per cent297 with the most common disabilities reported as visual and mobility related. The ratio of males and females is similar, showing 49.7 per cent for males and 50.3 per cent for females. Further, 60.8 per cent of people with disabilities reported difficulties in accessing health services generally.

Morocco has made significant achievements in developing the necessary legal and policy framework for promoting the rights of young persons with disabilities related to Sexual and reproductive health and rights (SRHR) and gender based violence (GBV) prevention and response services in recent years. Many laws in this framework are currently in the final stages of approval. Morocco has signed and ratified CEDAW, CRC, and the CRPD and has been developing a national legal framework for their implementation. The new 2011 national Constitution includes the right to health to all citizens, and it states the state’s intention to mobilise all available means to facilitate the equal access of citizens to the rights of treatment, health care, social protection, health coverage, solidarity, and to living in a healthy environment. The Constitution also recognises the principle of gender equality in all civil, political, economic, social, cultural, and environmental rights and freedoms; the right to physical and moral integrity of individuals; and the principle of equality and combating all forms of discrimination. Morocco adopted Law N° 103-13 on combating violence against women in 2018.300 The Criminal Code of 2003 has also been amended to prohibit and punish discrimination on the basis of disability.

KEY POINTS

·       Young persons with disabilities experience context-specific barriers that prevent them from exercising their rights on an equal basis with others.

·       States have an obligation to design context-specific laws, policies, and programs that address humanitarian crises, poverty, rurality, high HIV/AIDS prevalence, institutionalization, and other circumstances that disproportionately increase the risk that the rights of young persons with disabilities will be violated.

·       Young persons are at disproportionate risk of GBV and sexual exploitation in conflict and post-conflict environments and in the aftermath of natural disasters.

·       States have a positive obligation to protect the rights of persons with disabilities during and in their responses to humanitarian crises.

·       Poverty itself is a barrier to young persons with disabilities in accessing and enjoying SRHR and GBV prevention and response services on an equal basis with others.

·       The majority of young persons with disabilities in developing countries live in isolated rural areas that lack access to disability-inclusive SRH and GBV services in comparison to urban-based services.

Dear Eoghan, 

Colleagues from UNICEF have kindly shared the following recent evaluations that are on or include children with disabilities: 

 

Dear Eoghan and other members,

I am trying to think about the disabilities question. My experience with this specifically is a bit more implementation linked than evaluation. In Nepal, I am working with a Finnish/EU funded water resources development project (RVWRMP), and we have been trying to work on issues related to disability, but it is tricky. Apart from anything else, in the remote mountains where we are working, there are surprisingly few PWD (much fewer than there ought to be really). I think that is a combination of reasons – possibly severely disabled babies are not cared for, or simply don’t get the necessary medical care to survive. Health care for everyone is a problem. Some kids are sent down to schools on the plains. There is a level of shame and not wanting to admit that there is a PWD in the family – as if it is severe, it might make things difficult for marrying off the siblings – so some may be kept hidden at home. But another issue is that many people don’t self identify as having a disability. I have sat in meetings trying to raise the issue, and there will be someone sitting in the group, missing a leg, but they won’t call themselves disabled! Even access to mass meetings and trainings can be problematic for people with limited mobility. In the mountains there is no way that anyone can use a wheelchair, so if they can’t walk or hop, it is difficult for an adult to attend a meeting far from their home.

We consider disability within social inclusion (and rights). However, we also have the problem that we are disaggregating data already by sex, caste, ethnicity, location. If we have to start disaggregating everything by PWD we will have a mess! We do some targeted info gathering. For instance, when reporting hygiene activities during COVID, we counted work with PWD. And we are doing targeted household surveys to try to find out the functional disabilities and how they might be served by being involved in the project activities. But I don’t want to set targets as it is a bit artificial.

The other issue is that every disability is different – you can’t just lump everyone under the same heading. People with eyesight problems may be offended if they are considered the same as people with disabilities. I have also heard (from a Finnish disability activist and researcher) that there can be sub-divisions for other reasons – for instance some Organisations of PWD in India divide themselves into sub-groups according to caste!! The Brahmins don’t want to mix with the Dalits!

I have also met with organised groups of PWD in western Nepal – they have good ideas, however they tend to be more city based and not really active at all out in our project areas – I guess that is probably the case in many countries.

It  has been easier for us to talk about disability with WASH – and I learned that when talking about toilet access, rather than talking only about PWD, I expand it to include frail elderly – that gets everyone’s attention as everyone has an elderly person at home, and we will all become old one day!! We haven’t done so much on disability within agriculture – but we do have PWD involved in the home garden groups.

The Washington Group has done work on disability. I’m attaching their short list of questions – very much related to functional disability. http://www.washingtongroup-disability.com/

Re evaluation – I suspect it is difficult unless you do things at a very local level. Mind you – the experience of COVID has moved us very quickly into using more online tools such as Zoom or Google, and it is amazing how well it works. So it might be feasible to do interviews with PWD in their own homes (depending on the disability). I would imagine the most useful technique would be some form of qualitative evaluation – most significant change, etc. – rather than trying for large scale quantitative evaluation (as you are unlikely to collect enough data for that).

Anyway – good luck in collecting the info. I will watch with interest.

Best wishes, Pamela

Dear Eoghan,

I find your topic very interesting, disability inclusion is a challenge in program intervention. In general the UN rural interventions are note focused on a disability inclusion approach. So if we considered this group like a type of vulnerable persons, we can judge the importance of this topic in the UN interventions.

For example in our recent study of the socio-economic character of households, we discovered that on average in an household composed of 14 people only 5 are active. Among the 9 inactive that represent the more vulnerable in the house, 5 are on average children, 2 are aged persons and 2 are people with disability.  

I think the challenge is in the whole process from project planning to evaluation. We need to include in the project target a specific scope for disability inclusion dimension like gender and climate smart and other dimensions. If we include the disability dimension in the results framework easily the evaluation terms will take in charge the accountability of the intervention for disability inclusion.

Thank you for this important topic,

Bassirou

Dear Mr. Molloy,

I am happy to see that you have raised the question, does evaluation ascertain the changes in the quality of life of the disabled following the completion of a programme/project? Naturally, such changes can be positive or negative. Unfortunately, some efforts at 'modernisation seem to have a negative effect on the disabled, especially on those living in less affluent countries.

For instance, pdf documents are supposed to be 'better looking' than simple text files or doc files. But this is not easily accessible to the visually impaired living in poor countries, because the software needed to make such documents accessible to the visually impaired is just too exprensive for most of them. Moreover, the pc's needed to run them are too expensive for the target group named here.

I suggest you get in touch with the organisations for the disabled in as many countries as possible via e-mail and request their views. Of course, this may provide some linguistic difficulties as personnel in many such organisations do not speak standard English or only know their own language.

Be that as it may, you might find some useful bits of information from the real world from this source. Meanwhile, I am a little sceptical about published materials, because they are usually written by the local non-handicapped people who are educated in cities and have little notion about how the disabled in the rural areas live.

I hope this would be of some use to you, and wish you every success.

Best wishes!

Lal Manavado.