What about stakeholders? Evaluation of social protection programs.
Dear Members of the EVALFORWARD,
Let me introduce myself, I am an expert in rural development and participatory practices. I am Lebanese and interested in learning from your experiences around the world, in developing practices and methodologies for stakeholders’ engagement etc.
As you have already faced, many projects and programs are today focusing activities and interventions on social protection. Social protection is a key element for achieving food security and rural development.
Anyone of you who find it interesting can share with the group some lessons learned based on the following key questions:
1. Have you ever been involved in the evaluation of social protection programs? What is the approach to assessing such programs?
2. What are the key elements that any expert would be looking for in social protection activities/programs?
3. Coming back to the ultimate goal of contributing to the development of rural societies and to their food security, what could be the elements to investigate and research and what indicators would be evaluated to reach a clear understanding on the community development?
Patricia R. Sfeir
Lebanon
Isha Miranda
Visiting Lecturer and Independent Evaluator Independent ConsultantDear Patricia,
see below my comments on your questions 1 and 2.
1. Have you ever been involved in the evaluation of social protection programs? What is the approach to assessing such programs?
Yes, nationally and sub-nationally as well as internationally. At national level in my country the Government has created a program called "Samurdhi", in English "Prosperity". It is a social protection program for citizens living well below the poverty line. The vision is "To make a Poverty Free Empowered and Prosperous Sri Lanka by 2030”. The mission is “Contributing to economic development through the building up of a poverty free prosperous country by empowering disadvantaged people (economically, socially, politically, physically, psychologically, legally and environmentally) and minimizing regional disparity through delivering effective, efficient, speedy and productive solutions in a people-friendly manner through the satisfactory contribution of the network of Departmental, Community Based Organizations and Micro-Finance Institutions and professionals with the collaboration of the private, public, people and political sectors and local and global agencies“.
At sub-national level, micro finance institutions, NGOs and other government sub-national entities have taken over most of the targets from economic to social achievements. Therefore, most programs circle it around this objective. However, there are gaps showing in this program due to various social and political environment embedded into the systems.
Initially the program adopted the longstanding welfare approach using both monetary approach and non- monetary approach.
Then it expanded to address the multidimensional aspects of poverty such as economy (consumption and assets), human development (education, health, safe sanitation, safe drinking water, electricity), socio-cultural dimension (dignity and network), political dimensions (power and voice) and protective aspects (conflict, natural disasters, risk of eviction).
For more information:
http://www.samurdhi.gov.lk/web/images/stories/publications/english/samurdhi_programme.pdf
http://repository.kln.ac.lk/xmlui/bitstream/handle/123456789/5305/MK%20Nadeeka-43-64.pdf?sequence=1
2. What are the key elements that any expert would be looking for in social protection activities/programs?
The evaluation questions should address both economic and social aspects of the social protection programme,to assess its contribution to community development in rural areas:
• Is the programme sustainable, contributing to a stable community rather than creating dependency?
• Are the program elements linked with national priorities in terms of livelihood development? F.i. are there links with agriculture/non agriculture sectors, government Agribase/non Agriculture trade subsides/welfare programmes, and activities relating alternative and product development non traditional agribase products?
• Is the program considering land Management, Agriculture land distribution and harvesting technics?
• Are there activities supporting livelihoods such as market development activities relating to the local areas, market expandable beyond the local area, usable technology and introducing new methods, product development?
• Financial inclusion, control over income over expenditure management, loan management.
• Family management and prevention of addiction: such as alcoholism, drugs (locally and internally made stuff) local gambling, abuses and harassment on women and children.
Noemi Pace
Dear Patricia,
Thanks for raising this topic in the EVAL-ForwARD Community of Practice.
Here are my answers to your queries (and links for more information).
1) Have you ever been involved in the evaluation of social protection programs? What is the approach to assessing such programs?
Yes. The Transfer Project, a multi-country research initiative to assess the effectiveness of social protection interventions (mainly cash transfers programs) in Africa, produced a series of quantitative and qualitative impact evaluations of social protection programs. The list of countries is available here: https://transfer.cpc.unc.edu/countries-2/
Clicking on each country, it is possible to get access to reports, survey instruments and publications. The survey instruments provide information on the key questions asked to assess the effectiveness of the program on a large set of domains, i.e. consumption, food security, agricultural production, income generating activities, education, time use, etc. (see for example for Lesotho https://transfer.cpc.unc.edu/methodology-and-tools/instruments/instruments-by-country/lesotho-instruments/).
The reports provide details on the methodology adopted for the impact evaluation, details on the study sample, main results of the impact evaluation (see for example for Lesotho https://transfer.cpc.unc.edu/wp-content/uploads/2015/09/Lesotho-Child-Grants-Programme_Follow-up_April-2014.pdf)
2) What are the key elements that any expert would be looking for in social protection activities/programs?
We should consider three key aspects: i) implementation of the program; ii) effectiveness of the program; iii) efficiency of the program.
i) Implementation: we should make sure that the target population is actually included in the program. There may be exclusion (poor households not included) and inclusion errors (wealthy households included). We should make sure to minimize both kinds of errors to meet the planned target population. Another important aspect in implementation is related to the actual activities of the program. For example, if we are evaluating a cash transfers program, we should check whether the payments are regular and of the right amount.
ii) Effectiveness: if the objective of a social protection program is to reduce poverty we should investigate the impact of the program of outcomes that proxy income/wealth, such as consumption expenditure, income generating activities, food security etc. If the objective of a social protection program is to improve education and health, we should assess the impact of the program on school attendance, drop out, school absenteeism, illness, etc. The impact evaluation reports available in the Transfer Project are a good example of the key outcomes considered to assess the effectiveness of the programs
iii) Efficiency: we should make sure that the interventions are cost-effective, i.e. overall benefits greater than overall costs. An example of this kind of study is here: http://www.fao.org/3/a-i3815e.pdf
3) Coming back to the ultimate goal of contributing to the development of rural societies and to their food security, what could be the elements to investigate and research and what indicators would be evaluated to reach a clear understanding on the community development?
A good and robust way to assess the impact of a program on the community is represented by the Local-Economy-Wide Impact Evaluation (LEWIE). This kind of analysis investigates the impact of a program on the whole economy, taking into account spillover and general equilibrium effects. Details on this kind of studies are here: http://www.fao.org/3/a-i4458e.pdf and on the cited references.
All the best,
Noemi Pace
Social Poicies and Rural Institutions Division FAO
Abubakar Muhammad Moki
Commissioner Policy Development and Capacity Building Office of the President-Cabinet SecretariatDear Patricia R. Sfeir,
My organisation has a 'Framework for Analyzing Public Policies-A Practical Guide' that is used in evaluations generally and can as well be applied to social protection programmes and policies. Key highlights in the framework are as follows:
1. Each evaluation covers the duration agreed with the stakeholders.
2. Evaluation is on three thematic areas, namely effects, implementation and cross cutting issues.
3. Effects cover evaluation of impact, effectiveness, relevance, unintended effects and equity (including Gender and Human Rights).
4. Implementation evaluation covers specific characteristics of social policies, processes, cost and efficiency, feasibility and adaptability.
5. Cross cutting issues cover sustainability/durability, partnerships and synergies.
Please let me know if this is helpful.
Grateful
Abubakar Muhammad Moki
Uganda