Thanks for raising this topic in the EVAL-ForwARD Community of Practice.
Here are my answers to your queries (and links for more information).
1) Have you ever been involved in the evaluation of social protection programs? What is the approach to assessing such programs?
Yes. The Transfer Project, a multi-country research initiative to assess the effectiveness of social protection interventions (mainly cash transfers programs) in Africa, produced a series of quantitative and qualitative impact evaluations of social protection programs. The list of countries is available here: https://transfer.cpc.unc.edu/countries-2/
Clicking on each country, it is possible to get access to reports, survey instruments and publications. The survey instruments provide information on the key questions asked to assess the effectiveness of the program on a large set of domains, i.e. consumption, food security, agricultural production, income generating activities, education, time use, etc. (see for example for Lesotho https://transfer.cpc.unc.edu/methodology-and-tools/instruments/instruments-by-country/lesotho-instruments/).
2) What are the key elements that any expert would be looking for in social protection activities/programs?
We should consider three key aspects: i) implementation of the program; ii) effectiveness of the program; iii) efficiency of the program.
i) Implementation: we should make sure that the target population is actually included in the program. There may be exclusion (poor households not included) and inclusion errors (wealthy households included). We should make sure to minimize both kinds of errors to meet the planned target population. Another important aspect in implementation is related to the actual activities of the program. For example, if we are evaluating a cash transfers program, we should check whether the payments are regular and of the right amount.
ii) Effectiveness: if the objective of a social protection program is to reduce poverty we should investigate the impact of the program of outcomes that proxy income/wealth, such as consumption expenditure, income generating activities, food security etc. If the objective of a social protection program is to improve education and health, we should assess the impact of the program on school attendance, drop out, school absenteeism, illness, etc. The impact evaluation reports available in the Transfer Project are a good example of the key outcomes considered to assess the effectiveness of the programs
iii) Efficiency: we should make sure that the interventions are cost-effective, i.e. overall benefits greater than overall costs. An example of this kind of study is here: http://www.fao.org/3/a-i3815e.pdf
3) Coming back to the ultimate goal of contributing to the development of rural societies and to their food security, what could be the elements to investigate and research and what indicators would be evaluated to reach a clear understanding on the community development?
A good and robust way to assess the impact of a program on the community is represented by the Local-Economy-Wide Impact Evaluation (LEWIE). This kind of analysis investigates the impact of a program on the whole economy, taking into account spillover and general equilibrium effects. Details on this kind of studies are here: http://www.fao.org/3/a-i4458e.pdf and on the cited references.
All the best,
Noemi Pace
Social Poicies and Rural Institutions Division FAO
RE: What about stakeholders? Evaluation of social protection programs.
Dear Patricia,
Thanks for raising this topic in the EVAL-ForwARD Community of Practice.
Here are my answers to your queries (and links for more information).
1) Have you ever been involved in the evaluation of social protection programs? What is the approach to assessing such programs?
Yes. The Transfer Project, a multi-country research initiative to assess the effectiveness of social protection interventions (mainly cash transfers programs) in Africa, produced a series of quantitative and qualitative impact evaluations of social protection programs. The list of countries is available here: https://transfer.cpc.unc.edu/countries-2/
Clicking on each country, it is possible to get access to reports, survey instruments and publications. The survey instruments provide information on the key questions asked to assess the effectiveness of the program on a large set of domains, i.e. consumption, food security, agricultural production, income generating activities, education, time use, etc. (see for example for Lesotho https://transfer.cpc.unc.edu/methodology-and-tools/instruments/instruments-by-country/lesotho-instruments/).
The reports provide details on the methodology adopted for the impact evaluation, details on the study sample, main results of the impact evaluation (see for example for Lesotho https://transfer.cpc.unc.edu/wp-content/uploads/2015/09/Lesotho-Child-Grants-Programme_Follow-up_April-2014.pdf)
2) What are the key elements that any expert would be looking for in social protection activities/programs?
We should consider three key aspects: i) implementation of the program; ii) effectiveness of the program; iii) efficiency of the program.
i) Implementation: we should make sure that the target population is actually included in the program. There may be exclusion (poor households not included) and inclusion errors (wealthy households included). We should make sure to minimize both kinds of errors to meet the planned target population. Another important aspect in implementation is related to the actual activities of the program. For example, if we are evaluating a cash transfers program, we should check whether the payments are regular and of the right amount.
ii) Effectiveness: if the objective of a social protection program is to reduce poverty we should investigate the impact of the program of outcomes that proxy income/wealth, such as consumption expenditure, income generating activities, food security etc. If the objective of a social protection program is to improve education and health, we should assess the impact of the program on school attendance, drop out, school absenteeism, illness, etc. The impact evaluation reports available in the Transfer Project are a good example of the key outcomes considered to assess the effectiveness of the programs
iii) Efficiency: we should make sure that the interventions are cost-effective, i.e. overall benefits greater than overall costs. An example of this kind of study is here: http://www.fao.org/3/a-i3815e.pdf
3) Coming back to the ultimate goal of contributing to the development of rural societies and to their food security, what could be the elements to investigate and research and what indicators would be evaluated to reach a clear understanding on the community development?
A good and robust way to assess the impact of a program on the community is represented by the Local-Economy-Wide Impact Evaluation (LEWIE). This kind of analysis investigates the impact of a program on the whole economy, taking into account spillover and general equilibrium effects. Details on this kind of studies are here: http://www.fao.org/3/a-i4458e.pdf and on the cited references.
All the best,
Noemi Pace
Social Poicies and Rural Institutions Division FAO