I would like to share with you our experience in the implementation of the GAFSP MMI project in Senegal.
We have set up a participatory M&E system in which the communities are the central elements. For each PO (producer organisation), we selected a team of supervisors and facilitators from the communities. We trained these people in digital data collection via kobo and also in the facilitation of a qualitative survey in order to empower them in data collection. The training was difficult, given the level of some facilitators, but it must be acknowledged that they are very much empowered and accepted by the communities for the provision of reliable data.
We are continuing this process to make monitoring booklets available to the grassroots producer organisers, which are filled in every year so that the facilitators, via the tablet, will go on missions to collect data in the monitoring booklets of the value chain.
It is necessary to have an inclusive approach and to define an M&E guideline that makes producers responsible for the implementation of the monitoring-evaluation system.
[this contribution was originally posted in French]
Thank you for this broad and rich theme.
A project can be defined as an action or compound actions to bring about change or change on a topic or topics in a defined locality. The ToC theorizes the expected change, as does the results framework, which groups together expected results on the basis of planned and budgeted activities throughout the execution of the project. However, it is important to note that the ToC must be more objective than a vision in its definition, it must be based on the results of a diagnosis of the intervention area in a contextual (social, economic, environmental, life style, etc.) and situational way in relation to the project. The definition of a problem tree is very important for objectively parameterising the ToC. Now, it is a theorising of the expected change in the project defined on the basis of contextual and situational evidence at the start of the project, so its revision is not essential if the project framework remains intact. However, if, for example, the project's intervention framework is revised at the end of a mid-term review, as well as the results framework, it becomes imperative to revise the ToC in order to be consistent with the evolution of the intervention. This task is the responsibility of the project team and stakeholders who have a clear picture of the intervention and who can also decide in a participatory manner on these different changes.
Following the example of the Global Agriculture & Food Security Program Missing Middle Initiative (GAFS MMI) Senegal project, after the definition of the annual programs of work and budgets and partnerships, the project team, with the support of the stakeholders, began a revision of the project results framework followed by a slight re-parameterisation of the ToC because the changes did not require a global revision, which may be the case in other projects or programmes depending on the re-planning and revision.
[contribution posted originally in French]
Very interesting topic and I find all responses interesting too to help to establish an evaluation system at national level. My advice is very limited as I don’t have information on the ongoing monitoring system and the vision or specifics core goals to 2030, as the evaluation system will inspired from this two reference documents.
I find the more difficult phases is to establish a monitoring system with validated year targets. A SMER (Six Months Evaluation Report) could be adopted. The first SMER would be for analysis of the progress of goals and target and if necessary make some reorientation in the programme and the second in the same line with a consolidation part for the annual evaluation. However the SMER approach could be challenging for evaluation since evaluation is for results and the evidence needs time. It would be more relevant for evaluation to have a five year vision format.
So if you decide to keep the yearly period for monitoring and evaluation, I advise to take this approach of SMER with a template based on specifics goals and targets that can help in defining the evaluation quantitative and qualitative questions.
I also recommend using digital collection at the national level and validating the results of the evaluation through a participatory approach.
Thank you for this very appreciated exchange.
I'm a National Monitoring, Evaluation and Knowledge Management Officer in Senegal and perhaps my small experience can provide some responses to the two questions. New profiles are emerging and developing which combine these two missions and this is because there is a close and proved relationship between these two functions.
The GAFSP MMI (Global Agriculture and Food Security Program Missing Middle Initiative) project format I work for is a pilot. The M&E and the KM functions are important for data treatment, capitalization and workshops on lesson learned. In this project, Knowledge Management is a participatory work that includes the members of the Coordination Unit: after the coordination mission, our goal is to decide on the basis of the lesson learning and knowledge capitalization if to scale up the pilot initiative or not and in which way.
In case of bigger programs or plans composed by many project components, you need to establish a M&E and KM Department with a team composed by the Team Leader, the M&E Officer for data management, the M&E Officer for Knowledge Management and a M&E officer for data communication. More options and formats for the team composition are possible depending on the type of organization.
I find your topic very interesting, disability inclusion is a challenge in program intervention. In general the UN rural interventions are note focused on a disability inclusion approach. So if we considered this group like a type of vulnerable persons, we can judge the importance of this topic in the UN interventions.
For example in our recent study of the socio-economic character of households, we discovered that on average in an household composed of 14 people only 5 are active. Among the 9 inactive that represent the more vulnerable in the house, 5 are on average children, 2 are aged persons and 2 are people with disability.
I think the challenge is in the whole process from project planning to evaluation. We need to include in the project target a specific scope for disability inclusion dimension like gender and climate smart and other dimensions. If we include the disability dimension in the results framework easily the evaluation terms will take in charge the accountability of the intervention for disability inclusion.
Thank you for this important topic,
I’m joining this interesting discussion, occasion for us to share from the ongoing rapid evaluation assessment proposed by the Coordination Unit for the GAFSP MMI Senegal. Today in this pandemic situation, the rapid evaluation is an effective support for pertinent policies decision and programmatic action for most affected people. In December, in the baseline study, we had used the FIES (Food Insecurity Experience Scale) to evaluate the prevalence of food insecurity, at the time we had a database in FIES module for household and individual level. From this situation, we try to use evaluation with this tool to estimate the impact of the Covid 19 on food security and rural vulnerability. A comparative analysis from data FIES December 2019 to data FIES for June 2020 will give use an effective information on the impact of the covid 19 on food security. In parallel we will use the socio-economic tools to produce more evidence in the rural situation in this pandemic. Our data covers the intervention areas of the project. The Tambacounda region are defined like the more poor region in Senegal. Now from extrapolation the data can be used to estimate the actual situation of prevalence of food insecurity. The results will be used by the project to prepare the post pandemic planification and necessary action and too the more affected sub-areas where producers and households in general need more support.
This rapid evaluation exercise are in process, so this is a short extract to explain in the urgency situation if you have baseline data how rapid evaluation with practice tool, can help to improve data quality and policies on food security....
I promise after the realization to produce a capitalization document if necessary to share with members.
Thank you very much for this link shared. The initiative is actually ongoing in Senegal the world Bank is in collaboration with the National office of Statistics to drive the evaluation between June to April 2021 and the FAO particularly the Statistics Division contribute by training the office members on the FIES tool adapted to Covid-19.
This type of evaluation are a macro level evaluation at national level and we take the results to be aligned neatly in national policies responses. However we need to have more specific rapid evaluation in the project areas with micro data in relation to the project goals and indicators. We are in a state for proposal for this type of specific rapid evaluation but our approach will be for a scope proposed by FAO for rapid assessment without the phone surveys.
Thank you for your feedback
Dear Moustapha and others,
Thank you for these contributions, which tell us a lot about the issues of Monitoring and Evaluation.
In most cases of definition of Monitoring and Evaluation Plans, the actors do not focus on the field and the object of the system, we are more concentrated on the functions and tools according to information needs sponsored by the donor or other stakeholders. If, however, the main purpose of monitoring and evaluation is to accompany the implementation in the direction of a permanent critical reflection to achieve the results and to alert on the critical conditions for the implementation of the intervention, we have a real challenge in the revision of monitoring and evaluation plans. For some, monitoring and evaluation are two complementary functions and for others it is a single monitoring-evaluation function with permanent monitoring activities and periodic and thematic evaluation activities following intervention guidelines. Today we are in the era of results-based management and results-based budgeting, hence the essential place of monitoring and evaluation for the success of programs or even the achievement of the SDGs. Programs invest more in achievements and communication and less in monitoring and evaluation, which requires a lot of resources conditioned by the employer of the intervention. Today with the digitization of systems we have less human resources but that does not prevent the cost of material resources remaining high for a good implementation. In most countries, especially developing countries, development policies do not give much importance to monitoring and evaluation of the SDGs. They compile data received from programs with partners to express progress, which is not a relevant resolution for the objective nature of monitoring and evaluation. In 2016 I proposed an ADMA (African Durable Measurement Agency) project to overcome this problem in order to solve the issue of relevance and inconsistency of interventions in the same locality and to note the interventions by their contribution to the achievement of SDG ... (see attachment).
As part of the implementation of the Global Agriculture and Food Security Program Missing middle Initiative (GAFSP MMI) Senegal project, the FAO team based in the field with had among its missions the establishment of a participatory and inclusive Monitoring and Evaluation system. Today we decided to capitalize on this experience in order to share it for beneficial purposes. In summary, the process started with a diagnosis of the essentials in terms of monitoring and evaluation or even data management near the beneficiary producer organizations at the office level with the technical team and at the field level with the farmers. This diagnostic mission made it possible to scope the monitoring and evaluation system with a clear object and its fields of application. This mission has also made it possible to have a look and discussion on the tools currently used by farmers in the context of their activities in order to integrate them in the system after discussion and reinforcement. The discussion makes it possible to see the shortcomings with farmers and other stakeholders in order to take the initiative together to strengthen the system in this direction. After this phase, the unit proceeded with the design of the device and the tools shared with the producer organizations for validation. It is in the implementation that we made the producer organizations responsible for their participation and reinforcement so that after the project they would be able to unfold the device. Each producer organization has its team formed by the coordination unit in a progressive manner for the implementation of the device. They are in the collection of quantitative data with KoBo collect and qualitative data with Dictaphones and focal points in the management of the database with KoBo collect. Today the implementation of the system is participatory in all the functions that make it up in its entirety. We will not fail to share the experience after capitalization with a lot of information on the methods and approaches adopted.
Dear Carlos and Dear members,
Thank you to receive my modest contribution.
What is your experience with the use of theories of change?
I have just done 1 year at FAO as a monitoring and evaluation officer under the GAFSP Missing Middle Initiative program. I have learned many things at you@FAO in the training of M&E guideline FAO. I think FAO has a solid experience in evaluation with many practical cases. if I'm not wrong the ToC is not a very popular tool.
In my previous experiences as a follow-up and evaluation officer, we used TOC for a smart project or program communication but also to share the project's intervention logic with stakeholders (partners, beneficiaries, etc.) in order to receive to promote the critical review focused on the key points of the project. It is a tool requested by several donors with different formats not far away, in the form of a diagram or table that always respects the same logic in the short term for direct results, medium term for effects and long term for impacts. The expected purpose would be to develop a resulting sentence i.e. the theory that in a simple text explains the project or program in complexity
What are the main added value of these theories, from your own perspective?
At FAO, I discovered in the project documents, the focus was more on the logical frameworks and results frameworks, which are also as relevant tools as the ToC but very complex for communication. ToC can be a tool that encompasses these other two tools while allowing a reading on the logic of intervention and the relevance of the activities or actions to be carried out. ToC traces the response from problems to resolutions. If it is a program the ToC is the best tool to define a vision but also to plan a participatory scaling up.
As part of GAFSP MMI Senegal, I had taken the initiative by drawing inspiration from GAFSP MMI's Global ToC to define a project-specific ToC that aligns with this global program while registering with PNUAD (National program of UN for development) and CPP 2019-2023. in the form of a table, the tool had facilitated the reading of the intervention and we also led to a revision of the logical framework with the integration of new results that take into account the gender dimension. shared with the partners, in a single sentence, we harmonized our understanding of the project's intervention logic. this tool had also led to the development of a M&E guideline that takes into account all the project's outcome dimensions
Have theories-of-change in your view made a difference in the programmes and projects that you have evaluated, especially when compared to other planning tools like log-frames and result chains?
Toc is a strategic tool for project or program evaluations. beyond the results defined in the logical or outcome framework, in an evaluation, the ToC allows contributions to be defined to macro indicators referring to broader policies or programs that include intervention. with Toc, we have often identified factors for successful intervention beyond the expected outcomes in relation to the baseline situation in the area of intervention. it is a tool that in an evaluation allows to see the cross-cutting themes and the chain justifying the contribution of the project
In my humble opinion, I think FAO needs to define a zero hunger change theory ( if not existing). This theory will draw inspiration from the representations to define the theories of their country programs and finally from these country program theories will draw inspiration from the theories of change specific to each project in the representation. This work will help to improve the logic of intervention and greatly facilitate evaluations at all levels. Like FPMIS for budget monitoring, such a portal could be developed to track changes. All programs and projects can be included in this portal with an alignment of expected results and expected changes to the changes that define zero hunger in its globality.
I think the gender dimension is very important in agriculture programs. In Senegal for example the GAFSP MMI pilot project is focused on women and youth, the most vulnerable to food acces and welbeing. Wathever the evaluator must do, the difference between rural women entrepreneurs who have generated incomes by commercialisation and women small agriculture who have generated nature incomes for their families' food needs in a small land needs to be considered. For many women in rural areas the agriculture activities are a support for families' needs and if you evaluate the results on the entrepreneurships side you risk to have a unadopted approch. Sometimes in some of speculation we find women entrepreneurs but for the majority the activity of women in agricultural sector in the rural areas is for families daily subsistence and for the evaluator it is very important to focus to the women contribution in respective families subsistences from her activities with quantitative and qualitative tools. It is too very important to invite men to give her means for women contribution in the families and to understand how they perceive the women activities in the families and outside the families. Many of the women lands are next to the house or the Community space (commonly called "champs de case")...
Au Sénégal, comme dans beaucoup de pays la disponibilité des données agricoles reste toujours un challenge. LA FAO Sénégal dans le cadre du programme GAFSP MMI, est entrain d'expérimenté une nouvelle initiative dans ce domaine pour faciliter la disponibilité des données du secteur agricole. le principal souci est l'agriculture des exploitation familiale ou des petits exploitants agricoles qui représente plus de 60% de la production agricole au Sénégal. Ainsi dans cette nouvelle initiative, les données sont cherchés à la base par le biais des organisations de producteurs. il 'agit de former des animateurs et superviseurs issus des organisations de producteurs en collecte de données quanti et quali d'évaluation et de suivi agricoles via des tablettes Android qui sont mis à leurs dispositions contenant l'application KOBOcollect. dans cette perspectives avec l'entrée des organisations de producteurs qui sont sous la supervision des Directions régionales de l'agriculture nous pensons établir un dispositif qui permettra d'avoir en permanence des données actualisées. Aujourd'hui, nous sommes en phase pilote de cette initiative dans la région de Tambacounda avec le projet GAFSP MMI inscrit dans une logique de renforcement de l'autonomisation des organisation de producteurs pour contribuer aux renforcements de la productivité des petits exploitants qui sont affiliés. la mise à l'échelle de cette nouvelle dispositive avec l'entrée OP, pourrait permettre au Sénégal de faire face à ce challenge de données de la production agricole au niveau national. Ainsi l'expérience pourrait aussi partager par d'autres de la sous région voire ailleurs dans le monde.