Using synthesis and meta-analysis to make the most of evaluative evidence: what is your experience?

Dear Members,

Thanks to those who contributed to the discussion on the use of synthesis and meta-analysis in development evaluation. The exchange supported my preparation for the What Works Global Summit 2019 (wwgs2019.org), where synthesis and meta-analyses are discussed as tools for designing, implementing and assessing programmes and policies. A synthesis is the integration of existing knowledge and findings relevant to a topic, and has as its main objective to increase the applicability of evaluation findings and develop new knowledge through the integration process. It is promoted as an approach that addresses the challenge of "information overload", delivering products that distil relevant evidence for decision-making.

Here are the main issues shared by participants: 

• Synthesis work needs to be focused and pragmatic in order to attain its goal.
• Evaluation documents needs to be easily available, and the researchers need to interact with key stakeholders throughout the preparation process.
• Meta-analyses of evaluations allows the identification of structural themes that affect performance, which is of interest to donors and programme managers.
• A more holistic approach is required to enhance the engagement and outreach of the learning, i.e. by organizing dedicated workshops to discuss the findings of the evaluation synthesis.
• Meta-analysis can provide very valuable insights on the performance of a programme or approach, and guide primary data collection. It can however be very time-consuming and thus not always feasible to undertake within the timeframe of an evaluation.
• Meta-analysis can be an important tool for decision-makers but is not well known among evaluators. More training on its usage could help broadening awareness and application of the approach.

Taking on this last point, EvalForward will organize a webinar soon for members interested in learning more on synthesis and meta-analysis. Stay tuned!

Carlos

Dears, greetings to you all,

The little I will like to contribute is that to attain a goal by using synthetic approach you need to be pragmatist.

And also to have ready access to document and focus people, without focus people a project is nothing but just a written document so the people involved counts. 

Cheers.

Farmers conferences, schools, and clubs are widely used best practices since the 1990s, they are part of the farming system communication approach

Elamin 

Dears

I am trying to link a best practice close to what is mentioned in Chitra’s comment (targeting rural poor). The practice I am sharing might help in engaging poor in development in general. 

The Practice is as follows: years ago (2012) we conducted two farmer conferences as one of the knowledge sharing methods we were testing as part of a knowledge sharing research. The project was funded by IFAD through IDRC. It proved to be the best method among 5 other methods. In the two conferences with 6 months time gap, farmers were the heroes of the podium, where they shared their experience and success through various ways: short videos, presentations and even singing (Zajal in Arabic).

Farmer conferences are very rare, only Syria did it long time ago.

Farmer conferences proved to be a great method and welcomed by both farmer and agricultural experts who attended the event.

Best

Regards

Dear Carlos,

In IFAD’s Independent Office of Evaluation, we prepare every year a synthesis report based on a meta-analysis of project and country evaluations conducted in the previous year as well as independent evaluation ratings from 2002 to today. This Annual Report on Results and Impact of IFAD Operations (ARRI) aims to: i) report on results and impacts from IFAD operations, and to ii) identify lessons and systemic issues to be addressed in IFAD’s programmes.

The ARRI allows management to engage with a comprehensive yet synthetic overview of the Organisation’s performance and achievements and to compare IFAD’s outcomes with those of other international financial institutions.

To prepare the ARRI we conduct quantitative analyses on all final evaluations in the previous 10 years and qualitative analysis of evaluations conducted in the last year. All evaluations, their ratings and other data are included in the ARRI database, which is a building block of our knowledge management system.

Through the meta-analyses of evaluations, themes that are important to IFAD’s performance emerge which the Executive Board requests the Evaluation Office to focus on. Therefore, each ARRI has a “learning theme” and in the case of 2018 this was the targeting of the rural poor, a very relevant means by which we can contribute to the Sustainable Development Goals.  Based on the latest ARRI recommendations, IFAD’s board confirmed our focus on the poorest in remote rural areas and reviews of our targeting methods at programme and project level are ongoing.

ARRI is the flagship of our evaluation office and therefore attracts management attention. One lesson I can share from this years’ experience is that the importance of the learning theme (targeting of the rural poor) prompted us to adopt a more holistic approach to the development of the Report, by organising a Conference on Rural Inequalities which allowed to enhance the engagement and outreach of the learning on targeting the rural poor and addressing inequalities, beyond the scope of the ARRI itself.

Chitra

Dear Carlos,

Just a few supplementary remarks on the subject; strictly speaking, they may not seem relevant, but I think one has to be a pragmatist in order to use a synthetic approach to attain a goal. In my view, pragmatism is implicit to synthesis.

Therefore, taking every factor that has a bearing on a successful outcome becomes a logical necessity every synthesis should embody. This entails a comprehensive analysis of not only documented data, but also the specific external contextual variables like the needs and capacities of the potential beneficiaries (real people in the target area) and the resources at one's disposal to implement and sustain a given project or plan. Obviously, this has to be undertaken before planning a project based on a holistic synthesis of the relevant facts as revealed by the pre-project analysis. At this point, I think it would be very useful to carry out a pre-implementation evaluation of the planned project so that its strengths and weaknesses could be ascertained before actual implementation.

If this is tenable, then meta-analysis by itself would not  be sufficient to underwrite the success of a project, and it may under certain conditions give one a false sense of completeness and correctness that could be undesirable. Hope this would be of some use.

cheers!

Lal.

My concern is Ecological capacity of the definite area/spatial. Here comes the Remote Sensing, because it help in land capacity assessment, air quality & disaster. This is missing in all NGO based agro analysis. 

Dear Malika,

Thanks for sharing your experience with document analysis. It is indeed a great way to have early insights into the effectiveness of a programme or policy, although -depending on the amount of materials for review- it could also be a very demanding task.

You raised a very good point regarding the issue of accessibility to documents. With the advent of the internet it is assumed that information is becoming globally available, while digitalization is making reports and research more and more accessible through online means. 

What is the perspective in the global south? Are government (evaluation) reports and research from academia easier to access? Are they available in a format and language that make them suitable for synthesis and meta-analysis?

Best regards,

Carlos

In order to have a full picture of the Government gender approach in an agricultural value chain, we used mixed methods approach coupled with literature review and analysis of Ministerial strategies document.

What gave us most useful insights were the document analysis and focus group.

Document analysis requires access to the documents and is demanding

Dear All,

A colleague has shared with me a link to a series of synthesis reports of impact assessments on various agricultural topics done by the CGIAR:

https://ispc.cgiar.org/publications/all?keywords=&year=All&type=1571&workstream=1568

Hope you find it useful too!

Best regards,

Carlos

Dear Olivier and Lal,

Thank you very much for your contributions.

Regarding a definition of synthesis (or synthetic approach), for the purpose of this discussion, we can define it as the process of reviewing, assessing and synthesising existing literature or data to produce a series of outputs (products and services).

A first step in this process is to review the quality of the literature or data that will be aggregated to ensure that is comparable and meet the research protocol requirements. Afterwards, the synthesis is conducted often by academic disciplinary experts, but can also be done by inter- or transdisciplinary working groups or evaluators drawing on knowledge from across academia and beyond, the latter to ensure a comprehensive analysis and avoid the pitfals raised by Lal.

There are many guidelines out there on how to conduct synthesis reviews, especially in the areas of health and education. Perhaps the ones better know and applied in the field of agriculture and rural development are those developed by 3IE and Campbell Collaboration, accessible at this link: https://www.3ieimpact.org/our-expertise/synthesis

Best regards,

Carlos

I am happy to see this most timely question raised at this time, soon after the previous discussion on the difficulties in ensuring adequate funding for agriculture.

A synthetic approach by definition, involves paying due attention to every influence, positive and negative that bears on achieving a clearly defined objective. In my experience, this seems to be the most difficult thing to achieve even when sufficient material and guidance is clearly available. Let me be explicit.

First of all, there is a common belief that evaluation ought to be on some result with reference to a plan or a 'project specification' if you will. This is akin to checking whether some product meets a manufacturer's specifications and costings. This approach seems to be attractive because of its 'clear-cut' and scientific' nature. But, how useful is this in real life?

When applied, this method would evaluate as successful any 'prestige project' as long as it meets the project specifications and budget. Its inadequacy lies in that it ignores taking into account its utility to its supposed beneficiaries. accruing Such potential benefits depends on the beneficiaries' need for it and their ability to use the project product with sufficient skill. An up-to-date cardiology unit in a remote area of a poor country where health personnel a problem is an example of such a potentially successfully completed but rather a useless project.

A synthetic approach would have ascertained the greatest health needs of the people of such an area, the available man-power, ensured a sustainable funding mechanism, maintenance  capabilities before a project design is undertaken. I firmly believe what is most important is such a pre-project evaluation before one can carry out a meaningful post-project one.

You would have noticed that in a holistic synthetic approach one distinguishes clearly between the 'theoretical' project on paper and the real one that represents people doing their assigned tasks. Without the latter, a project is nothing but paper, but when what is on paper takes into account not only the abilities, but also the frailties of the people involved, it would be justifiable to evaluate a project plan a very useful guide to action.

I know my views are not those of the majority of the experts. But as a synthetist and an analyst, I think my approach is justifiable, for it looks at a project as a set of actions by some people  to enable a target group to achieve some desirable goal in a sustainable way.

Thank you.

Cheers!

Lal Manavado.

Greetings to all of you,

I know what is meta-analysis and how it is an important tool to the decision-maker but I have never used this tool. I would like to learn a lot about how to make it. It would be better if you can give a guideline or training on its usage for interested people.

Thank you