Operationalisation of the Quality of Science criterion in the 2020 CGIAR Research Programmes (CRP) reviews
The 2020 CRP reviews focused on three criteria: Quality ofScience, Effectiveness (in the OECD-DAC meaning),and Future Orientation/Sustainability.
For the assessment of the Quality of Science criterion, were adopted two elements: scientific credibility and legitimacy, which are two of the four elements constituting the Quality of Research for Development Framework (Qo4RD) by the Independent Science and Partnership Council (ISPC) in 2017 and successively refreshed by the Independent Science for Development Council (ISDC). The four elements which constitute the QoR4D framework being relevance, scientific credibility, legitimacy, and effectiveness, andtodayform the basis for a common frame of reference across CGIAR.
Scientific credibility requires that research findings be robust and that sources of knowledge be dependable and sound. It includes a clear demonstration that data used are accurate, that the methods used to procure the data are fit for purpose, and that findings are clearly presented and logically interpreted. It recognizes the importance of good scientific practice, such as peer review.
In the 2020 CRP reviews the evaluation of scientific credibility covered outputs, where these consisted mainly of published results, germplasm, digital tools, and technical reports, as well as leadership, research staff, processes, and incentives for achieving and maintaining the high scientific credibility of those outputs. The quantitative bibliometric analysis was fully integrated with the qualitative analysis of these other elements.
Assessing scientific credibility also included, among other things, the track records of research teams, use of state-of-the-art research literature and methods, and novelty.
Legitimacy means that the research process is fair and ethical and perceived as such. This feature encompasses the ethical and fair representation of all involved and consideration of the interests and perspectives of intended users. It suggests transparency, sound management of potential conflicts of interest, recognition of the responsibilities that go with public funding, genuine involvement of partners in co-design, and recognition of partners’ contributions. Partnerships are built on trust and mutual commitment to delivery of agreed-upon outcomes.
In the 2020 CRP reviews, the evaluation of legitimacy focused on the analysis of how CRPs partnerships were effectively built and functioning on the basis of mutual understanding, trust, and commitment, with clear recognition of each one’s perspective, needs, role and contribution. Robust multi-stakeholder partnerships should be good indicators for assessing research legitimacy. Assessments of fairness and the ethical aspects of research implementation have been standard features of the reviews.
The 2020 CRP Reviews demonstrated that by adopting mixed-method approach, it is possible to evaluate quality of science using rigorous quantitative bibliometrics analysis in combination with the qualitative assessment of many other important elements of research for development initiatives.
Paolo Sarfatti
Team Leader Nutrition Research FacilityOperationalisation of the Quality of Science criterion in the 2020 CGIAR Research Programmes (CRP) reviews
The 2020 CRP reviews focused on three criteria: Quality of Science, Effectiveness (in the OECD-DAC meaning), and Future Orientation/Sustainability.
For the assessment of the Quality of Science criterion, were adopted two elements: scientific credibility and legitimacy, which are two of the four elements constituting the Quality of Research for Development Framework (Qo4RD) by the Independent Science and Partnership Council (ISPC) in 2017 and successively refreshed by the Independent Science for Development Council (ISDC). The four elements which constitute the QoR4D framework being relevance, scientific credibility, legitimacy, and effectiveness, and todayform the basis for a common frame of reference across CGIAR.
Scientific credibility requires that research findings be robust and that sources of knowledge be dependable and sound. It includes a clear demonstration that data used are accurate, that the methods used to procure the data are fit for purpose, and that findings are clearly presented and logically interpreted. It recognizes the importance of good scientific practice, such as peer review.
In the 2020 CRP reviews the evaluation of scientific credibility covered outputs, where these consisted mainly of published results, germplasm, digital tools, and technical reports, as well as leadership, research staff, processes, and incentives for achieving and maintaining the high scientific credibility of those outputs. The quantitative bibliometric analysis was fully integrated with the qualitative analysis of these other elements.
Assessing scientific credibility also included, among other things, the track records of research teams, use of state-of-the-art research literature and methods, and novelty.
Legitimacy means that the research process is fair and ethical and perceived as such. This feature encompasses the ethical and fair representation of all involved and consideration of the interests and perspectives of intended users. It suggests transparency, sound management of potential conflicts of interest, recognition of the responsibilities that go with public funding, genuine involvement of partners in co-design, and recognition of partners’ contributions. Partnerships are built on trust and mutual commitment to delivery of agreed-upon outcomes.
In the 2020 CRP reviews, the evaluation of legitimacy focused on the analysis of how CRPs partnerships were effectively built and functioning on the basis of mutual understanding, trust, and commitment, with clear recognition of each one’s perspective, needs, role and contribution. Robust multi-stakeholder partnerships should be good indicators for assessing research legitimacy. Assessments of fairness and the ethical aspects of research implementation have been standard features of the reviews.
The 2020 CRP Reviews demonstrated that by adopting mixed-method approach, it is possible to evaluate quality of science using rigorous quantitative bibliometrics analysis in combination with the qualitative assessment of many other important elements of research for development initiatives.