Having been involved in reviewing and evaluating agricultural research for development projects and programs for several decades, I would like to share some observations.
Value of bibliometrics and almetrics
In spite of some of the negative coverage of bibliometrics in current literature, they have an important function in evaluating the quality of published agriculturel for development research papers. Published papers already have passed a high quality threshold as they have been peer-reviewed by experienced scientists. Most international journals have rejection rates of over 90% - only the highest quality papers are published. Bibliometrics provide a means to further assess quality through number of citations, journal Impact factor (IF), quartile ranking and h-indices of authors among other bibliometrics. Citations and h-indices reflect the quality of the published research within the scientific community. Altmetrics demonstrate interest in the paper among the authors’ peer group. The recent publication by Runzel et al (2021) clearly illustrates how combinations of bibliometrics and altmetrics can be successfully used to evaluate the quality of almost 5000 papers published by the CGIAR Research Programs during 2017-2020. The Technical Note – Bibliometric analysis to evaluate quality of science in the context of the One CGIAR greatly expands the number of potential bibliometrics that could be used to evaluate quality.
Are there alternatives to citations and IF? The giant scientific publishing companies such as Elsevier use citations and IFs to monitor the quality of their journals. Higher IF translates into higher sales of journal subscriptions. As such companies own most of the scientific journals, any alternatives would need to be endorsed by them – this is unlikely as they seem to be happy with the status quo. Currently there do not appear to be any recognized alternatives. A recent paper by Slafer and Savin (2020) notes that the quality of a journal (IF) as a proxy for the likely impact of a paper is acceptable when the focus of the evaluation is on recently published papers.
Importance of qualitative indicators
Qualitative indicators of research quality are just as important as bibliometrics and other quantitative indicators and should always be used alongside bibliometrics. The 2020 evaluations of the CGIAR Research Programs (https://cas.cgiar.org/publications) effectively used a range of qualitative indicators to evaluate inputs, processes and outputs under the umbrella of the Quality of Research for Development Framework using the assessment elements: relevance, credibility, legitimacy and effectiveness.
IDRC recently revised its quality of research assessment – firmly anchored in qualitative assessment – to more effectively assess quality in a development context (IDRC, 2017). Of interest is the move to use indicators that look at positioning for use. IDRC has successfully used the RQ+ Instrument to evaluate 170 research studies (McClean and Sen, 2019).
Subjectivity in qualitative evaluation cannot be eliminated but it can be reduced by employing a team of evaluators and by better defining the criteria, indicators and descriptions.
Scientists often raise the issue that they are most interested in the impact of their research rather than its qualitative assessment. Evaluation of effectiveness in the context of positioning for use allows assessment of potential impact through indicators such as stakeholder engagement, gender integration, networking and links with policy makers.
Integrating quantitative (including bibliometrics) and qualitative indicators
The on-going development and refining of quantitative and qualitative indicators provides the potential to integrate them to provide more comprehensive evaluation of quality of research for development. This is an exciting area for future evaluations.
McClean R. K. D. and Sen K. (2019) Making a difference in the real world? A meta-analysis of the quality of use-oriented research using the Research Quality Plus approach. Research Evaluation 28: 123-135.
Runzel M., Sarfatti P. and Negroustoueva S. (2021) Evaluating quality of science in CGIAR research programs: Use of bibliometrics. Outlook on Agriculture 50: 130-140.
Slafer G. and Savin R. (2020) Should the impact factor of the year of publication or the last available one be used when evaluating scientists? Spanish Journal of Agricultural Research 18: 10pgs.
Jill Lenné
Editor in Chief, Outlook on Agriculture and Independent Consultant
RE: How to evaluate science, technology and innovation in a development context?
Evaluation of quality of research for development
Having been involved in reviewing and evaluating agricultural research for development projects and programs for several decades, I would like to share some observations.
Value of bibliometrics and almetrics
In spite of some of the negative coverage of bibliometrics in current literature, they have an important function in evaluating the quality of published agriculturel for development research papers. Published papers already have passed a high quality threshold as they have been peer-reviewed by experienced scientists. Most international journals have rejection rates of over 90% - only the highest quality papers are published. Bibliometrics provide a means to further assess quality through number of citations, journal Impact factor (IF), quartile ranking and h-indices of authors among other bibliometrics. Citations and h-indices reflect the quality of the published research within the scientific community. Altmetrics demonstrate interest in the paper among the authors’ peer group. The recent publication by Runzel et al (2021) clearly illustrates how combinations of bibliometrics and altmetrics can be successfully used to evaluate the quality of almost 5000 papers published by the CGIAR Research Programs during 2017-2020. The Technical Note – Bibliometric analysis to evaluate quality of science in the context of the One CGIAR greatly expands the number of potential bibliometrics that could be used to evaluate quality.
Are there alternatives to citations and IF? The giant scientific publishing companies such as Elsevier use citations and IFs to monitor the quality of their journals. Higher IF translates into higher sales of journal subscriptions. As such companies own most of the scientific journals, any alternatives would need to be endorsed by them – this is unlikely as they seem to be happy with the status quo. Currently there do not appear to be any recognized alternatives. A recent paper by Slafer and Savin (2020) notes that the quality of a journal (IF) as a proxy for the likely impact of a paper is acceptable when the focus of the evaluation is on recently published papers.
Importance of qualitative indicators
Qualitative indicators of research quality are just as important as bibliometrics and other quantitative indicators and should always be used alongside bibliometrics. The 2020 evaluations of the CGIAR Research Programs (https://cas.cgiar.org/publications) effectively used a range of qualitative indicators to evaluate inputs, processes and outputs under the umbrella of the Quality of Research for Development Framework using the assessment elements: relevance, credibility, legitimacy and effectiveness.
IDRC recently revised its quality of research assessment – firmly anchored in qualitative assessment – to more effectively assess quality in a development context (IDRC, 2017). Of interest is the move to use indicators that look at positioning for use. IDRC has successfully used the RQ+ Instrument to evaluate 170 research studies (McClean and Sen, 2019).
Subjectivity in qualitative evaluation cannot be eliminated but it can be reduced by employing a team of evaluators and by better defining the criteria, indicators and descriptions.
Scientists often raise the issue that they are most interested in the impact of their research rather than its qualitative assessment. Evaluation of effectiveness in the context of positioning for use allows assessment of potential impact through indicators such as stakeholder engagement, gender integration, networking and links with policy makers.
Integrating quantitative (including bibliometrics) and qualitative indicators
The on-going development and refining of quantitative and qualitative indicators provides the potential to integrate them to provide more comprehensive evaluation of quality of research for development. This is an exciting area for future evaluations.
References
IDRC (2017) Towards Research Excellence for Development: The Research Quality Plus Assessment Instrument. Ottawa, Canada. <https://www.idrc.ca/sites/default/files/sp/Documents%20EN/idrc_rq_asses…;
McClean R. K. D. and Sen K. (2019) Making a difference in the real world? A meta-analysis of the quality of use-oriented research using the Research Quality Plus approach. Research Evaluation 28: 123-135.
Runzel M., Sarfatti P. and Negroustoueva S. (2021) Evaluating quality of science in CGIAR research programs: Use of bibliometrics. Outlook on Agriculture 50: 130-140.
Slafer G. and Savin R. (2020) Should the impact factor of the year of publication or the last available one be used when evaluating scientists? Spanish Journal of Agricultural Research 18: 10pgs.
Jill Lenné
Editor in Chief, Outlook on Agriculture and Independent Consultant