What are the attributes of research excellence? There are several of them, broadly consensual: research carried out according to good scientific practices, following widely accepted and verifiable ethical standards, validated by a significant number of peers, bringing important advances to its field of knowledge and to society.
Once the attributes were fulfilled, everything would be ok, right? It would, if attributes existed regardless of context, including the time/space in which they are claimed. Now, many would ask: how to deny universal concepts such as ethics, impact, best practices, validation by those who are recognized? It’s not about denying it, of course, but just recognizing that the metrics need to be adjusted.
We accept with some normality bibliometric indicators to measure the quality and define, to a large extent, the excellence of the research. Bibliometric indicators say important things. Scientometrics and bibliometrics have evolved a great deal in the present century. They are a fundamental axis of evidence-based science evaluations.
References that vary in time and space
The numbers of scientific publications databases are superlative. The three major databases in the world, Web of Science (WoS) by Clarivate Analytics, Scopus by Elsevier, and Dimensions by Digital Science, collect, store, and provide information from thousands of journals, millions of publications and researchers and billions of citations.
Singh et al. (2021) [i] compared the numbers and characteristics of these three databases. In mid-2020, WoS had 13,600 indexed journals with more than 13.2 million articles; Scopus had more than 40.3 thousand journals with more than 18 million publications; and Dimensions had 77,400 journals and over 28 million publications. The overlaps and specificities indicated important variations (Singh et al., 2021):
- the journals present in the WoS database represented 1/3 of the journals indexed by the Scopus database, and less than 1% of WoS was not covered by the Scopus database;
- the journals in the Scopus database represented half of those indexed in the Dimensions database, and less than 3% of the publications in Scopus were not in Dimensions.
Dimensions is the most recent database, launched in 2018, and contains more than twice the number of journals indexed in Scopus (launched in 2004 and the second largest database) and more than five times those indexed in WoS (created, as such, in 1997) [ii]. There are 35,500 “new” journals. So, accepting that all databases index journals with a serious editorial policy, the horizon of publication possibilities has expanded considerably in recent years.
Dimensions covers a higher percentage of social, arts, and humanities journals: 8%, 11.4% and 17.9% respectively for WoS, Scopus and Dimensions.
What happened? Were we referring to incomplete databases or what?
The different publishing dynamics (articles, reviews, books and chapters, annals, etc.) in different areas of knowledge are notorious. For articles and reviews, in the three major databases, life sciences account for 44% of publications, on average; Exact and technological sciences together add up to so much more. These numbers alone indicate different publishing dynamics.
As we known, in the social sciences and in the arts and humanities, the most frequent publications are in books, book chapters, conference annals and policies and artistic production documents.
In a recent study, Bin et al. (2022) [iii], the analyses of publications and citations from a universe of more than 21,000 doctors who studied their doctorate with and without scholarship grants in São Paulo, revealed substantial differences in publication patterns: while doctors who had a doctoral scholarship, when compared to doctors who did not have scholarships, have published an average of 2.1 times more articles in journals in the areas of life and exact sciences; no difference was found in the areas of social, humanities, linguistics, literature and arts among those same groups.
On the other hand, in these areas, scholarships from former FAPESP (São Paulo Research Foundation), CAPES (Coordination for the Improvement of Higher Education Personnel) and CNPq (National Council for Science and Technology Development) in the areas of social and humanities, published between four and seven times more books and book chapters than their counterparts who studied for a doctorate without a scholarship.
It is a totally different pattern of scientific production. This pattern has obvious reflections on the number of citations, impact factor and H index indicators.
are superlative.
So, is there a problem to tackle?
Indicators of publication in journals, impact factor, citations (including those weighted by area of knowledge and temporality, such as the field weighted citation impact from Scival/Scopus), now considered benchmarks for evaluating research, are obviously important.
It’s not about demonizing them. Rather, it is about improving the representation of what excellence in research is, that today is limited by the way in which the assessment system has progressed.
The resulting problems are several and are well reported in the literature. The Leiden Manifesto, from 2015, is a reference that summarizes the main weaknesses of a ruler that works for some measures but does not work for many others.
Despite initiatives such as DORA, which will be 10 years old in May 2023, the Leiden Manifesto and dozens of others asking for more and better indicators, we continue to measure the quality of research in any area and in any space by the number of publications, the journal in which the article is published, the number of citations and the H index. [iv]
While this subject does not effectively enter the agenda of development agencies and scientific journals, nothing or little will change.
The good news is that there are already signs that go by the name of Responsible Research Assessment, whose main initiative today is the Research Assessment Reform Agreement. [v]
Research Assessment Reform:
In December 2022, the Coalition to Advance Research Assessment to implement the Agreement on Research Assessment Reform, document published at the end of July of the same year by around 350 organizations representing more than 40 countries. The Agreement opened for adhesions in September of that year. Its first paragraph already indicates the mission:
“As signatories of this Agreement, we agree on the need to reform research assessment practices. Our vision is that the assessment of research, researchers and research organizations recognizes the diverse outputs, practices and activities that maximize the quality and impact of research. This requires basing assessment primarily on qualitative judgement, for which peer review is central, supported by responsible use of quantitative indicators. Among other purposes, this is fundamental for: deciding which researchers to recruit, promote or reward, selecting which research proposals to fund, and identifying which research units and organizations to support”.
The ten commitments of the Agreement are:
Core commitments:
1. Recognize the diversity of contributions to, and careers in research, in accordance with the needs and nature of the research.
2. Base research assessment primarily on qualitative assessment for which peer review is central, supported by responsible use of quantitative indicators.
3. Abandon inappropriate [iv] uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index.
4. Avoid the use of rankings of research organizations in research assessment.
Support Commitments:
5. Commit resources to reforming research assessment as is needed to achieve the organizational changes committed to.
6. Review and develop research assessment criteria, tools and processes.
7. Raise awareness of research assessment reform and provide transparent communication, guidance, and training on assessment criteria and processes as well as their use.
8. Exchange practices and experiences to enable mutual learning within and beyond the Coalition.
9. Communicate progress made on adherence to the Principles and implementation of the Commitments.
10. Evaluate practices, criteria and tools based on solid evidence and the state-of-the-art in research on research, and make data openly available for evidence gathering and research.
As we can see, the Agreement does not propose little. The question is: to what extent will the culture of promoting and recognizing researchers and research organizations change and, by extension, the meaning of research excellence?
There is a long and winding road ahead. In the timeframe proposed by the Agreement, signatories must adopt the core commitments after one year of accession and demonstrate that they have implemented at least one cycle of the new assessment process within five years.
Whether the Agreement, among other initiatives in the same direction, will or will not lead to a change in the criteria of quality and excellence in the world of research, will depend on several factors, all of them linked to the world of research itself.
In this context, more research of research, or more science of science, is necessary in the world and in Brazil. Without it, we won’t know what’s going on, and we won’t be able to influence our paths.
Author
Sergio Salles-Filho is a professor at the Department of Science and Technology Policy at the Institute of Geosciences at Unicamp and general coordinator of the project “Research and Innovation Research: Indicators, Methods and Evidence of Impacts”, funded by the São Paulo Research Foundation (FAPESP Process 2021/15.091-8).
References
[i] Singh, V. K., Singh, P., Karmakar, M., Leta, J., & Mayr, P. (2021). The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis. Scientometrics, 126(6), 5113–5142. https://doi.org/10.1007/s11192-021-03948-5
[ii] The origin of WoS dates back to the mid-1960s, with the Science Citation Index, followed by several other databases that were later consolidated
[iii] Bin, A., Salles-Filho, S., Spatti, A. C., Mena-Chalco, J. P., & Colugnati, F. A. B. (2022). How much does a Ph.D. scholarship program impact an emerging economy research performance? Scientometrics. https://doi.org/10.1007/s11192-022-04487-3
[iv] The case of the ‘Qualis Periódicos’ journal classification by Capes, used to determine the importance of journals in which we publish in Brazil and to assess the excellence of postgraduate programs, is a Brazilian initiative that seems to try to combine local specificities with the journal’s impact factor assigned by international sources. It’s difficult to determine whether these are well-intentioned efforts poorly executed. The result is a chimera that is neither one thing nor the other – and, on top of that, difficult to eliminate. Publishing in a Qualis A1 or A2 journal has become an end in itself in many postgraduate programs. In this case, the responsibility lies not only with Capes but also with the scientific community that is involved.
[v] For other initiatives in the same direction, see Curry, S., De Rijcke, S., Hatch, A., Pillay, D. G., Van der Weijden, I., & Wilsdon, J. The changing role of funders in responsible research assessment: Progress, obstacles and the way ahead. RoRI Working Paper, 3. 2020.
[vi] The meaning of the term ‘inappropriate’ is not clear in the Agreement.