How can we measure the relationship between science and policies? Insights from Overton

As a colleague professor from Unicamp used to say — who, unfortunately, is no longer with us — in academia, specialization isn´t a prerogative of knowledge; ignorance can be specialized too. He used to say this when noticed something important that we overlooked.

We are largely unaware of the impacts that Brazilian science has been producing, in our own country and worldwide. 

Brazilian scientific publications are referenced in documents produced by 1,028 organizations specialized in formulating and influencing public policies. We are talking about government entities, think tanks, non-governmental organizations (NGOs) and intergovernmental organizations (IGOs) from 127 countries. Their research papers on topics of national and global interest make references to around 50,000 scientific articles authored by at least one Brazilian researcher (mapped by the Overton database, covering the period from 1945 to 2023).

These organizations include the Institute of Applied Economic Research (IPEA), the World Bank, the United Nations Educational, Scientific and Cultural Organization (UNESCO) and the Intergovernmental Panel on Climate Change (IPCC), among others.

Prominent researchers from institutions such as the University of São Paulo (USP), State University of Campinas (Unicamp), Federal University of Rio Grande do Sul (UFRGS), Federal University of Rio de Janeiro (UFRJ), Oswaldo Cruz Foundation (Fiocruz), Federal University of São Paulo (UNIFESP), Federal University of Minas Gerais (UFMG) and many others are referenced in these policy documents.

Understanding the intersection between politics and science has never been as crucial as it is today. This challenge becomes even more complex when we consider the problems and social transformations occurring on different scales, such as climate change, poverty, inequalities, technological developments and their impacts. It is a fact that politics influences science, and science influences politics. The issue that greatly worries and engages funding agencies, science and technology institutions and policy-makers lies in how to measure and report the outcomes of this interaction.

How can we generate assessments, indicators, and information that measure the impacts of research beyond academic circles?

Altmetrics,  an emerging field of study, offers some potential solutions to this question. The alternative metrics measure how the research results are shared online (e.g., social media) and discussed within the political domain, setting agendas and guiding decision-making processes.

These metrics also quantify the portion of articles from publicly-funded projects that are referenced in policy documents. Let’s consider the following example: over the past decade, 106 projects were concluded under the Global Climate Change Research Program, a funding initiative by the São Paulo Research Foundation (FAPESP) aimed at promoting research on the intersection of science and climate policies. These projects resulted in the publication of 1,044 articles, of which 11% were referenced in parliamentary transcripts, documents and reports from NGOs, think tanks, and other policy literature. This percentage is twice as great as the global average citation rate of 5%, (calculated by studies conducted using the Overton database).

Proxies for assessing this relationship are currently feasible due to databases that provide information that can be used for institutions, editors, researchers, financiers and other organizations, that now can establish the connection between academic research and the political, technological, and social spheres. Noteworthy platforms that cross data and show the relationship between science and public policies include Altmetric.com, PlumX Metrics, Dimensions, and Overton.

The Altmetric.com platform, part of the Digital Science group, was launched in 2012. Since then, it has been monitoring a wide range of alternative metrics, encompassing over 35 million research outputs, including journal articles, datasets, images, white papers, and reports. Additionally, it tracks 191 million mentions across various platforms such as Twitter, Facebook, patents, news websites, blogs, Wikipedia and policy documents.

PlumX, founded in 2012 and later acquired by Elsevier, gathers metrics related to citations, usage, downloads, mentions, and social media activity. It also has a feature that shows policy citations in scholarly articles.

The Dimensions database, developed in 2018 as part of the Digital Science group, alongside Scopus and Web of Science, is one of the world’s primary academic, scientific, and technological databases. Powered by artificial intelligence, Dimensions offers comprehensive data and analytical tools covering publications, citations, research funding, altmetrics, clinical trials, patents, and policy documents, providing a thorough view of the research lifecycle.[1]

Overton, released in 2019 by the same founder of Altmetric.com, Euan Adie, is currently the largest collection of political documents from diverse sources worldwide. Its main objective is “to assist users in finding, understanding, and measuring their influence on government policies”.[2]

What are the strengths and limitations of Overton?

The Overton database traces extensive political coverage from over 1,500 sources [3], a substantially greater number in comparison to other databases [4]. Moreover, the database covers various fields of knowledge [5]. The platform also maps influences between academic research and public policies, namely scholarly publications citations in policy documents and policy documents citations in public policies. This comprehensive approach facilitates a broader assessment of research impact and offers valuable insights for evidence-based policy-making.

This can be accomplished through conventional quantitative analysis — comparing variables such as field of knowledge, publication year, funding, number of authors, and affiliated institutions [6] — and qualitative studies, focused on mapping topics and themes [7] .

For instance, the COVID-19 pandemic has sparked numerous debates regarding the dissemination, visibility, and transparency of scientific production. Overton could shed light on various aspects related to the influence (or lack) of research findings in the formulation of public health policies. Indeed, several studies that utilized information and indicators from the database have demonstrated its potential in assessing the reach of publications and their observed effects across academic, social, and political spheres.

Yin et al. [8] have shown that many policy documents used during the pandemic to address health emergencies heavily rely on recent, peer-reviewed, and high-impact scientific literature. Similarly, Bornmann et al. [9] conclude that scientific evidence has been incorporated into policy documents concerning critical decisions in international climate diplomacy. Although these studies are relatively recent, their aim is to measure the intersection of academic and political engagement by exploring which policy sources cite specific research, the geographic prevalence of research citations, and the funders associated with research mentioned in policy documents.

Despite its advantages, the Overton database has some limitations. These include potential biases in data coverage and sampling [8]. For instance, citation data may contain “false positives” when articles are referenced on other platforms without linking back to the original source [10]. Additionally, the platform may encounter challenges in disambiguating author names and only maps publications with digital identifiers. It is important to approach the use of alternative metrics with caution and critical thinking, considering that the concept (altmetrics) is relatively new. Comparing results across different providers can be challenging, and biases toward specific knowledge areas and regions may exist. Another crucial issue is regarding the ownership of these tools. The majority of these platforms are privately owned and foreign.

Tools like Overton contribute to reducing our knowledge gaps regarding the impact of scientific research and enhancing our understanding of how society makes use of  scientific knowledge. They open up possibilities for research focused on “research on research” or the “science of science policy.” Such tools hold particular relevance for funding agencies, research organizations, higher education institutions, and government bodies, as evidence-based decision-making should guide the development of public policies and strategic plans for Research, Development and Innovation.

Expectations are that the scientific community and society  will engage in extensive debates regarding the influence of research on public policies and the significance of digital tools and measurement methodologies. Gradually, we are paving the way to generate evaluations, indicators, and information that capture the impact of research beyond its immediate sphere of production.