Alternative science metrics
Who publishes most; which article is quoted where and how often? – For many years, everything in science has revolved around the so-called impact factors that determine who is successful and who is not. In the field of impact measurement, altmetrics can be understood as non-traditional metrics proposed as an alternative to more traditional citation impact metrics through evaluating the attention a publication receives in online media – including platforms such as Facebook and Twitter, but also other sources such as policy documents. Altmetrics assess a variety of citation and usage data of scientific contributions. The aim is to produce a multi-faceted image of the public perception and relevance of research results. Altmetrics are intended as a supplement to, not a replacement for, other impact measurements.
Where do altmetrics data come from?
The Austrian Science Fund FWF receives data from the UK company Altmetric.com, currently one of the most important altmetrics providers. The company tracks and collates 19 online data sources for mentions or uses of specified publications. Depending on type, they may include just one source (e.g. Twitter or Wikipedia) or many individual sources, such as the approximately 9,000 blogs analysed by the company. The overall result of this search related to one specific publication is presented in the shape of a “donut” (see image on the left), whose colours illustrate the heterogeneity of the sources of attention for a given publication. The data sources used by Altmetric.com are comprehensive and diverse, but not uncontroversial: one finds no empirical or conceptual justification for the selection of sources, and despite the wealth of data, many politically relevant sources at national/regional level are not included. The main focus lies on English-language sources, although several German-language and Austrian daily or weekly newspapers are included, such as the online versions of Der Spiegel, Der Standard or Die Presse.
What is – and what isn’t - the relevance of altmetrics?
The debate about the information value of altmetrics is ongoing. But they do obviously provide a broader picture of the impact of research output that goes beyond the confines of the science system and is not equivalent to purely scientific success. This type of impact is assessed as being the “societal impact”, although it is not yet entirely clear what exactly is measured in this field. One understands that mentions of publications in patents, textbooks and medical-clinical guidelines or policy documents are seen as a good way of capturing the societal impact of research. Mentions or reviews on Twitter or Facebook, on the other hand, are not necessarily an indicator of the relevance of research output but mainly serve to demonstrate the interest and curiosity of a specific readership. Against this backdrop, the greatly differentiated visualisation of the results by Altmetric.com does seem useful, but the aggregation of data to a total value seems questionable. The relevance of the magnitude of this value is hard to interpret: what does a value of 1,348 as shown in the picture above really mean? In addition to an explanation of the complicated computational basis, one would need some information about whether 1,348 constitutes an average or perhaps an outstanding value. This would presuppose a standardisation of such data. The difficulty of substantive interpretation is exacerbated by deficiencies in the empirical material. The total score seems dominated by Twitter data, whose usefulness is dubious. The ease of manipulation of certain indicators (such as download or read rates) is another problem. The Altmetric company is aware of these weaknesses and makes them an issue for discussion. A final judgement seems premature, since altmetrics are still very much at the beginning of their development. So, what do altmetrics tell us? Fundamentally, they demonstrate the attention paid to research output by various groups in society. Moreover, certain indicators have the potential of demonstrating “societal impact”, while other – quite prominent – sources of altmetrics such as Twitter or Facebook seem inappropriate. And what is it that altmetrics obviously cannot do? – Measuring the quality of research.
Why is the FWF testing altmetrics?
The legitimacy of a research funding organisation is based, inter alia, on the scientific quality and the scientific impact of the funded research, which can be captured by analysing citations in scientific journals. – Such analyses are already conducted by the FWF. While societal relevance is hard to capture, demonstrating this type of relevance instead of merely postulating it is becoming more and more important. Hence, the FWF is testing altmetrics in an attempt to find out whether a direct impact of basic research beyond the confines of the science system can be proven. It is also interesting to see whether and in what way alternative indicators represent an additional useful input to assessing the relevance of basic research. While altmetrics do have weaknesses, they also show potential. The inherent diversity and possibilities thus make testing them worthwhile.
What the FWF is not going to use altmetrics for!
The testing of Altmetric.com relates to funded research in its totality. The data material will not be used to assess individual scholars or have an impact on future funding decisions.
Personal details Ralph Reimann is a member of the FWF’s scientific team in the ”Strategy – Policy, Evaluation, Analysis” Department with a focus on data analyses. More information at: http://orcid.org/0000-0002-5151-4154
More information FWF Altmetrics results
This contribution is based on the following publications:
