logoRD.png
  • Article Galaxy Blog

August 29, 2019

Metrics Matter: Getting Top Value from Research Performance Data

Posted by: Mitja-Alexander Linss

Like it or not, the world of scholarly research is fiercely competitive. To be successful, a range of research stakeholders—from librarians to journal publishers to scientists—must continuously demonstrate the quality and impact of their research outputs. As a result, research performance metrics have become essential tools.

Perhaps most significantly, research performance metrics are used to show return on research value—in order to attract and secure more research funding.  

But getting funding isn't the only thing research metrics are used for. Here are just a few examples of how librarians and information managers at scientific corporations and academic institutions use performance metrics to support their institutional goals: 

  • Understand what's been done in a particular area of study
  • Identify the best opportunities for progressing research and making the greatest impact
  • Drive decisions on whether to allocate or withdraw personnel, space, and other resources
  • Build data that can be used to showcase and promote the institution externally

Research performance metrics are used to support career-level goals, too. Individual researchers and scientists, for example, use impact metrics to showcase their own scholarly influence, as well as to identify and attract collaborators. And some academic librarians even include research performance data in their dossiers when seeking a promotion (or if applicable, tenure).

Using Research Performance Metrics Effectively
Like any type of metric, research performance metrics are only worthwhile when they provide meaningful, accurate information. And extracting useful information from research performance metrics isn’t easy.

Certain metrics provide easy to understand data that's helpful when making certain types of quick decisions (e.g. using the at-a-glance Altmetric Donut to decide whether to purchase a specific peer-reviewed article). But for those who depend on research performance metrics to secure funding and inform more strategic decisions, extracting relevant, high-value information is a complex task. With so many types of metrics available, just choosing the right tools to use can be a challenge.  

Here’s a high-level look at some of the most popular research performance metrics available today.

TYPES OF PERFORMANCE METRICS: 

Journal Impact Factor (JIF)

TYPE: Journal-level (Citation-based)

OVERVIEW: Aims to reflect a journal’s performance by measuring the average number of citations received by articles in a journal during a two-year period.

CALCULATION: Journal X current year citations ÷ total number of articles Journal X published during the two previous years

PROS:

  • Simple calculation based on historical data

CONS:

  • Doesn’t adjust for the distribution of citations, increasing potential for skewed results

  • 2-year publication window is too short, resulting in significant variation from year to year

  • Not comparable across different subject areas due to different citation patterns among disciplines


CiteScore

TYPE: Journal-level (Citation-based)

OVERVIEW: Aims to reflect a journal’s performance by measuring the average citations per document that a title receives over a three-year period. It considers all content published in a journal (not just peer-reviewed articles).

CALCULATION: Total number of Journal X citations in a given year to content published in the last three years ÷ the total number of documents published in Journal X in the past three years

PROS:

  • Free, no subscription required

  • Increased transparency (numerator and denominator both include all document types).

CONS:

  • Creates bias against journals that publish a lot of rarely-cited content, like editorials, news, and letters

  • Not comparable across different subject areas due to different citation patterns among disciplines


Source Normalized Impact per Paper (SNIP)

TYPE: Journal-level (Citation-based)

OVERVIEW: Aims to reflect a journal’s performance, while accounting for the citation potential in different fields.

CALCULATION: Journal X citation count per paper ÷ citation potential in its subject area

PROS:

  • Allows for cross-discipline comparisons

CONS:

  • Normalization reduces transparency


Altmetrics

TYPE: Article-level

OVERVIEW: Aims to reflect an article’s impact by measuring the social activity around it.

CALCULATION: Tracks a wide range of online sources to capture the social activity around academic research and provides weighted scores based on the source type (e.g. Facebook vs. Twitter vs. a blog post vs. Google+)

PROS:

  • Social monitoring begins the moment an article is published

  • Holistic and nuanced view of attention, impact and influence

CONS:

  • Validation behind the weighting system isn’t clear

  • Tendency to focus on English-speaking sources


h-Index

TYPE: Author-level

OVERVIEW: Aims to reflect scholarly productivity and impact by measuring citations from an individual author’s publications.

CALCULATION: Counts the number of articles published by an author that have received at least that same number of citations (ex: an h-index of 12 means the author has published at least 12 papers that have each been cited at least 12 times)

PROS:

  • Measures career performance (not skewed by a small number of highly cited papers)

CONS:

  • Can be inconsistent based on which database is used for the calculation (e.g. larger databases can result in a higher h-index)

  • Can be skewed by self-citations

  • Not comparable across disciplines


NOTE: This chart is intended to provide a quick overview of the range of metrics available today. It does not provide a comprehensive list of metrics, nor does it detail all the pros and cons of each metric listed.

The Journal Impact Factor (JIF) has long been the default metric used for a variety of decision-making purposes, from acquisitions to promotions, which affect individuals and institutions alike. It’s often considered a proxy for quality, but the JIF has come under increasing scrutiny due to a number of pitfalls specific to this metric—and to citation-based metrics in general.

Now, with the recent surge of alternative performance metrics, librarians and information managers have a much larger toolbox at their disposal—which they can use to inform decision-making at a more granular level.

But the vast number of research performance metrics available today is both a blessing and a curse. The trick for those charged with tracking performance is to identify—and focus resources on—a combination of metrics (as well as mentions, downloads, reviews, etc.) that best aligns with the organization’s specific goals. 

We’d love to hear more about how you’re using performance metrics in your institutions. And if you're one of our customers, let us know how we can help facilitate your performance tracking and analysis.

 

Topics: research altmetrics scientists journal publishers performance metrics information managers librarians CiteScore JIF SNIP h-index