Inequality in Measuring Scholarly Success: Variation in the H-index Within and Between Disciplines

Overview

Journal PLoS One

Date 2025 Jan 24

PMID 39854341

Authors

Ryan Light

Aaron Gullickson

Jill Ann Harrison

Affiliations

Soon will be listed here.

Abstract

Scholars and university administrators have a vested interest in building equitable valuation systems of academic work for both practical (e.g., resource distribution) and more lofty purposes (e.g., what constitutes "good" research). Well-established inequalities in science pose a difficult challenge to those interested in constructing a parsimonious and fair method for valuation as stratification occurs within academic disciplines, but also between them. The h-index, a popular research metric, has been formally used as one such method of valuation. In this article, we use the case of the h-index to examine how the distribution of research metrics reveal within and between discipline inequalities. Using bibliometric data from 1960-2019 on over 50,000 high performing scientists-the top 2% most frequently cited authors-across 174 disciplines, we construct random effects within-between models predicting the h-index. Results suggest significant within-discipline variation in several forms, specifically sole-authorship and female penalties. Results also show that a sole authorship penalty plays a significant role in well-known between-discipline variation. Field-specific models emphasize the "apples-to-oranges," or incommensurable, property of cross-discipline comparison with significant heterogeneity in sole-authorship and female penalties within fields. In conclusion, we recommend continued caution when using the h-index or similar metrics for valuation purposes and the prioritization of substantive valuations from disciplinary experts.

References

Niles M, Schimanski L, McKiernan E, Alperin J . Why we publish where we do: Faculty publishing values and their relationship to review, promotion and tenure expectations. PLoS One. 2020; 15(3):e0228914. PMC: 7065820. DOI: 10.1371/journal.pone.0228914. View

Ioannidis J, Boyack K . Citation metrics for appraising scientists: misuse, gaming and proper use. Med J Aust. 2020; 212(6):247-249.e1. DOI: 10.5694/mja2.50493. View

Hirsch J . An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A. 2005; 102(46):16569-72. PMC: 1283832. DOI: 10.1073/pnas.0507655102. View

Forrester N . Mental health of graduate students sorely overlooked. Nature. 2021; 595(7865):135-137. DOI: 10.1038/d41586-021-01751-z. View

Parish A, Boyack K, Ioannidis J . Dynamics of co-authorship and productivity across different fields of scientific research. PLoS One. 2018; 13(1):e0189742. PMC: 5761855. DOI: 10.1371/journal.pone.0189742. View

Lanahan L, Graddy-Reed A, Feldman M . The Domino Effects of Federal Research Funding. PLoS One. 2016; 11(6):e0157325. PMC: 4915724. DOI: 10.1371/journal.pone.0157325. View

McKiernan E, Schimanski L, Munoz Nieves C, Matthias L, Niles M, Alperin J . Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations. Elife. 2019; 8. PMC: 6668985. DOI: 10.7554/eLife.47338. View

Lariviere V, Ni C, Gingras Y, Cronin B, Sugimoto C . Bibliometrics: global gender disparities in science. Nature. 2013; 504(7479):211-3. DOI: 10.1038/504211a. View

Hofstra B, Kulkarni V, Galvez S, He B, Jurafsky D, McFarland D . The Diversity-Innovation Paradox in Science. Proc Natl Acad Sci U S A. 2020; 117(17):9284-9291. PMC: 7196824. DOI: 10.1073/pnas.1915378117. View

10.

Fortunato S, Bergstrom C, Borner K, Evans J, Helbing D, Milojevic S . Science of science. Science. 2018; 359(6379). PMC: 5949209. DOI: 10.1126/science.aao0185. View

11.

Edwards M, Roy S . Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition. Environ Eng Sci. 2017; 34(1):51-61. PMC: 5206685. DOI: 10.1089/ees.2016.0223. View

12.

Ioannidis J, Baas J, Klavans R, Boyack K . A standardized citation metrics author database annotated for scientific field. PLoS Biol. 2019; 17(8):e3000384. PMC: 6699798. DOI: 10.1371/journal.pbio.3000384. View

13.

Mihaljevic H, Tullney M, Santamaria L, Steinfeldt C . Reflections on Gender Analyses of Bibliographic Corpora. Front Big Data. 2021; 2:29. PMC: 7931878. DOI: 10.3389/fdata.2019.00029. View

14.

Ioannidis J, Boyack K, Baas J . Updated science-wide author databases of standardized citation indicators. PLoS Biol. 2020; 18(10):e3000918. PMC: 7567353. DOI: 10.1371/journal.pbio.3000918. View

15.

Kelly C, Jennions M . The h index and career assessment by numbers. Trends Ecol Evol. 2006; 21(4):167-70. DOI: 10.1016/j.tree.2006.01.005. View

16.

Hicks D, Wouters P, Waltman L, de Rijcke S, Rafols I . Bibliometrics: The Leiden Manifesto for research metrics. Nature. 2015; 520(7548):429-31. DOI: 10.1038/520429a. View

17.

Huang J, Gates A, Sinatra R, Barabasi A . Historical comparison of gender inequality in scientific careers across countries and disciplines. Proc Natl Acad Sci U S A. 2020; 117(9):4609-4616. PMC: 7060730. DOI: 10.1073/pnas.1914221117. View

18.

Maggio L, Jeffrey A, Haustein S, Samuel A . Becoming metrics literate: An analysis of brief videos that teach about the h-index. PLoS One. 2022; 17(5):e0268110. PMC: 9075661. DOI: 10.1371/journal.pone.0268110. View

19.

Morgan A, Way S, Hoefer M, Larremore D, Galesic M, Clauset A . The unequal impact of parenthood in academia. Sci Adv. 2021; 7(9). PMC: 7904257. DOI: 10.1126/sciadv.abd1996. View

20.

Purvis A . The h index: playing the numbers game. Trends Ecol Evol. 2006; 21(8):422. DOI: 10.1016/j.tree.2006.05.014. View