The Disruption Index Suffers From Citation Inflation and Is Confounded by Shifts in Scholarly Citation Practice

Abstract

Measuring the rate of innovation in academia and industry is fundamental to monitoring the efficiency and competitiveness of the knowledge economy. To this end, a disruption index (CD) was recently developed and applied to publication and patent citation networks (Funk & Owen-Smith, 2017; Wu et al., 2019; Park et al., 2023). Here we show that CD systematically decreases over time due to secular growth in research and patent production, following two distinct mechanisms unrelated to innovation – one structural and the other behavioral. The structural explanation follows from ‘citation inflation’ (Petersen et al., 2018), an inextricable feature of real citation networks attributable to increasing reference list lengths, which causes CD to systematically decrease. We demonstrate this causal link by way of mathematical deduction, computational simulation, multi-variate regression, and quasi- experimental juxtaposition of the disruptiveness of PNAS versus PNAS Plus articles, which differ primarily in their article lengths. The behavioral explanation reflects shifts associated with techno-social factors (e.g. self-citation practices) that increase the rate of triadic closure in citation networks and confounds efforts to measure disruptive innovation using CD. Combined, these two mechanisms render CD unsuitable for cross-temporal analysis, and thereby call into question a growing body of research employing CD to measure innovation trends and identify co-factors associated with team assembly.

Keywords: Disruption index, citation inflation, confounded variable, measurement error, Research Evaluation, Citation Analysis, Science Policy

JEL Classification: D39, J24, J44, C40