Internal Meta-Analysis Makes False-Positives Easier To Produce and Harder To Correct
22 Pages Posted: 14 Nov 2018
Date Written: January 31, 2018
In several recent papers, researchers have used or praised internal meta-analysis, the practice of statistically aggregating all studies in a paper, to arrive at a summary assessment of the evidence. Internal meta-analysis is argued to increase statistical power while solving the file-drawer problem. In this article, we show that internal meta-analysis rests on two critical assumptions: (1) that the meta-analysis includes every study that was conducted, and (2) that researchers attempted only one analysis per study (i.e., that p-hacking was completely absent). We demonstrate that even trivially minor violations of these assumptions invalidate internal meta-analysis. For example, p-hacking that only slightly increases the false-positive rate of individual studies, from 5% to 8%, increases the false-positive rate of a 5-study internal meta-analysis to 45%, and a 10-study internal meta-analysis to 82%. Making matters worse, it is prohibitively difficult to correct false-positive internal meta-analyses. We recommend to never draw inferences about the existence of an effect from an internal meta-analysis. We further suggest that both internal and external meta-analyses are useful only as an exploratory tool.
Suggested Citation: Suggested Citation