Internal Meta-Analysis Makes False-Positives Easier To Produce and Harder To Correct

22 Pages Posted: 14 Nov 2018

See all articles by Joachim Vosgerau

Joachim Vosgerau

Bocconi University

Uri Simonsohn

Ramon Llull University - ESADE Business School

Leif D. Nelson

University of California, Berkeley - Haas School of Business

Joseph P. Simmons

University of Pennsylvania - The Wharton School

Date Written: January 31, 2018

Abstract

In several recent papers, researchers have used or praised internal meta-analysis, the practice of statistically aggregating all studies in a paper, to arrive at a summary assessment of the evidence. Internal meta-analysis is argued to increase statistical power while solving the file-drawer problem. In this article, we show that internal meta-analysis rests on two critical assumptions: (1) that the meta-analysis includes every study that was conducted, and (2) that researchers attempted only one analysis per study (i.e., that p-hacking was completely absent). We demonstrate that even trivially minor violations of these assumptions invalidate internal meta-analysis. For example, p-hacking that only slightly increases the false-positive rate of individual studies, from 5% to 8%, increases the false-positive rate of a 5-study internal meta-analysis to 45%, and a 10-study internal meta-analysis to 82%. Making matters worse, it is prohibitively difficult to correct false-positive internal meta-analyses. We recommend to never draw inferences about the existence of an effect from an internal meta-analysis. We further suggest that both internal and external meta-analyses are useful only as an exploratory tool.

Suggested Citation

Vosgerau, Joachim and Simonsohn, Uri and Nelson, Leif D. and Simmons, Joseph P., Internal Meta-Analysis Makes False-Positives Easier To Produce and Harder To Correct (January 31, 2018). Available at SSRN: https://ssrn.com/abstract=3271372

Joachim Vosgerau (Contact Author)

Bocconi University ( email )

Via Sarfatti, 25
Milan, MI 20136
Italy

Uri Simonsohn

Ramon Llull University - ESADE Business School ( email )

Avinguda de la Torre Blanca, 59
Sant Cugat del Vall├Ęs, 08172
Spain

HOME PAGE: http://urisohn.com

Leif D. Nelson

University of California, Berkeley - Haas School of Business ( email )

545 Student Services Building, #1900
2220 Piedmont Avenue
Berkeley, CA 94720
United States

Joseph P. Simmons

University of Pennsylvania - The Wharton School ( email )

3733 Spruce Street
Philadelphia, PA 19104-6374
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
177
rank
162,308
Abstract Views
657
PlumX