99% Impossible: A Valid, or Falsifiable, Internal Meta-Analysis

Journal of Experimental Psychology: General

29 Pages Posted: 14 Nov 2018 Last revised: 12 Jul 2019

See all articles by Joachim Vosgerau

Joachim Vosgerau

Bocconi University

Leif D. Nelson

University of California, Berkeley - Haas School of Business

Uri Simonsohn

ESADE Business School; Ramon Llull University - ESADE Business School; ESADE Business School

Joseph P. Simmons

University of Pennsylvania - The Wharton School

Date Written: July 11, 2019

Abstract

Several researchers have relied on, or advocated for, internal meta-analysis, which involves statistically aggregating multiple studies in a paper to assess their overall evidential value. Advocates of internal meta-analysis argue that it provides an efficient approach to increasing statistical power and solving the file-drawer problem. Here we show that the validity of internal-meta-analysis rests on the assumption that no studies or analyses were selectively reported. That is, the technique is only valid if (1) all conducted studies were included (i.e., an empty file-drawer), and (2) for each included study, exactly one analysis was attempted (i.e., there was no p-hacking). We show that even very small doses of selective reporting invalidate internal-meta-analysis. For example, the kind of minimal p-hacking that increases the false-positive rate of one study to just 8% increases the false-positive rate of a 10-study internal meta-analysis to 83%. If selective reporting is approximately zero, but not exactly zero, then internal meta-analysis is invalid. To be valid, (1) an internal meta-analysis would need to exclusively contain studies that were properly pre-registered, (2) those pre-registrations would have to be followed in all essential aspects, and (3) the decision of whether to include a given study in an internal meta-analysis would have to be made before any of those studies are run.

Suggested Citation

Vosgerau, Joachim and Nelson, Leif D. and Simonsohn, Uri and Simmons, Joseph P., 99% Impossible: A Valid, or Falsifiable, Internal Meta-Analysis (July 11, 2019). Journal of Experimental Psychology: General, Available at SSRN: https://ssrn.com/abstract=3271372 or http://dx.doi.org/10.2139/ssrn.3271372

Joachim Vosgerau (Contact Author)

Bocconi University ( email )

Via Sarfatti, 25
Milan, MI 20136
Italy

Leif D. Nelson

University of California, Berkeley - Haas School of Business ( email )

545 Student Services Building, #1900
2220 Piedmont Avenue
Berkeley, CA 94720
United States

Uri Simonsohn

ESADE Business School ( email )

Av. de Pedralbes, 60-62
Barcelona, 08034
Spain

Ramon Llull University - ESADE Business School ( email )

Avinguda de la Torre Blanca, 59
Sant Cugat del Vallès, 08172
Spain

HOME PAGE: http://urisohn.com

ESADE Business School ( email )

Av. de Pedralbes, 60-62
Barcelona, 08034
Spain

Joseph P. Simmons

University of Pennsylvania - The Wharton School ( email )

3733 Spruce Street
Philadelphia, PA 19104-6374
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
505
Abstract Views
3,996
Rank
102,751
PlumX Metrics