Methods Matter: P-Hacking and Causal Inference in Economics

28 Pages Posted: 17 Sep 2018

See all articles by Abel Brodeur

Abel Brodeur

University of Ottawa - Department of Economics; IZA Institute of Labor Economics

Nikolai Cook

University of Ottawa - Department of Economics

Anthony G. Heyes

University of London, Royal Holloway College - Department of Economics

Abstract

The economics 'credibility revolution' has promoted the identification of causal relationships using difference-in-differences (DID), instrumental variables (IV), randomized control trials (RCT) and regression discontinuity design (RDD) methods. The extent to which a reader should trust claims about the statistical significance of results proves very sensitive to method. Applying multiple methods to 13,440 hypothesis tests reported in 25 top economics journals in 2015, we show that selective publication and p-hacking is a substantial problem in research employing DID and (in particular) IV. RCT and RDD are much less problematic. Almost 25% of claims of marginally significant results in IV papers are misleading.

Keywords: research methods, causal inference, p-curves, p-hacking, publication bias

JEL Classification: A11, B41, C13, C44

Suggested Citation

Brodeur, Abel and Cook, Nikolai and Heyes, Anthony, Methods Matter: P-Hacking and Causal Inference in Economics. IZA Discussion Paper No. 11796, Available at SSRN: https://ssrn.com/abstract=3249910

Abel Brodeur (Contact Author)

University of Ottawa - Department of Economics ( email )

200 Wilbrod Street
Ottawa, Ontario K1N 6N5
Canada

HOME PAGE: http://https://sites.google.com/site/abelbrodeur/

IZA Institute of Labor Economics ( email )

P.O. Box 7240
Bonn, D-53072
Germany

Nikolai Cook

University of Ottawa - Department of Economics ( email )

Social Sciences Building Room 9005
Ottawa, Ontario K1N 6N5
Canada

Anthony Heyes

University of London, Royal Holloway College - Department of Economics ( email )

Royal Holloway College
Egham
Surrey, Surrey TW20 0EX
United Kingdom
44-(0)1784 44-3907 (Phone)
44-(0)1784 43-9534 (Fax)

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
202
Abstract Views
1,030
rank
187,576
PlumX Metrics