The Influence of Hidden Researcher Decisions in Applied Microeconomics

42 Pages Posted: 19 May 2020

See all articles by Nick Huntington-Klein

Nick Huntington-Klein

California State University, Fullerton

Andreu Arenas

University of Barcelona

Emily Beam

National University of Singapore (NUS) - NUS Business School

Marco Bertoni

University of Padua

Jeffrey Bloem

University of Minnesota

Pralhad H Burli

Idaho National Laboratory

Naibin Chen

Pennsylvania State University

Paul L.E. Grieco

Pennsylvania State University - Department of Economics

Godwin Ekpe

Northern Illinois University

Todd Pugatch

Oregon State University; IZA

Martin Hugo Saavedra

Oberlin College - Department of Economics

Yaniv Stopnitzky

University of San Francisco - Department of Economics

Abstract

Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect's sign varied as well. The standard deviation of estimates across replications was 3-4 times the typical reported standard error.

Keywords: replication, metascience, research

JEL Classification: C81, C10, B41

Suggested Citation

Huntington-Klein, Nick and Arenas, Andreu and Beam, Emily and Bertoni, Marco and Bloem, Jeffrey and Burli, Pralhad H and Chen, Naibin and Grieco, Paul L.E. and Ekpe, Godwin and Pugatch, Todd and Saavedra, Martin Hugo and Stopnitzky, Yaniv, The Influence of Hidden Researcher Decisions in Applied Microeconomics. Available at SSRN: https://ssrn.com/abstract=3602409 or http://dx.doi.org/10.2139/ssrn.3602409

Nick Huntington-Klein (Contact Author)

California State University, Fullerton ( email )

800 N State College St
Fullerton, CA 92831
United States

Andreu Arenas

University of Barcelona

Gran Via de les Corts Catalanes, 585
Barcelona, 08007
Spain

Emily Beam

National University of Singapore (NUS) - NUS Business School ( email )

1 Business Link
Singapore, 117592
Singapore

Marco Bertoni

University of Padua ( email )

Via 8 Febbraio
Padova, Vicenza 2-35122
Italy

Jeffrey Bloem

University of Minnesota

Pralhad H Burli

Idaho National Laboratory

Naibin Chen

Pennsylvania State University

University Park
State College, PA 16802
United States

Paul L.E. Grieco

Pennsylvania State University - Department of Economics ( email )

University Park
State College, PA 16802
United States

Godwin Ekpe

Northern Illinois University

Todd Pugatch

Oregon State University ( email )

Corvallis, OR 97331
United States

HOME PAGE: http://https://sites.google.com/site/toddpugatch/

IZA ( email )

P.O. Box 7240
Bonn, D-53072
Germany

Martin Hugo Saavedra

Oberlin College - Department of Economics ( email )

Oberlin, OH 44074
United States

HOME PAGE: http://sites.google.com/view/martinsaavedra/

Yaniv Stopnitzky

University of San Francisco - Department of Economics ( email )

2130 Fulton Street
San Francisco, CA 94117-1080
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
167
Abstract Views
736
rank
226,356
PlumX Metrics