The Influence of Hidden Researcher Decisions in Applied Microeconomics

42 Pages Posted: 19 May 2020

See all articles by Nick Huntington-Klein

Nick Huntington-Klein

California State University, Fullerton

Andreu Arenas

University of Barcelona

Emily Beam

University of Vermont - Department of Economics; University of Michigan at Ann Arbor

Marco Bertoni

University of Padua

Jeffrey Bloem

University of Minnesota

Pralhad H Burli

Idaho National Laboratory

Naibin Chen

Pennsylvania State University

Paul L.E. Grieco

Pennsylvania State University - Department of Economics

Godwin Ekpe

Northern Illinois University

Todd Pugatch

Oregon State University; IZA

Martin Hugo Saavedra

Rutgers University

Yaniv Stopnitzky

University of San Francisco - Department of Economics

Abstract

Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect's sign varied as well. The standard deviation of estimates across replications was 3-4 times the typical reported standard error.

Keywords: replication, metascience, research

JEL Classification: C81, C10, B41

Suggested Citation

Huntington-Klein, Nick and Arenas, Andreu and Beam, Emily and Beam, Emily and Bertoni, Marco and Bloem, Jeffrey and Burli, Pralhad H and Chen, Naibin and Grieco, Paul L.E. and Ekpe, Godwin and Pugatch, Todd and Saavedra, Martin Hugo and Stopnitzky, Yaniv, The Influence of Hidden Researcher Decisions in Applied Microeconomics. IZA Discussion Paper No. 13233, Available at SSRN: https://ssrn.com/abstract=3602409 or http://dx.doi.org/10.2139/ssrn.3602409

Nick Huntington-Klein (Contact Author)

California State University, Fullerton ( email )

800 N State College St
Fullerton, CA 92831
United States

Andreu Arenas

University of Barcelona

Gran Via de les Corts Catalanes, 585
Barcelona, 08007
Spain

Emily Beam

University of Vermont - Department of Economics ( email )

347 Old Mill
Burlington, VT 05405-4160
United States

University of Michigan at Ann Arbor ( email )

500 S. State Street
Ann Arbor, MI 48109
United States

Marco Bertoni

University of Padua ( email )

Jeffrey Bloem

University of Minnesota

Pralhad H Burli

Idaho National Laboratory

Naibin Chen

Pennsylvania State University

University Park
State College, PA 16802
United States

Paul L.E. Grieco

Pennsylvania State University - Department of Economics ( email )

University Park
State College, PA 16802
United States

Godwin Ekpe

Northern Illinois University

Todd Pugatch

Oregon State University ( email )

Corvallis, OR 97331
United States

HOME PAGE: http://https://sites.google.com/site/toddpugatch/

IZA ( email )

P.O. Box 7240
Bonn, D-53072
Germany

Martin Hugo Saavedra

Rutgers University ( email )

New Brunswick, NJ
United States

Yaniv Stopnitzky

University of San Francisco - Department of Economics ( email )

2130 Fulton Street
San Francisco, CA 94117-1080
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
238
Abstract Views
1,226
Rank
241,422
PlumX Metrics