The Reproducibility of Economics Research: A Case Study

43 Pages Posted: 18 Jan 2023

See all articles by Sylverie Herbert

Sylverie Herbert

Banque de France

Hautahi Kingi

Cornell University, Department of Economics, Students

Flavio Stanchi

Cornell University

Lars Vilhuber

Cornell University - Department of Economics; U.S. Census Bureau - Center for Economic Studies

Date Written: December 10, 2021

Abstract

Given the importance of reproducibility for the scientific ethos, more and more journals have pushed for transparency of research through data availability policies. If the introduction and implementation of such data policies improve the availability of researchers' code and data, what is the impact on reproducibility? We describe and present the results of a large reproduction exercise in which we assess the reproducibility of research articles published in the American Economic Journal: Applied Economics, which has implemented a data availability policy since 2005. Our replication success rate is relatively moderate, with 37.78% of replication attempts successful. 68 of 162 eligible replication attempts successfully replicated the article's analysis (41.98%) conditional on non-confidential data. A further 69 (42.59%) were at least partially successful. A total of 98 out of 303 (32.34%) relied on confidential or proprietary data, and were thus not reproducible by this project. We also conduct several bibliometric analyses of reproducible vs. non-reproducible articles and show that replicable papers do not provide citation bonuses for authors.

Keywords: Replication, Reproducibility, Transparency, Replicability, Journal Policies

JEL Classification: B41, C80, C81, C87, C88

Suggested Citation

Herbert, Sylverie and Kingi, Hautahi and Stanchi, Flavio and Vilhuber, Lars, The Reproducibility of Economics Research: A Case Study (December 10, 2021). Banque de France Working Paper, Forthcoming, Available at SSRN: https://ssrn.com/abstract=4325149 or http://dx.doi.org/10.2139/ssrn.4325149

Sylverie Herbert (Contact Author)

Banque de France ( email )

31 rue Croix des Petits Champs
Paris, NY 75001
France

Hautahi Kingi

Cornell University, Department of Economics, Students ( email )

Ithaca, NY
United States

HOME PAGE: http://hautahikingi.com/

Flavio Stanchi

Cornell University

Lars Vilhuber

Cornell University - Department of Economics ( email )

Ithaca, NY
United States

U.S. Census Bureau - Center for Economic Studies ( email )

4700 Silver Hill Road
Washington, DC 20233
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
85
Abstract Views
360
Rank
521,195
PlumX Metrics