Reproducibility in Management Science

60 Pages Posted: 29 Nov 2023

See all articles by Miloš Fišar

Miloš Fišar

Masaryk University

Ben Greiner

WU Vienna University of Economics and Business

Christoph Huber

WU Vienna University of Economics and Business

Elena Katok

University of Texas at Dallas

Ali Ozkes

Université Côte d'Azur - SKEMA Business School

Management Science Reproducibility Collaboration

Independent

Date Written: November 1, 2023

Abstract

With the help of more than 700 reviewers we assess the reproducibility of nearly 500 articles published in the journal Management Science before and after the introduction of a new Data and Code Disclosure policy in 2019. When considering only articles for which data accessibility and hard- and software requirements were not an obstacle for reviewers, the results of more than 95% of articles under the new disclosure policy could be fully or largely computationally reproduced. However, for almost 29% of articles at least part of the dataset was not accessible for the reviewer. Considering all articles in our sample reduces the share of reproduced articles to 68%. The introduction of the disclosure policy increased reproducibility significantly, since only 12% of articles accepted before the introduction of the disclosure policy voluntarily provided replication materials, out of which 55% could be (largely) reproduced. Substantial heterogeneity in reproducibility rates across different fields is mainly driven by differences in dataset accessibility. Other reasons for unsuccessful reproduction attempts include missing code, unresolvable code errors, weak or missing documentation, but also soft- and hardware requirements and code complexity. Our findings highlight the importance of journal code and data disclosure policies, and suggest potential avenues for enhancing their effectiveness.

Keywords: reproducibility, replication, crowd science

JEL Classification: A10, B40, M20

Suggested Citation

Fišar, Miloš and Greiner, Ben and Huber, Christoph and Katok, Elena and Ozkes, Ali and Reproducibility Collaboration, Management Science, Reproducibility in Management Science (November 1, 2023). Available at SSRN: https://ssrn.com/abstract=4620006 or http://dx.doi.org/10.2139/ssrn.4620006

Miloš Fišar

Masaryk University

Zerotinovo nam. 9
60177 Brno, 603 00
Czech Republic

Ben Greiner

WU Vienna University of Economics and Business ( email )

Welthandelsplatz 1, Building D1, 3rd Floor
Vienna, 1020
Austria

Christoph Huber (Contact Author)

WU Vienna University of Economics and Business ( email )

Welthandelsplatz 1, Building D1, 3rd Floor
Vienna, 1020
Austria

Elena Katok

University of Texas at Dallas ( email )

Jindal School of Management
800 W. Campbell Dr.
Richardson, TX 75080
United States

HOME PAGE: http://www.utdallas.edu/~ekatok/

Ali Ozkes

Université Côte d'Azur - SKEMA Business School

5 Quai Marcel Dassault
Suresnes, 92150
France

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
169
Abstract Views
712
Rank
314,682
PlumX Metrics