Reproducibility of Empirical Results: Evidence from 1,000 Tests in Finance

38 Pages Posted: 6 Apr 2022

See all articles by Christophe Pérignon

Christophe Pérignon

HEC Paris - Finance Department

Olivier Akmansoy

CNRS

Christophe Hurlin

University of Orleans

Anna Dreber

Stockholm School of Economics - Department of Economics

Felix Holzmeister

University of Innsbruck - Department of Economics

Juergen Huber

University of Innsbruck

Magnus Johannesson

Stockholm School of Economics - Department of Economics

Michael Kirchler

University of Innsbruck

Albert J. Menkveld

Vrije Universiteit Amsterdam

Michael Razen

University of Innsbruck

Utz Weitzel

VU University Amsterdam

Date Written: March 22, 2022

Abstract

We analyze the computational reproducibility of more than 1,000 empirical answers to six research questions in finance provided by 168 international research teams. Surprisingly, neither researcher seniority, nor the quality of the research paper seem related to the level of reproducibility. Moreover, researchers exhibit strong overconfidence when assessing the reproducibility of their own research and underestimate the difficulty faced by their peers when attempting to reproduce their results. We further find that reproducibility is higher for researchers with better coding skills and for those exerting more effort. It is lower for more technical research questions and more complex code.

Keywords: computational reproducibility, open science, credibility of research, multi-analyst study, data-availability policy, scientific publishing

JEL Classification: C80, C87

Suggested Citation

Pérignon, Christophe and Akmansoy, Olivier and Hurlin, Christophe and Dreber, Anna and Holzmeister, Felix and Huber, Juergen and Johanneson, Magnus and Kirchler, Michael and Menkveld, Albert J. and Razen, Michael and Weitzel, Utz, Reproducibility of Empirical Results: Evidence from 1,000 Tests in Finance (March 22, 2022). Available at SSRN: https://ssrn.com/abstract=4064172 or http://dx.doi.org/10.2139/ssrn.4064172

Christophe Pérignon (Contact Author)

HEC Paris - Finance Department ( email )

1 rue de la Liberation
Jouy-en-Josas Cedex, 78351
France

Olivier Akmansoy

CNRS ( email )

France

Christophe Hurlin

University of Orleans ( email )

Université d'Orléans
Rue de Blois B.P. 6739 45
France

Anna Dreber

Stockholm School of Economics - Department of Economics ( email )

P.O. Box 6501
Sveavagen 65
S-113 83 Stockholm
Sweden

Felix Holzmeister

University of Innsbruck - Department of Economics ( email )

Universitaetsstrasse 15
Innsbruck, 6020
Austria

HOME PAGE: http://www.holzmeister.biz

Juergen Huber

University of Innsbruck ( email )

Universitätsstraße 15
Innsbruck, Innsbruck 6020
Austria

Magnus Johanneson

Stockholm School of Economics - Department of Economics ( email )

P.O. Box 6501
Sveavagen 65
S-113 83 Stockholm
Sweden
+46 8 736 9443 (Phone)

HOME PAGE: http://www.hhs.se/Faculty/showperson.htm?personid=198

Michael Kirchler

University of Innsbruck ( email )

Innsbruck
Austria

Albert J. Menkveld

Vrije Universiteit Amsterdam ( email )

De Boelelaan 1105
Amsterdam, 1081HV
Netherlands
+31 20 5986130 (Phone)
+31 20 5986020 (Fax)

Michael Razen

University of Innsbruck ( email )

Innsbruck
Austria

Utz Weitzel

VU University Amsterdam ( email )

De Boelelaan 1105
Amsterdam
Netherlands

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
229
Abstract Views
735
rank
182,074
PlumX Metrics