Reproducibility of Empirical Results: Evidence from 1,000 Tests in Finance

Abstract

We analyze the computational reproducibility of more than 1,000 empirical answers to six research questions in finance provided by 168 international research teams. Surprisingly, neither researcher seniority, nor the quality of the research paper seem related to the level of reproducibility. Moreover, researchers exhibit strong overconfidence when assessing the reproducibility of their own research and underestimate the difficulty faced by their peers when attempting to reproduce their results. We further find that reproducibility is higher for researchers with better coding skills and for those exerting more effort. It is lower for more technical research questions and more complex code.

Keywords: computational reproducibility, open science, credibility of research, multi-analyst study, data-availability policy, scientific publishing

JEL Classification: C80, C87