Forking Paths in Empirical Studies

48 Pages Posted: 5 Jan 2022 Last revised: 10 Feb 2023

Date Written: February 9, 2023

Abstract

We propose a theoretical framework that characterizes the diversity of outcomes in empirical studies, depending on the nature and number of design choices that researchers make. We outline several ways to exploit the multiplicity of outcomes. Our ideas are illustrated in two studies. The first is an exercise of equity premium prediction with ten decision layers. We find that each additional degree of freedom in the protocol expands the average range of t-statistics by at least 30%. The second study pertains to portfolio sorts and shows that resorting to forking paths instead of bootstrapping in multiple testing greatly raises the bar in order to reach significant results. In the first case, at the 5% confidence level, the threshold for statistics is 4.5, while in the second, it is at least 8.2, a bar much higher than those currently used in the literature.

Keywords: p-hacking, publication bias, design choices, empirical protocol, model averaging, multiple testing

Suggested Citation

Coqueret, Guillaume, Forking Paths in Empirical Studies (February 9, 2023). Available at SSRN: https://ssrn.com/abstract=3999379 or http://dx.doi.org/10.2139/ssrn.3999379

Guillaume Coqueret (Contact Author)

EMLYON Business School ( email )

23 Avenue Guy de Collongue
Ecully, 69132
France

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
352
Abstract Views
1,533
Rank
162,434
PlumX Metrics