The Road to False Positives: Sample Selection and Specification Choice in Randomized and Natural Experiments

During 2005-2007, the SEC conducted a randomized trial in which it removed shortsale restrictions from one-third of the Russell 3000 firms. Early studies found no effect of removing the restrictions on short interest, share returns, volatility, or price efficiency. In prior work, we confirm the lack of evidence for a natural causal channel that could explain these results (Black et al., 2024). Yet over 80 studies report evidence for a wide range of indirect effects on firms from the experiment. Given the lack of a causal channel, many of these results are likely to be false positives. We confirm that suspicion by closely reexamining the principal results from 10 of these studies using a simple specification with firm and year fixed effects. None of the results survive. We then examine best-match specifications that closely follow the sample selection, methodology and specification reported in the respective papers. Again, we mostly obtain null results. The gaps between reported and best match results reflects reported coefficients being much larger in magnitude, reported standard errors being much smaller than best-match, or both. The large gaps between best-match and reported results implies that the study authors made choices that are not evident from the explanation of their research design, which strongly affected the reported results. Our results suggest that researchers retain extensive discretion over the sample and model specification, even (as here) for a true randomized experiment. The choices in these studies produced statistically significant results when other reasonable choices would not. We draw lessons from our analysis for empirical practice and for ways in which researchers can find false positives results.

Keywords: natural experiments, causal channels, specification choice, Regulation SHO, SEC experiment, short-sale experiment

Suggested Citation: Suggested Citation

Black, Bernard S. and Desai, Hemang and Litvak, Kate and Yoo, Woongsun and Yu, Jeff Jiewei, The Road to False Positives: Sample Selection and Specification Choice in Randomized and Natural Experiments (November 17, 2024). SMU Cox School of Business Research Paper No. 25-2, Northwestern Law & Econ Research Paper No. 25-07, Available at SSRN: https://ssrn.com/abstract=5024145 or http://dx.doi.org/10.2139/ssrn.5024145