Split-Sample Strategies for Avoiding False Discoveries

70 Pages Posted: 4 Jul 2017 Last revised: 1 May 2023

See all articles by Michael L. Anderson

Michael L. Anderson

U.C. Berkeley - Department of Agricultural and Resource Economics

Jeremy Magruder

University of California, Berkeley - Department of Agricultural & Resource Economics

Date Written: June 2017

Abstract

Preanalysis plans (PAPs) have become an important tool for limiting false discoveries in field experiments. We evaluate the properties of an alternate approach which splits the data into two samples: An exploratory sample and a confirmation sample. When hypotheses are homogeneous, we describe an improved split-sample approach that achieves 90% of the rejections of the optimal PAP without requiring preregistration or constraints on specification search in the exploratory sample. When hypotheses are heterogeneous in priors or intrinsic interest, we find that a hybrid approach which prespecifies hypotheses with high weights and priors and uses a split-sample approach to test additional hypotheses can have power gains over any pure PAP. We assess this approach using the community-driven development (CDD) application from Casey et al. (2012) and find that the use of a hybrid split-sample approach would have generated qualitatively different conclusions.

Suggested Citation

Anderson, Michael L. and Magruder, Jeremy, Split-Sample Strategies for Avoiding False Discoveries (June 2017). NBER Working Paper No. w23544, Available at SSRN: https://ssrn.com/abstract=2996305

Michael L. Anderson (Contact Author)

U.C. Berkeley - Department of Agricultural and Resource Economics ( email )

207 Giannini Hall, MC 3310
Berkeley, CA 94720-3310
United States

Jeremy Magruder

University of California, Berkeley - Department of Agricultural & Resource Economics ( email )

Berkeley, CA 94720
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
41
Abstract Views
576
PlumX Metrics