Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias

52 Pages Posted: 1 Oct 2018

See all articles by Christopher M. Snyder

Christopher M. Snyder

Dartmouth College - Department of Economics

Ran Zhuo

Harvard University - Department of Economics

Multiple version iconThere are 2 versions of this paper

Date Written: September 11, 2018

Abstract

The increasing demand for rigor in empirical economics has led to the growing use of auxiliary tests (balance, specification, over-identification, placebo, etc.) supporting the credibility of a paper's main results. We dub these "sniff tests" because standards for passing are subjective and rejection is bad news for the author. Sniff tests offer a new window into publication bias since authors prefer them to be insignificant, the reverse of standard statistical tests. Collecting a sample of nearly 30,000 sniff tests across 60 economics journals, we provide the first estimate of their aggregate probability-value (p-value) distribution. For the subsample of balance tests in randomized controlled trials (for which the distribution of p-values is known to be uniform absent publication bias, allowing reduced-form methods to be employed) estimates suggest that 45% of failed tests remain in the "file drawer" rather than being published. For the remaining sample with an unknown distribution of p-values, structural estimates suggest an even larger file-drawer problem, as high as 91%. Fewer significant sniff tests show up in top-tier journals, smaller tables, and more recent articles. We find no evidence of author manipulation other than a tendency to overly attribute significant sniff tests to bad luck.

Keywords: Publication Bias, Null Hypothesis, Specification Test, Balance Test, Placebo Test

JEL Classification: C18, A14, B41

Suggested Citation

Snyder, Christopher M. and Zhuo, Ran, Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias (September 11, 2018). Available at SSRN: https://ssrn.com/abstract=3247630 or http://dx.doi.org/10.2139/ssrn.3247630

Christopher M. Snyder (Contact Author)

Dartmouth College - Department of Economics ( email )

301 Rockefeller Hall
Hanover, NH 03755
United States
(603) 646-0642 (Phone)
(603) 646-2122 (Fax)

HOME PAGE: http://www.dartmouth.edu/~csnyder/

Ran Zhuo

Harvard University - Department of Economics ( email )

Littauer Center
Cambridge, MA 02138
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
15
Abstract Views
235
PlumX Metrics