Sniff Tests as a Screen in the Publication Process: Throwing Out the Wheat with the Chaff

44 Pages Posted: 17 Sep 2018 Last revised: 21 Jun 2023

See all articles by Christopher M. Snyder

Christopher M. Snyder

Dartmouth College - Department of Economics; National Bureau of Economic Research

Ran Zhuo

Harvard University - Department of Economics

Multiple version iconThere are 2 versions of this paper

Date Written: September 2018

Abstract

The increasing demand for empirical rigor has led to the growing use of auxiliary tests (balance, specification, over-identification, placebo, etc.) in assessing the credibility of a paper’s main results. We dub these “sniff tests” because rejection is bad news for the author and standards for passing are informal. Using a sample of nearly 30,000 published sniff tests collected from scores of economics journals, we study the use of sniff tests as a screen in the publication process. For the subsample of balance tests in randomized controlled trials, our structural estimates suggest that the publication process removes 46% of significant sniff tests, yet only one in ten of these is actually misspecified. For other tests, we estimate more latent misspecifiation and less removal. Surprisingly, more authors would be justified in attributing significant sniff tests to random bad luck.

Suggested Citation

Snyder, Christopher M. and Zhuo, Ran, Sniff Tests as a Screen in the Publication Process: Throwing Out the Wheat with the Chaff (September 2018). NBER Working Paper No. w25058, Available at SSRN: https://ssrn.com/abstract=3250604

Christopher M. Snyder (Contact Author)

Dartmouth College - Department of Economics ( email )

301 Rockefeller Hall
Hanover, NH 03755
United States
(603) 646-0642 (Phone)
(603) 646-2122 (Fax)

HOME PAGE: http://www.dartmouth.edu/~csnyder/

National Bureau of Economic Research ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Ran Zhuo

Harvard University - Department of Economics ( email )

Littauer Center
Cambridge, MA 02138
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
23
Abstract Views
408
PlumX Metrics