Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration

Published in Political Analysis: 2013, 21(1), 1-20

37 Pages Posted: 10 Feb 2020

See all articles by Macartan Humphreys

Macartan Humphreys

WZB Berlin Social Science Center

Raul Sanchez de la Sierra

University of Chicago

Peter Van der Windt

New York University (NYU) - New York University, Abu Dhabi

Date Written: May 19, 2012

Abstract

Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this “fishing” problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope for fishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete “mock report" of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but non-binding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but non-binding registration are discussed, the principle advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

Keywords: Congo Fishing, P-hacking, Pre Analysis Plan, Preregistration

JEL Classification: C93

Suggested Citation

Humphreys, Macartan and Sanchez de la Sierra, Raul and Van der Windt, Peter, Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration (May 19, 2012). Published in Political Analysis: 2013, 21(1), 1-20, Available at SSRN: https://ssrn.com/abstract=3520436

Macartan Humphreys (Contact Author)

WZB Berlin Social Science Center

Raul Sanchez de la Sierra

University of Chicago ( email )

1307 E 60th st
Chicago, IL Illinois 60637
United States
9174889151 (Phone)
60637 (Fax)

HOME PAGE: http://raulsanchezdelasierra.com

Peter Van der Windt

New York University (NYU) - New York University, Abu Dhabi ( email )

Abu Dhabi
United Arab Emirates

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
33
Abstract Views
243
PlumX Metrics