A Note on Dropping Experimental Subjects Who Fail a Manipulation Check

16 Pages Posted: 31 Oct 2015 Last revised: 10 Jan 2016

See all articles by Peter M. Aronow

Peter M. Aronow

Yale University - Department of Political Science

Jonathon Baron

Yale University - Department of Political Science

Lauren Pinson

Yale University - Department of Political Science

Date Written: October 14, 2015

Abstract

Dropping subjects after a post-treatment manipulation check is common practice across the social sciences, presumably to restrict estimates to a subpopulation of subjects who understand the experimental prompt. We show that this practice can lead to serious bias and argue for a focus on what is revealed without discarding subjects. Generalizing results developed in Lee (2009) and Zhang and Rubin (2003) to the case of multiple treatments, we provide sharp bounds for potential outcomes among those who would pass a manipulation check regardless of treatment assignment. These bounds may have large or infinite width, implying that this inferential target is often out of reach. As an application, we replicate Press, Sagan and Valentino (2013) with a design that does not drop subjects that failed the manipulation check and show that the findings are likely stronger than originally reported. We conclude with suggestions for practice, namely corrections to the experimental design.

Keywords: causal inference, randomized experiments, attrition, manipulation checks, partial identification, potential outcomes

JEL Classification: C42, C9

Suggested Citation

Aronow, Peter Michael and Baron, Jonathon and Pinson, Lauren, A Note on Dropping Experimental Subjects Who Fail a Manipulation Check (October 14, 2015). Available at SSRN: https://ssrn.com/abstract=2683588 or http://dx.doi.org/10.2139/ssrn.2683588

Peter Michael Aronow (Contact Author)

Yale University - Department of Political Science ( email )

P.O. Box 208301
New Haven, CT 06520-8269
United States

Jonathon Baron

Yale University - Department of Political Science ( email )

Box 208269
New Haven, DC 06520-8269
United States

Lauren Pinson

Yale University - Department of Political Science ( email )

Box 208269
New Haven, DC 06520-8269
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
406
rank
69,433
Abstract Views
2,213
PlumX Metrics