Comparing Performance of Methods to Deal with Differential Attrition in Lottery Based Evaluations

52 Pages Posted: 7 Oct 2016

See all articles by Gema Zamarro

Gema Zamarro

University of Arkansas - Department of Education Reform; Center for Economic and Social Research (CESR)

Kaitlin Anderson

Michigan State University

Jennifer L. Steele

American University

Trey Miller

RAND Corporation

Date Written: September 2016

Abstract

In randomized controlled trials, it is common for attrition rates to differ by lottery status, jeopardizing the identification of causal effects. Inverse probability weighting methods (Hirano et al, 2003; Busso et al., 2014) and estimation of informative bounds for the treatment effects (e.g. Lee, 2009; Angrist et al., 2006) have been used frequently to deal with differential attrition bias. This paper studies the performance of various methods by comparing the results using two datasets: a district-sourced dataset subject to considerable differential attrition, and an expanded state-sourced dataset with much less attrition, differential and overall. We compared the performance of different methods to correct for differential attrition in the district dataset, as well as we conducted simulation analyses to assess the sensitivity of bounding methods to their underlying assumptions. In our application, methods to correct differential attrition induced bias, whereas the unadjusted district level results were closer and more substantively similar to the estimated effects in the benchmark state dataset. Our simulation exercises showed that even small deviations from the underlying assumptions in bounding methods proposed by Angrist et al. (2006) increased bias in the estimates. In practice, researchers often do not have enough information to verify the extent to which these underlying assumptions are met, so we recommend using these methods with caution.

Keywords: Differential Attrition, Bounding Methods, Simulation

JEL Classification: C18, C15, C90

Suggested Citation

Zamarro, Gema and Anderson, Kaitlin and Steele, Jennifer L. and Miller, Trey, Comparing Performance of Methods to Deal with Differential Attrition in Lottery Based Evaluations (September 2016). EDRE Working Paper No. 2016-15. Available at SSRN: https://ssrn.com/abstract=2849193

Gema Zamarro (Contact Author)

University of Arkansas - Department of Education Reform ( email )

201 Graduate Education Building
Fayetteville, AR 72701
United States

Center for Economic and Social Research (CESR) ( email )

635 Downey Way
Los Angeles, CA 90089-3332
United States

HOME PAGE: http://works.bepress.com/gema_zamarro/

Kaitlin Anderson

Michigan State University ( email )

201H Erickson Hall
620 Farm Lane
East Lansing, MI 48933
United States

Jennifer L. Steele

American University ( email )

4400 Massachusetts Ave, NW
Washington, DC 20016
United States

Trey Miller

RAND Corporation ( email )

1776 Main Street
P.O. Box 2138
Santa Monica, CA 90407-2138
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
40
Abstract Views
375
PlumX Metrics