Definition and Diagnosis of Problematic Attrition in Randomized Controlled Experiments

Abstract

Attrition is the Achilles' Heel of the randomized experiment: It is fairly common, and it can completely unravel the benefits of randomization. Using the structural language of causal diagrams I demonstrate that attrition is problematic for identification of the average treatment effect (ATE) if -- and only if -- it is a common effect of the treatment and the outcome (or a cause of the outcome other than the treatment). I also demonstrate that whether the ATE is identified and estimable for all units in the experiment, or only for those units with observed outcomes, depends on two d-separation conditions. One of these is testable ex-post under standard experimental assumptions. The other is testable ex-ante so long as adequate measurement protocols are adopted. Missing at Random (MAR) assumptions are neither necessary nor sufficient for identification of the ATE.

Keywords: attrition, randomized controlled experiments, field experiments, causal diagrams, directed acyclic graphs, average treatment effect, nonparametric