When and Why is Attrition a Problem in Randomized Controlled Experiments and How to Diagnose it

Abstract

Attrition is the Achilles’ Heel of the randomized experiment: it is fairly common, and it can unravel the benefits of randomization. This study considers when and why attrition is a problem, and how it can be diagnosed. The extant literature remains ambiguous because it relies on the language of probability, whereas problematic attrition depends on the underlying causal relations. This ambiguity arises because causation implies correlation but not vice versa. Using the structural causal language of directed acyclic graphs I show attrition is a problem when it is an active collider between the treatment and the outcome, or when the latent outcome is a mediator between the treatment and the attrition. Moreover, whether observed outcomes are representative of all outcomes, or only comparable across experimental arms, depends on two d-separation conditions. One of these is directly testable from the data.

Keywords: attrition, randomized controlled experiments, field experiments, causal diagrams, directed acyclic graphs, average treatment effect, nonparametric

JEL Classification: C9, C90, C93, C99, C42