Learning, Private Information, and the Economic Evaluation of Randomized Experiments

Abstract

Randomized experiments (REs) are viewed as the "gold standard" for the treatment evaluation, but many REs are plagued by attrition or non-compliance, even among subjects receiving the more effective treatment. This paper constructs an economic model of decision-making in which individuals make utility maximizing choices that provides a rich framework for evaluating REs. We estimate the subject's utility associated with the receipt of alternative treatments, as revealed by dropout or compliance behavior, to evaluate treatment effectiveness. Utility is a function of both the "publicly observed" outcomes that are typically the focus of evaluation studies, and treatment side effects that are the private information of the subject. Participants enter the RE uncertain of treatment effectiveness and often the treatment received, and update their prior beliefs over the course of the experiment when deciding whether to drop out. We use the framework to analyze an influential AIDS clinical trial, ACTG 175, which has been used to tout the benefits of combination therapies for AIDS over the use of AZT alone. However, our analysis indicates that for many subjects, AZT yields the highest level of utility, despite having the smallest impact on the publicly observed outcome of the study, the patient's CD4 count. Significant and rapid learning is observed over the course of the experiment, so that early dropout is primarily driven by side effects, while later attrition reflects declining CD4 counts for many subjects. An important implication of our findings, not recognized using the standard evaluation approach, is that patient welfare may be enhanced by offering a menu of therapies, since no single treatment is preferred by a majority of patients.

Keywords: Learning, Side Effects, Attrition, Treatment Evaluation, Randomized Experiments, Clinical Trial, Discrete Choice Dynamic Programming, Simulation Estimation, AIDS, ACTG 175

JEL Classification: I10, D83, C90