Analyzing Experimental Data Using Regression: When is Bias a Practical Concern?

Abstract

Experimental researchers routinely use regression in order to control for pre-treatment covariates. This practice has become controversial in the wake of recent demonstrations showing that this type of regression is prone to bias in small samples. Bias may even remain when units are sampled from a larger population of infnite size. This paper uses a combination of simulated and actual examples to show that, as a practical matter, bias tends to be negligible when the sample size is greater than 20. When bias does occur, it tends to be small in relation to the standard error of the estimated average treatment effect.

Keywords: regression analysis, David Freedman, experiments, statistical analysis

JEL Classification: C90, C00