Designing for Power: Covariates, Indices, and Efficiency in Survey Experiments

Abstract

Survey experiments are widely used across the social sciences, yet many are underpowered. We examine two well-known, but unevenly applied, strategies for improving statistical efficiency: covariate adjustment and multi-item index outcomes. We formally evaluate when these practices increase power and revisit over four hundred recent social science survey experiments to empirically assess their prevalence and effectiveness. We show these techniques reduce distinct sources of variance and function as complements, not substitutes, for increasing power. However, the majority of survey experiments omit these practices or employ them ineffectively by adjusting for generic demographic covariates that yield minimal precision gains. Only a small minority of studies use specialized covariates or index outcomes. Fewer than five percent combine both practices. Drawing on simulations based on recent studies, we argue designs combining specialized covariates with a short index outcome offer a practical and robust way to increase the statistical power of survey experiments.

Keywords: survey experiment, statistical power, covariate adjustment, indices, survey experiments, experiments, experimental power