Download This PaperThe Trouble with Regression Analysis of (Factorial) Experiments
13 Pages Posted: 7 May 2020
Date Written: April 12, 2020
Abstract
The use of experiments has become more widespread in many previously non-experimental areas of the social sciences, including political science, economics, and public administration (to name a few). Because researchers in these areas are trained mainly in regression analysis, they tend to employ a regression framework in their experimental work. But the use of regression analysis for experiments—especially 2 x 2 (or more complex) factorial experiments—can sometimes mislead. Using simulated data, this note demonstrates how the results of regression analysis differ—sometimes substantially—from the results of analysis of variance (ANOVA). This happens because regression provides tests of simple effects while ANOVA tests main effects, a distinction that is too often overlooked. Running sets of regressions with and without an interaction term, as well as deviation (-1, 1) coding, are sometimes used as remedies—but these approaches have limitations. A better alternative is (main) effect (-.5, .5) coding, which is less well known but works well for generating regression slopes and p-values that correspond with ANOVA-style main and interaction effects. This note illustrates and compares these various approaches using a small, simulated dataset that is available online for readers to work with.
Keywords: experimental methods, regression analysis, ANOVA
JEL Classification: C90
Suggested Citation: Suggested Citation