The Trouble with Regression Analysis of (Factorial) Experiments

13 Pages Posted: 7 May 2020

See all articles by Gregg G. Van Ryzin

Gregg G. Van Ryzin

Rutgers, The State University of New Jersey - Rutgers University, Newark

Date Written: April 12, 2020

Abstract

The use of experiments has become more widespread in many previously non-experimental areas of the social sciences, including political science, economics, and public administration (to name a few). Because researchers in these areas are trained mainly in regression analysis, they tend to employ a regression framework in their experimental work. But the use of regression analysis for experiments—especially 2 x 2 (or more complex) factorial experiments—can sometimes mislead. Using simulated data, this note demonstrates how the results of regression analysis differ—sometimes substantially—from the results of analysis of variance (ANOVA). This happens because regression provides tests of simple effects while ANOVA tests main effects, a distinction that is too often overlooked. Running sets of regressions with and without an interaction term, as well as deviation (-1, 1) coding, are sometimes used as remedies—but these approaches have limitations. A better alternative is (main) effect (-.5, .5) coding, which is less well known but works well for generating regression slopes and p-values that correspond with ANOVA-style main and interaction effects. This note illustrates and compares these various approaches using a small, simulated dataset that is available online for readers to work with.

Keywords: experimental methods, regression analysis, ANOVA

JEL Classification: C90

Suggested Citation

Van Ryzin, Gregg G., The Trouble with Regression Analysis of (Factorial) Experiments (April 12, 2020). Available at SSRN: https://ssrn.com/abstract=3574111 or http://dx.doi.org/10.2139/ssrn.3574111

Gregg G. Van Ryzin (Contact Author)

Rutgers, The State University of New Jersey - Rutgers University, Newark ( email )

180 University Avenue
Newark, NJ 07102
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
73
Abstract Views
401
rank
397,936
PlumX Metrics