Statistical Analysis of Results from Laboratory Studies in Experimental Economics: A Critique of Current Practice

Abstract

Drawing on examples of recently published and widely-cited studies in experimental economics, we show that behavioral games are frequently analyzed in a manner that is prone to biased causal inference. First, deficiencies in design and implementation jeopardize the crucial assumption that treatments are statistically independent of potential outcomes. Researchers frequently do not randomly assign treatments or do not focus on randomly assigned factors when interpreting results. Second, many analyses of second mover behaviors in two-stage games, such as the ultimatum game and the trust game, are susceptible to bias. Third, uncontrolled stimuli, such as face-to-face interaction among subjects or the presentation of subjects’ photos, may also cause bias. Fourth, we discuss the limits of causal inference in repeated games, such as the public goods game. We recommend adjusting laboratory procedures and estimation methods in order to lessen reliance on substantive assumptions not grounded in experimental design.

Keywords: Experimental Economics, Laboratory Experiments, Causal Inference, Average Treatment Effect

JEL Classification: C13, C18, C51, C91