Addressing Validity and Generalizability Concerns in Field Experiments

83 Pages Posted: 1 Jul 2020

See all articles by Gerhard Riener

Gerhard Riener

Heinrich Heine University Dusseldorf - Duesseldorf Institute for Competition Economics (DICE)

Sebastian O. Schneider

Max Planck Institute for Research on Collective Goods

Valentin Wagner

Heinrich Heine University Dusseldorf - Duesseldorf Institute for Competition Economics (DICE)

Date Written: June 29, 2020

Abstract

In this paper, we systematically analyze the empirical importance of standard conditions for the validity and generalizability of field experiments: the internal and external overlap and unconfoundedness conditions. We experimentally varied the degree of overlap in disjoint sub-samples from a recruitment experiment with more than 3,000 public schools, mimicking small scale field experiments. This was achieved by using different techniques for treatment assignment. We applied standard methods, such as pure randomization, and the novel minMSE treatment assignment method. This new technique should achieve improved overlap by balancing covariate dependencies and variances instead of focusing on individual mean values. We assess the relevance of the overlap condition by linking the estimation precision in the disjoint sub-samples to measures of overlap and balance in general. Unconfoundedness is addressed by using a rich set of administrative data on institution and municipality characteristics to study potential self-selection. We find no evidence for the violation of unconfoundedness and establish that improved overlap, and balancedness, as achieved by the minMSE method, reduce the bias of the treatment effect estimation by more than 35% compared to pure randomization, illustrating the importance of, and suggesting a solution to, addressing overlap also in (field) experiments.

Keywords: External validity, field experiments, generalizability, treatment effect, overlap, balance, precision, treatment assignment, unconfoundedness, self-selection bias, site-selection bias

JEL Classification: C9, C90, C93, D04

Suggested Citation

Riener, Gerhard and Schneider, Sebastian O. and Wagner, Valentin, Addressing Validity and Generalizability Concerns in Field Experiments (June 29, 2020). MPI Collective Goods Discussion Paper, No. 2020/16, Available at SSRN: https://ssrn.com/abstract=3637868 or http://dx.doi.org/10.2139/ssrn.3637868

Gerhard Riener

Heinrich Heine University Dusseldorf - Duesseldorf Institute for Competition Economics (DICE) ( email )

Universitaetsstr. 1
Duesseldorf, NRW 40225
Germany

Sebastian O. Schneider (Contact Author)

Max Planck Institute for Research on Collective Goods ( email )

Kurt-Schumacher-Str. 10
D-53113 Bonn, 53113
Germany

Valentin Wagner

Heinrich Heine University Dusseldorf - Duesseldorf Institute for Competition Economics (DICE) ( email )

Universitaetsstr. 1
Duesseldorf, NRW 40225
Germany

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
31
Abstract Views
233
PlumX Metrics