Difference-in-Differences for Simple and Complex Natural Experiments

137 Pages Posted: 26 Jun 2023 Last revised: 28 Nov 2023

See all articles by Clément de Chaisemartin

Clément de Chaisemartin

SciencesPo - Sciences Po - Department of Economics

Xavier D'Haultfœuille

Center for Research in Economics and Statistics (CREST)

Date Written: June 21, 2023

Abstract

Much of science is concerned with causal inference, namely estimating the effect of a “treatment”
on an “outcome”. For that purpose, a gold-standard method is to run a randomized experiment,
where units’ exposure to treatment is determined randomly. Then, one can compare the average
outcome of treated and untreated units, to unbiasedly estimate the so-called average treatment
effect (ATE), namely the average effect of the treatment in the population of interest.

In the social sciences and in epidemiology, there are many questions which, to date, have never
been studied using a randomized experiment. For instance, one cannot randomly assign individ-
uals to a “smoking” treatment group, to ascertain that smoking causes lung cancer. Similarly,
labor economists are interested in the effect of the minimum wage on employment, but there
has never been an experiment where some firms are randomly assigned to a high-minimum-wage
group. Even if such an experiment had been run, its estimate of the minimum wage’s effect
would suffer from an important shortcoming. Research ethics require enlightened consent to
include subjects in an experiment, so experimental samples are usually not representative of
the population of interest. Then, the hypothetical minimum wage experiment would unbiasedly
estimate the minimum wage’s effect among firms that enrolled in the experiment, but that effect
would probably differ from the minimum wage’s effect among all firms. Firms willing to enroll
in an experiment that could mandate them to pay a higher minimum wage probably do not pay
many of their employees at the minimum wage, so a minimum wage increase may be less con-
sequential to them than to other firms. In a nutshell, even when they are feasible, randomized
experiments sometimes lack “external validity”: their findings may not be extrapolated from the
experimental sample to the population whose ATE the researcher would like to learn.

When randomized experiments are unfeasible, or when they would lack external validity, re-
searchers rely on natural experiments to estimate treatment effects. Natural experiments used
in the social sciences are often policy changes. For instance, a US state increases its minimum
wage while the neighboring state does not, thus giving researchers a treatment group facing a
high minimum wage, and a control group facing a lower minimum wage. Natural experiments
often affect an entire state, region, or province, so findings from studies leveraging natural ex-
periments typically apply to large and unselected populations, unlike findings from randomized
experiments. However, in natural experiments, assignment to the treatment is not randomized
by a researcher, it is decided by a policy maker. In a sharp reversal of Keynes’ famous quote,
modern applied researchers, who believe themselves to be quite exempt from any practical in-
fluences, have to hope that practical men will give them good natural experiments they can
work with. This fact has two important consequences. First, as policy makers do not randomly
choose where to implement a policy change, treated and control locations may not be compa-
rable, and a simple comparison of their mean outcome may not yield an unbiased estimator of
the ATE. In the minimum wage example, the treated and control states may for instance have
different employment levels even before the treated state increased its minimum wage. Then,
comparing their employment levels after that increase would capture the minimum wage’s effect
and pre-existing differences between the two states. Second, legislative changes are often full of
twists and turns. In the minimum wage example, treated states that raise their minimum wage
may then decrease it, or increase it again. And control states that initially did not change their
minimum wage may then decide to change it. Also, some states could implement large minimum
wage increases, while other states implement smaller increases. This creates lots of treatment
variation, that complicates the analysis and can even preclude researchers from estimating the
clean and simple effects they were initially after. Glass-half-empty researchers may lament over
policy makers’ erratic behavior, and wait till they deliver the perfect experiment to estimate the
parameters that live in their models. Glass-half-full researchers will try to learn the most from
what they have. This book hopes to become a companion for glass-half-full researchers, who
courageously try to analyze non-randomized, potentially complex natural experiments.

Suggested Citation

de Chaisemartin, Clément and d'Haultfoeuille, Xavier, Difference-in-Differences for Simple and Complex Natural Experiments (June 21, 2023). Available at SSRN: https://ssrn.com/abstract=4487202 or http://dx.doi.org/10.2139/ssrn.4487202

Clément De Chaisemartin (Contact Author)

SciencesPo - Sciences Po - Department of Economics ( email )

28, rue des Saints-Pères
Paris, Paris 75007
France

Xavier D'Haultfoeuille

Center for Research in Economics and Statistics (CREST) ( email )

5 avenue Henry le Chatelier
Palaiseau, 91120
France

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
6,452
Abstract Views
13,499
Rank
2,180
PlumX Metrics