Researchers' Degrees-of-Flexibility and the Credibility of Difference-in-Differences Estimates: Evidence from the Pandemic Policy Evaluations

72 Pages Posted: 7 Dec 2021 Last revised: 30 Mar 2025

See all articles by Joakim Weill

Joakim Weill

Board of Governors of the Federal Reserve System

Matthieu Stigler

Stanford University

Olivier Deschenes

University of California, Santa Barbara - College of Letters & Science - Department of Economics; National Bureau of Economic Research (NBER); IZA Institute of Labor Economics

Michael Springborn

University of California, Davis - College of Agricultural and Environmental Sciences - Department of Environmental Science and Policy

Date Written: December 2021

Abstract

The COVID-19 pandemic brought unprecedented policy responses and a large literature evaluating their impacts. This paper re-examines this literature and investigates the role of researchers' degrees-of-flexibility on the estimated effects of mobility-reducing policies on social-distancing behavior. We find that two-way fixed effects estimates are not robust to minor changes in usually-unexplored dimensions of the degree-of-flexibility space. While standard robustness tests based on the sequential addition of covariates are very stable, small changes in the outcome variable and its transformation lead to large and sometimes contradictory changes in the estimates, where the same policy can be found to significantly increase or decrease mobility. Yet, due to the large number of degrees-of-flexibility, one can focus on a set of results that appears stable, while ignoring problematic ones. We show that recently developed heterogeneity-robust difference-in-differences estimators only partially mitigate these issues, and discuss how a strategy of identifying the point at which a sequence of ever more-stringent robustness tests eventually fail could increase the credibility of policy evaluations.

Suggested Citation

Weill, Joakim and Stigler, Matthieu and Deschenes, Olivier and Springborn, Michael, Researchers' Degrees-of-Flexibility and the Credibility of Difference-in-Differences Estimates: Evidence from the Pandemic Policy Evaluations (December 2021). NBER Working Paper No. w29550, Available at SSRN: https://ssrn.com/abstract=3978408

Joakim Weill (Contact Author)

Board of Governors of the Federal Reserve System ( email )

20th Street and Constitution Avenue NW
Washington, DC 20551
United States

Matthieu Stigler

Stanford University ( email )

Stanford, CA 94305
United States

Olivier Deschenes

University of California, Santa Barbara - College of Letters & Science - Department of Economics ( email )

UC Santa Barbara
Santa Barbara, CA 93106
United States

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

IZA Institute of Labor Economics ( email )

P.O. Box 7240
Bonn, D-53072
Germany

Michael Springborn

University of California, Davis - College of Agricultural and Environmental Sciences - Department of Environmental Science and Policy ( email )

HOME PAGE: http://https://springborn.faculty.ucdavis.edu/

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
41
Abstract Views
281
PlumX Metrics