The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments

36 Pages Posted: 21 May 2019

See all articles by John A. List

John A. List

University of Chicago - Department of Economics; National Bureau of Economic Research (NBER); IZA Institute of Labor Economics

Dana Suskind

University of Chicago

Omar Al-Ubaydli

George Mason University - Department of Economics; Derasat; George Mason University - Mercatus Center

Multiple version iconThere are 2 versions of this paper

Date Written: May 1, 2019

Abstract

Policymakers are increasingly turning to insights gained from the experimental method as a means of informing public policies. Whether—and to what extent—insights from a research study scale to the level of the broader public is, in many situations, based on blind faith. This scale-up problem can lead to a vast waste of resources, a missed opportunity to improve people’s lives, and a diminution in the public’s trust in the scientific method’s ability to contribute to policymaking. This study provides a theoretical lens to deepen our understanding of the science of how to use science. Through a simple model, we highlight three elements of the scale-up problem:

(1) when does evidence become actionable (appropriate statistical inference);

(2) properties of the population; and

(3) properties of the situation.

We argue that until these three areas are fully understood and recognized by researchers and policymakers, the threats to scalability will render any scaling exercise as particularly vulnerable. In this way, our work represents a challenge to empiricists to estimate the nature and extent of how important the various threats to scalability are in practice, and to implement those in their original research.

JEL Classification: C9, C90, C91, C92, C93, D03

Suggested Citation

List, John A. and Suskind, Dana and Al-Ubaydli, Omar, The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments (May 1, 2019). University of Chicago, Becker Friedman Institute for Economics Working Paper No. 2019-73, Available at SSRN: https://ssrn.com/abstract=3391481 or http://dx.doi.org/10.2139/ssrn.3391481

John A. List (Contact Author)

University of Chicago - Department of Economics ( email )

1126 East 59th Street
Chicago, IL 60637
United States

National Bureau of Economic Research (NBER)

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

IZA Institute of Labor Economics

P.O. Box 7240
Bonn, D-53072
Germany

Dana Suskind

University of Chicago ( email )

5841 S. Maryland Avenue
MC 1035
Chicago, IL 60637
United States
7738349169 (Phone)

HOME PAGE: http://tmwcenter.uchicago.edu

Omar Al-Ubaydli

George Mason University - Department of Economics ( email )

4400 University Drive
Fairfax, VA 22030
United States
703-993-4538 (Phone)
703-993-1133 (Fax)

Derasat ( email )

Bahrain

George Mason University - Mercatus Center ( email )

3434 Washington Blvd., 4th Floor
Arlington, VA 22201
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
116
Abstract Views
501
rank
250,319
PlumX Metrics