Site Selection Bias in Program Evaluation

73 Pages Posted: 15 Sep 2012 Last revised: 15 Mar 2021

See all articles by Hunt Allcott

Hunt Allcott

New York University (NYU)

Sendhil Mullainathan

Harvard University - Department of Economics; National Bureau of Economic Research (NBER)

Date Written: September 2012

Abstract

"Site selection bias" can occur when the probability that a program is adopted or evaluated is correlated with its impacts. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials involving 8.6 million households across the U.S. Predictions based on rich microdata from the first ten replications substantially overstate efficacy in the next 101 sites. Several mechanisms caused this positive selection. For example, utilities in more environmentalist areas are more likely to adopt the program, and their customers are more responsive to the treatment. Also, because utilities initially target treatment at higher-usage consumer subpopulations, efficacy drops as the program is later expanded. The results illustrate how program evaluations can still give systematically biased out-of-sample predictions, even after many replications.

Suggested Citation

Allcott, Hunt and Mullainathan, Sendhil, Site Selection Bias in Program Evaluation (September 2012). NBER Working Paper No. w18373, Available at SSRN: https://ssrn.com/abstract=2147084

Hunt Allcott (Contact Author)

New York University (NYU) ( email )

Bobst Library, E-resource Acquisitions
20 Cooper Square 3rd Floor
New York, NY 10003-711
United States

Sendhil Mullainathan

Harvard University - Department of Economics ( email )

Littauer Center
Cambridge, MA 02138
United States
617-496-2720 (Phone)
617-495-7730 (Fax)

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States
617-588-1473 (Phone)
617-876-2742 (Fax)

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
36
Abstract Views
680
PlumX Metrics