The Design of Field Experiments With Survey Outcomes: A Framework for Selecting More Efficient, Robust, and Ethical Designs

71 Pages Posted: 14 Apr 2016 Last revised: 21 Sep 2017

See all articles by David E. Broockman

David E. Broockman

University of California, Berkeley - Department of Political Science

Joshua Kalla

University of California, Berkeley, Students

Jasjeet S. Sekhon

UC Berkeley

Date Written: September 20, 2017

Abstract

There is increasing interest in experiments where outcomes are measured by surveys and treatments are delivered by a separate mechanism in the real world, such as by mailers, door-to-door canvasses, phone calls, or online ads. However, common designs for such experiments are often prohibitively expensive, vulnerable to bias, and raise ethical concerns. We show how four methodological practices currently uncommon in such experiments have previously undocumented complementarities that can dramatically relax these constraints when at least two are used in combination: 1) online surveys recruited from a defined sampling frame 2) with at least one baseline wave prior to treatment 3) with multiple items combined into an index to measure outcomes and, 4) when possible, a placebo control. We provide a general and extensible framework that allows researchers to determine the most efficient mix of these practices in diverse applications. Two studies then examine how these practices perform in real applications. First, we examine the representativeness of online panel respondents recruited from a defined sampling frame and find that their representativeness compares favorably to phone panel respondents. Second, an original experiment successfully implements all four practices in the context of a door-to-door canvassing experiment. We conclude discussing potential extensions.

Suggested Citation

Broockman, David E. and Kalla, Joshua and Sekhon, Jasjeet S., The Design of Field Experiments With Survey Outcomes: A Framework for Selecting More Efficient, Robust, and Ethical Designs (September 20, 2017). Available at SSRN: https://ssrn.com/abstract=2742869 or http://dx.doi.org/10.2139/ssrn.2742869

David E. Broockman (Contact Author)

University of California, Berkeley - Department of Political Science ( email )

210 Barrows Hall
Berkeley, CA 94720
United States

HOME PAGE: http://www.davidbroockman.com

Joshua Kalla

University of California, Berkeley, Students ( email )

525 F. Haas School of Business
Berkeley, CA
United States

Jasjeet S. Sekhon

UC Berkeley ( email )

310 Barrows Hall
Berkeley, CA 94720
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
1,731
Abstract Views
8,936
rank
10,096
PlumX Metrics