The Design of Field Experiments With Survey Outcomes: A Framework for Selecting More Efficient, Robust, and Ethical Designs

71 Pages Posted: 14 Apr 2016 Last revised: 9 Apr 2017

David E. Broockman

Stanford Graduate School of Business

Joshua Kalla

University of California, Berkeley, Students

Jasjeet S. Sekhon

UC Berkeley

Date Written: April 7, 2017

Abstract

There is increasing interest in experiments where outcomes are measured by surveys and treatments are delivered by a separate mechanism in the real world, such as by mailers, door-to-door canvasses, phone calls, or online ads. However, common designs for such experiments are often prohibitively expensive, vulnerable to bias, and raise ethical concerns. We show how four methodological practices currently uncommon in such experiments have previously undocumented complementarities that can dramatically relax these constraints when at least two are used in combination: 1) online surveys recruited from a defined sampling frame 2) with at least one baseline wave prior to treatment 3) with multiple items combined into an index to measure outcomes and, 4) when possible, a placebo control. We provide a general and extensible framework that allows researchers to determine the most efficient mix of these practices in diverse applications. Two studies then examine how these practices perform in real applications. First, we examine the representativeness of online panel respondents recruited from a defined sampling frame and find that their representativeness compares favorably to phone panel respondents. Second, an original experiment successfully implements all four practices in the context of a door-to-door canvassing experiment. We conclude discussing potential extensions.

Suggested Citation

Broockman, David E. and Kalla, Joshua and Sekhon, Jasjeet S., The Design of Field Experiments With Survey Outcomes: A Framework for Selecting More Efficient, Robust, and Ethical Designs (April 7, 2017). Available at SSRN: https://ssrn.com/abstract=2742869

David E. Broockman (Contact Author)

Stanford Graduate School of Business ( email )

655 Knight Way
Stanford, CA 94305-5015
United States

Joshua Kalla

University of California, Berkeley, Students ( email )

525 F. Haas School of Business
Berkeley, CA
United States

Jasjeet S. Sekhon

UC Berkeley ( email )

310 Barrows Hall
Berkeley, CA 94720
United States

Paper statistics

Downloads
1,314
Rank
11,095
Abstract Views
5,302