Beyond the Breaking Point? Survey Satisficing in Conjoint Experiments

66 Pages Posted: 27 Apr 2017 Last revised: 30 Oct 2018

See all articles by Kirk Bansak

Kirk Bansak

Stanford University, Department of Political Science, Students

Jens Hainmueller

Stanford University - Department of Political Science; Stanford Graduate School of Business; Stanford Immigration Policy Lab

Daniel J. Hopkins

University of Pennsylvania

Teppei Yamamoto

Massachusetts Institute of Technology (MIT) - Department of Political Science

Date Written: October 30, 2018

Abstract

Recent years have seen a renaissance of conjoint survey designs within social science. To date, however, researchers have lacked guidance on how many attributes they can include within conjoint profiles before survey satisficing leads to unacceptable declines in response quality. This paper addresses that question using pre-registered, two-stage experiments examining choices among hypothetical candidates for U.S. Senate or hotel rooms. In each experiment, we use the first stage to identify attributes which are perceived to be uncorrelated with the attribute of interest, so that their effects are not masked by those of the core attributes. In the second stage, we randomly assign respondents to conjoint designs with varying numbers of those filler attributes. We report the results of these experiments implemented via Amazon's Mechanical Turk and Survey Sampling International. They demonstrate that our core quantities of interest are generally stable, with relatively modest increases in survey satisficing when respondents face large numbers of attributes.

Keywords: Conjoint Experiments, Survey Experiments, Survey Satisficing, Response Bias

JEL Classification: C83, C90, D72

Suggested Citation

Bansak, Kirk and Hainmueller, Jens and Hopkins, Daniel J. and Yamamoto, Teppei, Beyond the Breaking Point? Survey Satisficing in Conjoint Experiments (October 30, 2018). Stanford University Graduate School of Business Research Paper No. 17-33; MIT Political Science Department Research Paper No. 2017-16. Available at SSRN: https://ssrn.com/abstract=2959146 or http://dx.doi.org/10.2139/ssrn.2959146

Kirk Bansak

Stanford University, Department of Political Science, Students ( email )

Stanford, CA 94305
United States

Jens Hainmueller

Stanford University - Department of Political Science ( email )

Stanford, CA 94305
United States

HOME PAGE: http://www.stanford.edu/~jhain/

Stanford Graduate School of Business ( email )

655 Knight Way
Stanford, CA 94305-5015
United States

Stanford Immigration Policy Lab

30 Alta Road
Stanford, CA 94305
United States

Daniel J. Hopkins (Contact Author)

University of Pennsylvania ( email )

Stiteler Hall
Philadelphia, PA 19104
United States

HOME PAGE: http://www.danhopkins.org

Teppei Yamamoto

Massachusetts Institute of Technology (MIT) - Department of Political Science ( email )

77 Massachusetts Avenue
Cambridge, MA 02139
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
283
rank
101,527
Abstract Views
1,014
PlumX Metrics