Beyond the Breaking Point? Survey Satisficing in Conjoint Experiments

Recent years have seen a renaissance of conjoint survey designs within social science. To date, however, researchers have lacked guidance on how many attributes they can include within conjoint profiles before survey satisficing leads to unacceptable declines in response quality. This paper addresses that question using pre-registered, two-stage experiments examining choices among hypothetical candidates for U.S. Senate or hotel rooms. In each experiment, we use the first stage to identify attributes which are perceived to be uncorrelated with the attribute of interest, so that their effects are not masked by those of the core attributes. In the second stage, we randomly assign respondents to conjoint designs with varying numbers of those filler attributes. We report the results of these experiments implemented via Amazon's Mechanical Turk and Survey Sampling International. They demonstrate that our core quantities of interest are generally stable, with relatively modest increases in survey satisficing when respondents face large numbers of attributes.

Keywords: Conjoint Experiments, Survey Experiments, Survey Satisficing, Response Bias

JEL Classification: C83, C90, D72

Suggested Citation: Suggested Citation

Bansak, Kirk and Hainmueller, Jens and Hopkins, Daniel J. and Yamamoto, Teppei, Beyond the Breaking Point? Survey Satisficing in Conjoint Experiments (October 30, 2018). Stanford University Graduate School of Business Research Paper No. 17-33, MIT Political Science Department Research Paper No. 2017-16, Available at SSRN: https://ssrn.com/abstract=2959146 or http://dx.doi.org/10.2139/ssrn.2959146