The More You Ask, the Less You Get: When Additional Questions Hurt External Validity

59 Pages Posted:

See all articles by Ye Li

Ye Li

University of California, Riverside (UCR) - Department of Management and Marketing; Center for Decision Sciences, Columbia University

Antonia Krefeld-Schwalb

Columbia University - Columbia Business School

Daniel Wall

affiliation not provided to SSRN

Olivier Toubia

Columbia Business School - Marketing

Eric J. Johnson

Columbia Business School - Marketing

Daniel M. Bartels

University of Chicago - Booth School of Business

Date Written: October 16, 2020

Abstract

Preference elicitation tasks are widely used to predict behavior in marketing, finance, and public policy. We suggest that every time a respondent answers an additional elicitation question, two things happen: (1) We obtain information about some parameter(s) of interest, such as their discount rate or the partworth for a product attribute, and (2) the respondent increasingly adapts to the task—i.e., using a task-specific decision process specialized for this task but that may not apply to other tasks. Importantly, adaptation comes at the cost of potential mismatch between the task-specific decision process and real-world processes that generate the target behaviors, so that asking more questions can reduce external validity. Using mouse-tracking to trace decision processes, we show evidence that respondents increasingly rely on task-specific decision processes as more questions are asked, which reduces external validity of the task for predicting other related tasks and behavior. Two large studies found that external validity for predicting real-world intertemporal choices and credit scores—peaked after only a few elicitation questions. When measuring preferences, less can be more.

Suggested Citation

Li, Ye and Krefeld-Schwalb, Antonia and Wall, Daniel and Toubia, Olivier and Johnson, Eric J. and Bartels, Daniel M., The More You Ask, the Less You Get: When Additional Questions Hurt External Validity (October 16, 2020). Available at SSRN: https://ssrn.com/abstract=

Ye Li

University of California, Riverside (UCR) - Department of Management and Marketing ( email )

United States

Center for Decision Sciences, Columbia University

New York, NY
United States

Antonia Krefeld-Schwalb (Contact Author)

Columbia University - Columbia Business School ( email )

3022 Broadway
New York, NY 10027
United States

Daniel Wall

affiliation not provided to SSRN

Olivier Toubia

Columbia Business School - Marketing ( email )

New York, NY 10027
United States

Eric J. Johnson

Columbia Business School - Marketing ( email )

New York, NY 10027
United States

Daniel M. Bartels

University of Chicago - Booth School of Business ( email )

5807 S. Woodlawn Avenue
Chicago, IL 60637
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
7
Abstract Views
40
PlumX Metrics