A Picture is Not Worth 1,000 Words: Mitigating Treatment Recall Bias in Survey Experiments

35 Pages Posted: 14 Nov 2018

Date Written: March 20, 2018

Abstract

Information systems research routinely uses survey experiments as a research design. Treatment variables in survey experiments involve exposing one or more groups to new information, misinformation, or vignettes. We identify a problem of treatment recall bias that affects the magnitude and significance of a treatment’s observed effect. Respondents need to recall the salient features of the treatment in order for it to have an effect. When salient treatment details are not recalled, respondents are likely to answer survey questions in a way that is statistically indistinguishable from the control group. The treatment might still have an effect on the outcome in question, but the results could be masked by respondents who do not retain salient features of the treatment. Using new survey data, we highlight how the nature of the treatment itself – text over images – significantly affects the retention of treatment details. We also show how temporal factors such as survey duration, time on treatment, engagement with treatment, and a treatment to survey duration ratio affect treatment retention. Our findings suggest that the best results from survey experiments should employ text-based treatments and calibrate time on treatment and duration based upon the best practices presented below.

Keywords: Survey Experiment, Mechanical Turk, Treatment Recall Bias, Vignette Experiments, Manipulation Checks, Online Research, Online Surveys, Information Security, Information System Research

Suggested Citation

Jardine, Eric and Kelly, Jason, A Picture is Not Worth 1,000 Words: Mitigating Treatment Recall Bias in Survey Experiments (March 20, 2018). Available at SSRN: https://ssrn.com/abstract=3144414 or http://dx.doi.org/10.2139/ssrn.3144414

Eric Jardine (Contact Author)

Virginia Tech ( email )

57 Erb Street West
Waterloo, Ontario N2L 6C2
Canada

Jason Kelly

Virginia Tech ( email )

Blacksburg, VA 24061
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
10
Abstract Views
285
PlumX Metrics