A Picture is Not Worth 1,000 Words: Mitigating Treatment Recall Bias in Survey Experiments
35 Pages Posted: 14 Nov 2018
Date Written: March 20, 2018
Information systems research routinely uses survey experiments as a research design. Treatment variables in survey experiments involve exposing one or more groups to new information, misinformation, or vignettes. We identify a problem of treatment recall bias that affects the magnitude and significance of a treatment’s observed effect. Respondents need to recall the salient features of the treatment in order for it to have an effect. When salient treatment details are not recalled, respondents are likely to answer survey questions in a way that is statistically indistinguishable from the control group. The treatment might still have an effect on the outcome in question, but the results could be masked by respondents who do not retain salient features of the treatment. Using new survey data, we highlight how the nature of the treatment itself – text over images – significantly affects the retention of treatment details. We also show how temporal factors such as survey duration, time on treatment, engagement with treatment, and a treatment to survey duration ratio affect treatment retention. Our findings suggest that the best results from survey experiments should employ text-based treatments and calibrate time on treatment and duration based upon the best practices presented below.
Keywords: Survey Experiment, Mechanical Turk, Treatment Recall Bias, Vignette Experiments, Manipulation Checks, Online Research, Online Surveys, Information Security, Information System Research
Suggested Citation: Suggested Citation