The Survey Effect: Does Administering Surveys Affect Trainees' Behavior?

Abstract

This research demonstrates a survey effect, such that the act of administering surveys alters training effectiveness. Two aspects of survey administration were manipulated: the number of survey questions per training module (ranged from 1 to 30 across experimental conditions) and the type of survey questions (self-regulation or trainee reactions) across two studies focusing on self-administered online training. The number of survey questions had an indirect, negative effect on learning via the amount of time spent responding to survey questions. Furthermore, attrition increased when lengthy surveys were administered and this effect was moderated by pretraining motivation — adding additional survey questions increased the probability of dropping out for trainees who had low (rather than high) motivation to learn. The data quality also declined as the number of survey questions increased. Finally, learning performance was higher and the data quality was better when trainees were asked self-regulation rather than trainee reaction questions.

Keywords: Survey effect, Training, Attrition, Data quality, Self-regulation, Interruptions