Data Collection in a Flat World: Strengths and Weaknesses of Mechanical Turk Samples

Abstract

Mechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this paper we review recent research about MTurk and compare MTurk participants to community and student samples on a set of personality dimensions and classic decision-making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample’s attitudes, but similar to students’ attitudes. Finally, MTurk participants are less extraverted and have lower self-esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision-making biases: They are present biased, risk-averse for gains, risk-seeking for losses, show delay/expedite asymmetries, and show the certainty effect — with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection, and recommend that researchers using MTurk 1) include screening questions that gauge attention and language comprehension, 2) avoid questions with factual answers, and 3) consider how individual differences in financial and social domains may influence results.

Keywords: research methods, surveys, sampling, online research, external validity