Data Collection in a Flat World: Strengths and Weaknesses of Mechanical Turk Samples

Journal of Behavioral Decision Making, Forthcoming

12 Pages Posted: 6 Mar 2012 Last revised: 19 Jun 2014

See all articles by Joseph K. Goodman

Joseph K. Goodman

Fisher College of Business, The Ohio State University

Cynthia Cryder

Washington University in St. Louis - John M. Olin Business School

Amar Cheema

University of Virginia (UVA), McIntire School of Commerce

Date Written: March 5, 2012

Abstract

Mechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this paper we review recent research about MTurk and compare MTurk participants to community and student samples on a set of personality dimensions and classic decision-making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample’s attitudes, but similar to students’ attitudes. Finally, MTurk participants are less extraverted and have lower self-esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision-making biases: They are present biased, risk-averse for gains, risk-seeking for losses, show delay/expedite asymmetries, and show the certainty effect — with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection, and recommend that researchers using MTurk 1) include screening questions that gauge attention and language comprehension, 2) avoid questions with factual answers, and 3) consider how individual differences in financial and social domains may influence results.

Keywords: research methods, surveys, sampling, online research, external validity

Suggested Citation

Goodman, Joseph K. and Cryder, Cynthia and Cheema, Amar, Data Collection in a Flat World: Strengths and Weaknesses of Mechanical Turk Samples (March 5, 2012). Journal of Behavioral Decision Making, Forthcoming, Available at SSRN: https://ssrn.com/abstract=2016308

Joseph K. Goodman (Contact Author)

Fisher College of Business, The Ohio State University ( email )

Fisher Hall 542
2100 Neil Ave
Columbus, OH 43210
United States

HOME PAGE: http://u.osu.edu/goodman/

Cynthia Cryder

Washington University in St. Louis - John M. Olin Business School ( email )

One Brookings Drive
Campus Box 1133
St. Louis, MO 63130-4899
United States

Amar Cheema

University of Virginia (UVA), McIntire School of Commerce ( email )

125 Ruppel Drive
Charlottesville, VA 22903
United States
434-924-4350 (Phone)

HOME PAGE: http://www.commerce.virginia.edu/faculty/cheema

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
237
Abstract Views
1,586
rank
153,942
PlumX Metrics