Nonnaïveté Among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers

Forthcoming, Chandler, J., Mueller, P., Paolacci, G. Nonnaïveté among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers. Behavior Research Methods. DOI 10.3758/s13428-013-0365-7

19 Pages Posted: 5 Feb 2014

See all articles by Jesse Chandler

Jesse Chandler

Princeton University - Woodrow Wilson School of Public and International Affairs

Pam Mueller

Princeton University - Department of Psychology

Gabriele Paolacci

Erasmus University - Rotterdam School of Management

Date Written: February 4, 2014

Abstract

Crowdsourcing services — particularly Amazon Mechanical Turk — have made it easy for behavioral scientists to recruit research participants. However, researchers have overlooked crucial differences between crowdsourcing and traditional recruitment methods that provide unique opportunities and challenges. We show that crowdsourced workers are likely to participate across multiple related experiments and that researchers are overzealous in the exclusion of research participants. We describe how both of these problems can be avoided using advanced interface features that also allow prescreening and longitudinal data collection. Using these techniques can minimize the effects of previously ignored drawbacks and expand the scope of crowdsourcing as a tool for psychological research.

Keywords: Crowdsourcing, Internet research, Data quality, Longitudinal research, Mechanical Turk, MTurk

Suggested Citation

Chandler, Jesse and Mueller, Pam and Paolacci, Gabriele, Nonnaïveté Among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers (February 4, 2014). Forthcoming, Chandler, J., Mueller, P., Paolacci, G. Nonnaïveté among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers. Behavior Research Methods. DOI 10.3758/s13428-013-0365-7. Available at SSRN: https://ssrn.com/abstract=2390569

Jesse Chandler (Contact Author)

Princeton University - Woodrow Wilson School of Public and International Affairs ( email )

Princeton University
Princeton, NJ 08544-1021
United States

Pam Mueller

Princeton University - Department of Psychology ( email )

Green Hall
Princeton, NJ 08540
United States

Gabriele Paolacci

Erasmus University - Rotterdam School of Management ( email )

3000 DR Rotterdam
Netherlands

Register to save articles to
your library

Register

Paper statistics

Downloads
44
Abstract Views
314
PlumX Metrics