Running Experiments on Amazon Mechanical Turk

Judgment and Decision Making, Vol. 5, No. 5, 411-419

9 Pages Posted: 1 Jul 2010 Last revised: 25 Jan 2015

See all articles by Gabriele Paolacci

Gabriele Paolacci

Erasmus University - Rotterdam School of Management

Jesse Chandler

Princeton University - Princeton School of Public and International Affairs

Panagiotis G. Ipeirotis

New York University - Leonard N. Stern School of Business

Date Written: June 24, 2010

Abstract

Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by participants recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting participants, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.

Keywords: Experimentation, Judgment and decision-making, Online research

JEL Classification: C9, J2

Suggested Citation

Paolacci, Gabriele and Chandler, Jesse and Ipeirotis, Panagiotis G., Running Experiments on Amazon Mechanical Turk (June 24, 2010). Judgment and Decision Making, Vol. 5, No. 5, 411-419, Available at SSRN: https://ssrn.com/abstract=1626226

Gabriele Paolacci (Contact Author)

Erasmus University - Rotterdam School of Management ( email )

3000 DR Rotterdam
Netherlands

Jesse Chandler

Princeton University - Princeton School of Public and International Affairs ( email )

Princeton University
Princeton, NJ 08544-1021
United States

Panagiotis G. Ipeirotis

New York University - Leonard N. Stern School of Business ( email )

44 West Fourth Street
Ste 8-84
New York, NY 10012
United States
+1-212-998-0803 (Phone)

HOME PAGE: http://www.stern.nyu.edu/~panos

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
7,321
Abstract Views
30,322
Rank
1,752
PlumX Metrics