The Online Laboratory: Conducting Experiments in a Real Labor Market

31 Pages Posted: 16 Apr 2010 Last revised: 25 Jan 2015

See all articles by John J. Horton

John J. Horton

New York University (NYU) - Department of Information, Operations, and Management Sciences

David G. Rand

Massachusetts Institute of Technology (MIT)

Richard J. Zeckhauser

Harvard University - Harvard Kennedy School (HKS); National Bureau of Economic Research (NBER)

Multiple version iconThere are 2 versions of this paper

Date Written: April 16, 2010

Abstract

Online labor markets have great potential as platforms for conducting experiments. They provide immediate access to a large and diverse subject pool, and allow researchers to control the experimental context. Online experiments, we show, can be just as valid - both internally and externally - as laboratory and field experiments, while often requiring far less money and time to design and conduct. To demonstrate their value, we use an online labor market to replicate three classic experiments. The first finds quantitative agreement between levels of cooperation in a prisoner''s dilemma played online and in the physical laboratory. The second shows - consistent with behavior in the traditional laboratory - that online subjects respond to priming by altering their choices. The third demonstrates that when an identical decision is framed differently, individuals reverse their choice, thus replicating a famed Tversky-Kahneman result. Then we conduct a field experiment showing that workers have upward-sloping labor supply curves. Finally, we analyze the challenges to online experiments, proposing methods to cope with the unique threats to validity in an online setting, and examining the conceptual issues surrounding the external validity of online results. We conclude by presenting our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.

Keywords: Experimentation, Online Labor Markets, Internet

JEL Classification: J2, C93, C91, C92, C70

Suggested Citation

Horton, John J. and Rand, David G. and Zeckhauser, Richard J., The Online Laboratory: Conducting Experiments in a Real Labor Market (April 16, 2010). Experimental Economics, Forthcoming, Available at SSRN: https://ssrn.com/abstract=1591202

John J. Horton (Contact Author)

New York University (NYU) - Department of Information, Operations, and Management Sciences ( email )

44 West Fourth Street
New York, NY 10012
United States
6175952437 (Phone)

HOME PAGE: http://john-joseph-horton.com

David G. Rand

Massachusetts Institute of Technology (MIT) ( email )

77 Massachusetts Avenue
50 Memorial Drive
Cambridge, MA 02139-4307
United States

HOME PAGE: http://www.daverand.org

Richard J. Zeckhauser

Harvard University - Harvard Kennedy School (HKS) ( email )

79 John F. Kennedy Street
Cambridge, MA 02138
United States
617-495-1174 (Phone)
617-384-9340 (Fax)

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States
617-495-1174 (Phone)
617-496-3783 (Fax)

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
1,006
Abstract Views
18,307
Rank
37,127
PlumX Metrics