The Online Laboratory: Conducting Experiments in a Real Labor Market

38 Pages Posted: 10 May 2010 Last revised: 11 Feb 2023

See all articles by John J. Horton

John J. Horton

New York University (NYU) - Department of Information, Operations, and Management Sciences

David G. Rand

Massachusetts Institute of Technology (MIT)

Richard J. Zeckhauser

Harvard University - Harvard Kennedy School (HKS); National Bureau of Economic Research (NBER)

Multiple version iconThere are 2 versions of this paper

Date Written: May 2010

Abstract

Online labor markets have great potential as platforms for conducting experiments, as they provide immediate access to a large and diverse subject pool and allow researchers to conduct randomized controlled trials. We argue that online experiments can be just as valid - both internally and externally - as laboratory and field experiments, while requiring far less money and time to design and to conduct. In this paper, we first describe the benefits of conducting experiments in online labor markets; we then use one such market to replicate three classic experiments and confirm their results. We confirm that subjects (1) reverse decisions in response to how a decision-problem is framed, (2) have pro-social preferences (value payoffs to others positively), and (3) respond to priming by altering their choices. We also conduct a labor supply field experiment in which we confirm that workers have upward sloping labor supply curves. In addition to reporting these results, we discuss the unique threats to validity in an online setting and propose methods for coping with these threats. We also discuss the external validity of results from online domains and explain why online results can have external validity equal to or even better than that of traditional methods, depending on the research question. We conclude with our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.

Suggested Citation

Horton, John J. and Rand, David G. and Zeckhauser, Richard J., The Online Laboratory: Conducting Experiments in a Real Labor Market (May 2010). NBER Working Paper No. w15961, Available at SSRN: https://ssrn.com/abstract=1601712

John J. Horton (Contact Author)

New York University (NYU) - Department of Information, Operations, and Management Sciences ( email )

44 West Fourth Street
New York, NY 10012
United States
6175952437 (Phone)

HOME PAGE: http://john-joseph-horton.com

David G. Rand

Massachusetts Institute of Technology (MIT) ( email )

77 Massachusetts Avenue
50 Memorial Drive
Cambridge, MA 02139-4307
United States

HOME PAGE: http://www.daverand.org

Richard J. Zeckhauser

Harvard University - Harvard Kennedy School (HKS) ( email )

79 John F. Kennedy Street
Cambridge, MA 02138
United States
617-495-1174 (Phone)
617-384-9340 (Fax)

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States
617-495-1174 (Phone)
617-496-3783 (Fax)

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
81
Abstract Views
6,585
Rank
37,127
PlumX Metrics