The Dot-Guessing Game: A ‘Fruit Fly’ for Human Computation Research

6 Pages Posted: 8 May 2010

See all articles by John J. Horton

John J. Horton

New York University (NYU) - Department of Information, Operations, and Management Sciences

Date Written: May 4, 2010

Abstract

I propose a human computation task that has a number of properties that make it useful for empirical research. The task itself is simple: subjects are asked to guess the number of dots in an image. The task is useful because: (1) it is simple to generate examples; (2) it is a CAPTCHA; and (3) it has an objective solution that allows us to finely grade performances, yet subjects cannot simply look up the answer. To demonstrate its value, I conducted two experiments using the dot-guessing task. Across both experiments, I found that the “crowd” performed well, with the arithmetic mean guess besting most individual guesses. However, I found that subjects displayed well-known behavioral biases relevant to the design of human computation systems. In the first experiment, subjects reported whether the number of dots in an image was larger or smaller than a randomly chosen number. The randomly chosen reference number strongly anchored subjects' subsequent guesses. In the second experiment, subjects were asked to choose between an incentive payment contract that rewarded good work (and self-knowledge about relative ability) and a fixed payment contract. I found that selection of the incentive contract did not improve performance, nor did it reveal information about relative ability. However, subjects who chose the incentive contract were more likely to be male, suggesting that their choice in contracts was determined by differences in risk-aversion.

Keywords: Crowdsourcing, Human Computation, Mechanical Turk, Experimentation

JEL Classification: C91, C93, J16

Suggested Citation

Horton, John J., The Dot-Guessing Game: A ‘Fruit Fly’ for Human Computation Research (May 4, 2010). Available at SSRN: https://ssrn.com/abstract=1600372 or http://dx.doi.org/10.2139/ssrn.1600372

John J. Horton (Contact Author)

New York University (NYU) - Department of Information, Operations, and Management Sciences ( email )

44 West Fourth Street
New York, NY 10012
United States
6175952437 (Phone)

HOME PAGE: http://john-joseph-horton.com

Register to save articles to
your library

Register

Paper statistics

Downloads
139
rank
197,538
Abstract Views
976
PlumX Metrics