Using Online Labor Market Participants for Nonprofessional Investor Research: A Comparison of MTurk and Qualtrics Samples

Abstract

Recently, researchers have begun using online labor markets to recruit participants for experimental studies examining the judgments and decisions of nonprofessional investors. This study investigates the quality and generalizability of data collected from these sources by replicating an experimental task from Elliott, Hodge, Kennedy, and Pronk (2007) (“EHKP”) using nonprofessional investor participants from two popular online labor markets – Amazon’s Mechanical Turk (“MTurk”) and Qualtrics Online Sample (“Qualtrics”). Compared to Qualtrics participants, we find that MTurk participants pay greater attention to the experimental materials and better acquire and recall information. Further, the MTurk sample more closely replicates EHKP’s investment club member results on measures of information integration than does the Qualtrics sample. These results provide some evidence that many interesting research questions can be satisfactorily answered using nonprofessional investor participants from MTurk. We believe further investigation is needed before Qualtrics can be endorsed as a high-quality source of nonprofessional investor participants.

Keywords: nonprofessional investors; online labor markets; Amazon Mechanical Turk (MTurk); Qualtrics Online Sample; sources of data