The Shape of and Solutions to the MTurk Quality Crisis

27 Pages Posted: 3 Dec 2018

See all articles by Ryan Kennedy

Ryan Kennedy

University of Houston - Department of Political Science

Scott Clifford

University of Houston - Department of Political Science

Tyler Burleigh

affiliation not provided to SSRN

Ryan Jewell

University of Houston - Department of Political Science

Philip Waggoner

College of William & Mary

Date Written: October 24, 2018

Abstract

Amazon’s Mechanical Turk (MTurk) is widely used to collect affordable and high-quality survey responses. However, researchers recently noticed a substantial decline in data quality, sending shockwaves throughout the social sciences. The problem seems to stem from the use of Virtual Private Servers (VPSs) by respondents outside the U.S. to fool MTurk’s filtering system, but we know relatively little about the cause and consequence of this form of fraud. Analyzing 38 studies conducted on MTurk, we demonstrate that this problem is not new - we find a similar spike in VPS use in 2015. Utilizing two new studies, we show that data from these respondents is of substantially worse quality. Next, we provide two solutions for this problem using an API for an IP traceback application (IP Hub). We provide both a post-hoc method for identifying fraudulent respondents using an original R package (“rIP”) and an associated online application, and an a priori method using JavaScript and PHP code in Qualtrics to block fraudulent respondents from participating. We demonstrate the effectiveness of the screening procedure in a third study. Overall, our results suggest that fraudulent respondents pose a serious threat to data quality but can be easily identified and screened out.

Keywords: Mechanical Turk, MTurk, Fraud, Quality, Online Surveys, Survey Experiments

Suggested Citation

Kennedy, Ryan and Clifford, Scott and Burleigh, Tyler and Jewell, Ryan and Waggoner, Philip, The Shape of and Solutions to the MTurk Quality Crisis (October 24, 2018). Available at SSRN: https://ssrn.com/abstract=3272468 or http://dx.doi.org/10.2139/ssrn.3272468

Ryan Kennedy (Contact Author)

University of Houston - Department of Political Science ( email )

TX 77204-3011
United States
713-743-1663 (Phone)
713-743-3890 (Fax)

Scott Clifford

University of Houston - Department of Political Science ( email )

TX 77204-3011
United States

Tyler Burleigh

affiliation not provided to SSRN

Ryan Jewell

University of Houston - Department of Political Science ( email )

TX 77204-3011
United States

Philip Waggoner

College of William & Mary ( email )

300 James Blair Dr.
Williamsburg, VA 23187
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
215
rank
139,211
Abstract Views
1,467
PlumX Metrics