The Shape of and Solutions to the MTurk Quality Crisis

27 Pages Posted: 3 Dec 2018

See all articles by Ryan Kennedy

Ryan Kennedy

University of Houston - Department of Political Science

Scott Clifford

University of Houston - Department of Political Science

Tyler Burleigh

affiliation not provided to SSRN

Ryan Jewell

University of Houston - Department of Political Science

Philip Waggoner

University of Chicago

Date Written: October 24, 2018

Abstract

Amazon’s Mechanical Turk (MTurk) is widely used to collect affordable and high-quality survey responses. However, researchers recently noticed a substantial decline in data quality, sending shockwaves throughout the social sciences. The problem seems to stem from the use of Virtual Private Servers (VPSs) by respondents outside the U.S. to fool MTurk’s filtering system, but we know relatively little about the cause and consequence of this form of fraud. Analyzing 38 studies conducted on MTurk, we demonstrate that this problem is not new - we find a similar spike in VPS use in 2015. Utilizing two new studies, we show that data from these respondents is of substantially worse quality. Next, we provide two solutions for this problem using an API for an IP traceback application (IP Hub). We provide both a post-hoc method for identifying fraudulent respondents using an original R package (“rIP”) and an associated online application, and an a priori method using JavaScript and PHP code in Qualtrics to block fraudulent respondents from participating. We demonstrate the effectiveness of the screening procedure in a third study. Overall, our results suggest that fraudulent respondents pose a serious threat to data quality but can be easily identified and screened out.

Keywords: Mechanical Turk, MTurk, Fraud, Quality, Online Surveys, Survey Experiments

Suggested Citation

Kennedy, Ryan and Clifford, Scott and Burleigh, Tyler and Jewell, Ryan and Waggoner, Philip, The Shape of and Solutions to the MTurk Quality Crisis (October 24, 2018). Available at SSRN: https://ssrn.com/abstract=3272468 or http://dx.doi.org/10.2139/ssrn.3272468

Ryan Kennedy (Contact Author)

University of Houston - Department of Political Science ( email )

TX 77204-3011
United States
713-743-1663 (Phone)
713-743-3890 (Fax)

Scott Clifford

University of Houston - Department of Political Science ( email )

TX 77204-3011
United States

Tyler Burleigh

affiliation not provided to SSRN

Ryan Jewell

University of Houston - Department of Political Science ( email )

TX 77204-3011
United States

Philip Waggoner

University of Chicago ( email )

1101 East 58th Street
Chicago, IL 60637
United States

HOME PAGE: http://pdwaggoner.github.io

Register to save articles to
your library

Register

Paper statistics

Downloads
281
Abstract Views
1,816
rank
107,492
PlumX Metrics