Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures

38 Pages Posted: 23 Aug 2018 Last revised: 7 Oct 2019

See all articles by Sean A. Dennis

Sean A. Dennis

University of Central Florida - Kenneth G. Dixon School of Accounting

Brian M. Goodson

Clemson University

Chris Pearson

University of Kentucky

Date Written: March 14, 2019

Abstract

We identify a pervasive, yet previously undocumented threat to the reliability of MTurk data – and discuss how this specific issue is symptomatic of opportunities and incentives that facilitate fraudulent behavior within online recruitment platforms. In doing so, we explain how IP addresses were never intended to identify individuals and are, on their own, likely insufficient to identify and mitigate emergent risks around data integrity. We discuss MTurk samples for two independent studies that include alarming proportions of participants who circumvent an entire set of conventional sample screening methods – and provide disturbingly low-quality responses. These “bad actors” exploited inherent limitations of IP screening procedures by using virtual private servers (VPS) that concealed the IP address of their local devices. While service providers (e.g., TurkPrime) now offer tools that help target this VPS-enabled abuse, the underlying limitations of IP screening procedures remain – and fraudulent workers will likely continue finding ways to exploit these inherent limitations. Our findings and discussions emphasize the importance of continued diligence within the research community to identify and mitigate evolving threats to data integrity as quickly as possible.

Keywords: Amazon Mechanical Turk; online experiments; participant screening; online labor markets; TurkPrime; virtual private servers

Suggested Citation

Dennis, Sean A. and Goodson, Brian Matthew and Pearson, Chris, Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures (March 14, 2019). Available at SSRN: https://ssrn.com/abstract=3233954 or http://dx.doi.org/10.2139/ssrn.3233954

Sean A. Dennis (Contact Author)

University of Central Florida - Kenneth G. Dixon School of Accounting ( email )

University of Central Florida
P.O. Box 161400
Orlando, FL 32816-1400
United States

Brian Matthew Goodson

Clemson University ( email )

Clemson, SC 29631
United States

Chris Pearson

University of Kentucky ( email )

Lexington, KY 40506
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
1,460
Abstract Views
8,034
rank
12,242
PlumX Metrics