Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures
38 Pages Posted: 23 Aug 2018 Last revised: 7 Oct 2019
Date Written: March 14, 2019
We identify a pervasive, yet previously undocumented threat to the reliability of MTurk data – and discuss how this specific issue is symptomatic of opportunities and incentives that facilitate fraudulent behavior within online recruitment platforms. In doing so, we explain how IP addresses were never intended to identify individuals and are, on their own, likely insufficient to identify and mitigate emergent risks around data integrity. We discuss MTurk samples for two independent studies that include alarming proportions of participants who circumvent an entire set of conventional sample screening methods – and provide disturbingly low-quality responses. These “bad actors” exploited inherent limitations of IP screening procedures by using virtual private servers (VPS) that concealed the IP address of their local devices. While service providers (e.g., TurkPrime) now offer tools that help target this VPS-enabled abuse, the underlying limitations of IP screening procedures remain – and fraudulent workers will likely continue finding ways to exploit these inherent limitations. Our findings and discussions emphasize the importance of continued diligence within the research community to identify and mitigate evolving threats to data integrity as quickly as possible.
Keywords: Amazon Mechanical Turk; online experiments; participant screening; online labor markets; TurkPrime; virtual private servers
Suggested Citation: Suggested Citation