Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures

Abstract

We identify a pervasive, yet previously undocumented threat to the reliability of MTurk data – and discuss how this specific issue is symptomatic of opportunities and incentives that facilitate fraudulent behavior within online recruitment platforms. In doing so, we explain how IP addresses were never intended to identify individuals and are, on their own, likely insufficient to identify and mitigate emergent risks around data integrity. We discuss MTurk samples for two independent studies that include alarming proportions of participants who circumvent an entire set of conventional sample screening methods – and provide disturbingly low-quality responses. These “bad actors” exploited inherent limitations of IP screening procedures by using virtual private servers (VPS) that concealed the IP address of their local devices. While service providers (e.g., TurkPrime) now offer tools that help target this VPS-enabled abuse, the underlying limitations of IP screening procedures remain – and fraudulent workers will likely continue finding ways to exploit these inherent limitations. Our findings and discussions emphasize the importance of continued diligence within the research community to identify and mitigate evolving threats to data integrity as quickly as possible.

Keywords: Amazon Mechanical Turk; online experiments; participant screening; online labor markets; TurkPrime; virtual private servers