Virtual Private Servers and the Limitations of IP-Based Screening Procedures: Lessons from the MTurk Quality Crisis of 2018
41 Pages Posted: 23 Aug 2018 Last revised: 27 Mar 2019
Date Written: March 14, 2019
We identify a pervasive, yet previously undocumented threat to the reliability of MTurk data – and discuss how this specific issue is symptomatic of opportunities and incentives that facilitate fraudulent behavior within online recruitment platforms. In doing so, we explain how IP addresses were never intended to identify individuals and are, on their own, surprisingly ill-suited to help researchers address risks around sample screening and data integrity. We discuss MTurk samples for two independent studies that include alarming proportions of participants who circumvent an entire set of conventional sample screening methods – and provide disturbingly low-quality responses. These “bad actors” exploited inherent limitations of IP screening procedures by using virtual private servers (VPS) that concealed the IP address of their local devices. While service providers (e.g., Turk Prime) now offer tools that target this VPS-enabled abuse, these “black-boxed” tools fail to address the underlying limitations of IP screening procedures. Our findings suggest the importance of continued diligence within the research community to identify and mitigate evolving threats to data integrity.
Keywords: Amazon Mechanical Turk; online experiments; participant screening; online labor markets; TurkPrime; virtual private servers
Suggested Citation: Suggested Citation