MTurk Workers' Use of Low-Cost 'Virtual Private Servers' to Circumvent Screening Methods: A Research Note
21 Pages Posted: 23 Aug 2018 Last revised: 14 Sep 2018
Date Written: August 17, 2018
Emerging literature examines the integrity of MTurk-based samples in social science research and develops techniques to ensure this data is reliable. We identify a pervasive, yet previously undocumented threat to the integrity of MTurk data and propose techniques to mitigate this threat. MTurk samples for two independent studies included alarming proportions of participants who circumvented several accepted sample screening methods. These ‘imposters’ provided responses through virtual private servers (VPS) that concealed their physical locations, thereby rendering conventional location screening methods useless. VPSs also allow individuals to submit multiple responses from the same device without detection. We identified this issue by investigating large sets of unique IP addresses with identical GPS coordinates; each such IP address traces back to a server farm and VPS provider. VPS participants provide disturbingly low-quality open responses. This highlights the importance of considering VPS use, in conjunction with established screening procedures, when working with MTurk samples.
Keywords: Amazon Mechanical Turk; online experiments; participant screening; online labor markets; TurkPrime; virtual private servers
Suggested Citation: Suggested Citation