45 Pages Posted: 4 Sep 2015 Last revised: 13 Oct 2015
Date Written: October 12, 2015
Widely accepted as a low-cost, fast-turnaround solution with acceptable validity, Amazon’s Mechanical Turk (MTurk) is increasingly being used to source participants for academic studies. Yet two commonly raised concerns remain: the presence of quasi-professional respondents, or “Super-Turkers”, and the presence of “Spammers”, those that compromise quality while optimising their pay rate. We isolate the influence on research results of experienced subjects (Super-Turkers), and of unreliable subjects (Spammers), jointly and separately. Jointly including these subjects produces very similar results to jointly excluding them, yet effect sizes decrease disproportionately to their sample representation. Furthermore, separately including experienced subjects in research results is shown to be as problematic as inclusion of unreliable subjects, although the noise introduced by these subjects is divergent and measure dependent. Hence removing only one of these types of respondents can be even more damaging to the reliability of results, than including both.
Keywords: data collection, experimentation, field experiment, internet, Mechanical Turk
JEL Classification: C90, C91, D80
Suggested Citation: Suggested Citation
Deetlefs, Jeanette and Chylinski, Mathew and Ortmann, Andreas, MTurk ‘Unscrubbed’: Exploring the Good, the ‘Super’, and the Unreliable on Amazon's Mechanical Turk (October 12, 2015). UNSW Business School Research Paper No. 2015-20A. Available at SSRN: https://ssrn.com/abstract=2654056 or http://dx.doi.org/10.2139/ssrn.2654056