MTurk ‘Unscrubbed’: Exploring the Good, the ‘Super’, and the Unreliable on Amazon's Mechanical Turk

45 Pages Posted: 4 Sep 2015 Last revised: 13 Oct 2015

See all articles by Jeanette Deetlefs

Jeanette Deetlefs

UNSW Australia Business School, School of Marketing

Mathew Chylinski

University of New South Wales (UNSW)

Andreas Ortmann

UNSW Australia Business School, School of Economics

Date Written: October 12, 2015

Abstract

Widely accepted as a low-cost, fast-turnaround solution with acceptable validity, Amazon’s Mechanical Turk (MTurk) is increasingly being used to source participants for academic studies. Yet two commonly raised concerns remain: the presence of quasi-professional respondents, or “Super-Turkers”, and the presence of “Spammers”, those that compromise quality while optimising their pay rate. We isolate the influence on research results of experienced subjects (Super-Turkers), and of unreliable subjects (Spammers), jointly and separately. Jointly including these subjects produces very similar results to jointly excluding them, yet effect sizes decrease disproportionately to their sample representation. Furthermore, separately including experienced subjects in research results is shown to be as problematic as inclusion of unreliable subjects, although the noise introduced by these subjects is divergent and measure dependent. Hence removing only one of these types of respondents can be even more damaging to the reliability of results, than including both.

Keywords: data collection, experimentation, field experiment, internet, Mechanical Turk

JEL Classification: C90, C91, D80

Suggested Citation

Deetlefs, Jeanette and Chylinski, Mathew and Ortmann, Andreas, MTurk ‘Unscrubbed’: Exploring the Good, the ‘Super’, and the Unreliable on Amazon's Mechanical Turk (October 12, 2015). UNSW Business School Research Paper No. 2015-20A, Available at SSRN: https://ssrn.com/abstract=2654056 or http://dx.doi.org/10.2139/ssrn.2654056

Jeanette Deetlefs

UNSW Australia Business School, School of Marketing

Sydney, NSW 2052
Australia

Mathew Chylinski

University of New South Wales (UNSW) ( email )

Kensington
High St
Sydney, NSW 2052
Australia

Andreas Ortmann (Contact Author)

UNSW Australia Business School, School of Economics ( email )

High Street
Sydney, NSW 2052
Australia

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
329
Abstract Views
2,469
Rank
167,723
PlumX Metrics