Time-of-Day and Day-of-Week Variations in Amazon Mechanical Turk Survey Responses

Abstract

Social science research studies are frequently conducted on Amazon Mechanical Turk (MTurk). I use data from four previous economics studies conducted on Amazon Mechanical Turk, with a total of 2780 observations, to study how participant characteristics and behaviors depend on the day of the week and time of day of participation. Most notably, Saturday participants are older, less educated, and more likely to have low income compared to other participants. Controlling for demographics, Saturday participants are more likely to answer objective knowledge questions correctly and to provide reasonable inflation forecasts, less likely to provide "don't know" responses, and less likely to use social media as a primary source of economic news. Night participants are less likely to get economic news from print sources such as newspapers. Standard data cleaning procedures typically neither reduce nor exacerbate these patterns. Implications of these findings are especially important for researchers designing high-frequency surveys of macroeconomic expectations intended to enable identification of the effects of monetary policy announcements or other events of interest via a high frequency approach. Systematic day-of-week variation in respondents' knowledge and reported expectations, if not properly accounted for, could threaten such an identification scheme.

Keywords: Amazon Mechanical Turk, survey experiments, intertemporal effects, social sciences, research methods and design, data collection, economics, psychology, political science, crowdsourcing

JEL Classification: D80, D83, D84, E31