From Online Experiments to Big Experimental Data

Abstract

Always wanted to run an online experiment but did not know where to start? Or are you a seasoned practitioner interested in enriching your arsenal of tricks to solve practical challenges in online experiments? This chapter embarks in the arduous task to please both readers by reviewing history, methodological challenges, and current opportunities for online experimentation. It adopts an accessible language, avoids unnecessary jargon and formalisms, and draws lessons from experiments across all social science disciplines. It explains why it is important to preregister a study and how to solve practical problems related to data quality, attrition, attention, and bots, and it also reviews new methods for running optimized experiments or recruiting from social media. This chapter concludes by discussing how the merging of digital traces and experimental data at scale will shift the focus from Big Data to Big Experimental Data in computational social science (CSS).

Keywords: online experiments, data quality, dropouts, preregistration, recruitment, optimization, computational social science

JEL Classification: C9