Device Differences in Survey-Experimental Treatment Effects: A Cross-Country Exploration

Abstract

There has been a massive increase in the use of mobile devices among regular internet users, and accordingly, also among survey respondents of interest to political scientists. This paper investigates how the use of different devices (PC, tablet, or smartphone) affects responses to standard types of survey experiments in political science. In similar surveys in four countries, no overall differences are found in the effects of equivalence, framing, source cue, or item order priming experiments across users of different devices. While this is good news for survey researchers, some indication is found of larger item non-response, larger standard deviations of scales, and fewer words written in response to open-ended questions among users of mobile devices. Many results vary across countries, suggesting that some caution is warranted with regard to the generalizability of results from single countries.

Keywords: Survey experiment, device differences, bias