Fraud in Online Surveys: Evidence from a Nonprobability, Subpopulation Sample

Abstract

We hired a well-known market research firm whose surveys have been published in leading political science journals, including JEPS. Based on a set of rigorous "screeners,'' we detected what appears to be exceedingly high rates of identity falsification: Over 81 percent of respondents seemed to misrepresent their credentials to gain access to the survey and earn compensation. Similarly high rates of presumptive character falsification were present in panels from multiple sub-vendors procured by the firm. Moreover, we found additional, serious irregularities embedded in the data, including evidence of respondents using deliberate strategies to detect and circumvent one of our screeners, as well as pervasive, observable patterns reflecting that the survey had been taken repeatedly by a respondent or collection of respondents. This evidence offers particular reasons to be concerned about the quality of digital, non-probability, subpopulation samples, and calls for further, systematic research.

Keywords: surveys, fraud, methodology, samples, online surveys, military