Getting the Race Wrong: Sampling Biases and Black Voters in Online, Opt-In Polls

Abstract

Social scientists are increasingly reliant on online, opt-in surveys. Such samples enable researchers to conduct affordable experiments with results that often generalize. But can these surveys effectively represent Black respondents? This paper presents the results of surveys conducted online and in person using multiple samples in a racially diverse American city--Philadelphia--during its 2023 mayoral primary. The participation rate for online surveys promoted via Facebook and Instagram is just 0.4%, with White residents and those with college degrees markedly more likely to respond. Such biases help explain why neither our surveys nor public polls correctly identified the primary's ultimate winner, an establishment-backed Black Democrat. Even weighted, geographically stratified online surveys typically under-estimate the winner's support, although an in-person exit poll over-estimates her support. These results indicate important gaps in the populations represented in contemporary opt-in surveys and suggest that supplemental surveys via alternative modes may help reduce those gaps.

Keywords: public opinion, Black Americans, survey research, online surveys, opt-in surveys

JEL Classification: H00, H80