Representativeness of Remote Survey Methods in LMICs: A Cross-National Analysis of Pandemic-Era Studies

Abstract

Relative to survey methods where respondents are recruited and interviewed face to face, remote survey methods have become more common in low- and middle-income countries (LMICs) in recent years. These remote methods can often be deployed more quickly and cheaply, but it is unclear when, if ever, such methods yield nationally representative samples. We assess representativeness by comparing three remote methods against benchmark estimates from official nationally representative household surveys: (1) random-digit-dial (RDD phone survey), (2) phone follow-ups to face-to-face baseline surveys, and (3) social media recruitment for self-administered web surveys. A set of smaller face-to-face surveys are included for comparison as well. In all, we harmonize 32 datasets from nine countries, drawn primarily from the first two years of the COVID-19 pandemic.

We find that on average, most remote methods over-represent men, heads of households, younger, more educated, and more employed respondents, although there is meaningful variation across countries and methods. Surveys with nationally representative sampling frames tended to recruit more representative samples, with the social media samples deviating the most from national averages. Statistical adjustment and re-weighting did not successfully adjust for these imbalances in most cases. Overall, there appears to be a trade-off among survey methods between the marginal cost of recruitment and representativeness of the recruited samples. We recommend that researchers report the socio-demographic profiles of remotely recruited and interviewed samples to provide critical context when interpreting results, rather than relying on weighting alone.

Keywords: Survey Methods; Sampling Methods

JEL Classification: C28