Are 'Webographic' or Attitudinal Questions Useful for Adjusting Estimates from Web Surveys Using Propensity Scoring?

18 Pages Posted: 14 Aug 2007  

Matthias Schonlau

University of Waterloo

Arthur van Soest

Tilburg University; Netspar; RAND Corporation; Institute for the Study of Labor (IZA)

Arie Kapteyn

Dornsife Center for Economic and Social Research - University of Southern California; IZA Institute of Labor Economics

Date Written: June 2007

Abstract

Inference from Web surveys may be affected by non-random selection of Web survey participants. One approach to reduce selection bias is to use propensity scores and a parallel phone survey. This approach uses demographic and additional so-called Webographic or lifestyle variables to balance observed differences between Web survey respondents and phone survey respondents. Here the authors investigate some of the Webographic questions used by Harris Interactive, a commercial company specializing in Web surveys. Their Webographic questions include choice of activities such as reading, sports and traveling and perceptions about what would constitute a violation of privacy. They use data from an existing probability sample of respondents over 40 who are interviewed over the phone, and a corresponding sample of respondents interviewed over the Web. They find that Webographic questions differentiate between on and offline populations differently than demographic questions. In general, propensity score adjustment of variables in the Web survey works quite well for a number of variables of interest (including home ownership and labor force participation). For two outcomes, (having emotional problems and often experiencing pain) the process of adjusting for demographic variables leads to the discovery of an instance of Simpson's paradox, implying a differential mode effect or differential selection. They interpret this mainly as the result of a mode effect, where sensitive questions are more likely to receive a positive response over the Internet than over the phone.

Keywords: propensity scoring, Web survey, selection bias, Webographic variables

Suggested Citation

Schonlau, Matthias and van Soest, Arthur and Kapteyn, Arie, Are 'Webographic' or Attitudinal Questions Useful for Adjusting Estimates from Web Surveys Using Propensity Scoring? (June 2007). RAND Working Paper Series No. WR-506. Available at SSRN: https://ssrn.com/abstract=1006108 or http://dx.doi.org/10.2139/ssrn.1006108

Matthias Schonlau (Contact Author)

University of Waterloo ( email )

Waterloo, Ontario
Canada

Arthur H. O. van Soest

Tilburg University ( email )

P.O. Box 90153
Tilburg, DC 5000 LE
Netherlands

Netspar

P.O. Box 90153
Tilburg, 5000 LE
Netherlands

RAND Corporation ( email )

P.O. Box 2138
1776 Main Street
Santa Monica, CA 90407-2138
United States

Institute for the Study of Labor (IZA)

P.O. Box 7240
Bonn, D-53072
Germany

Arie Kapteyn

Dornsife Center for Economic and Social Research - University of Southern California ( email )

635 Downey Way
Los Angeles, CA 90089-3332
United States
310-448-5383 (Phone)

IZA Institute of Labor Economics

P.O. Box 7240
Bonn, D-53072
Germany

Paper statistics

Downloads
112
Rank
197,424
Abstract Views
1,235