Method Effects in Factorial Surveys: An Analysis of Respondents' Comments, Interviewers' Assessments, and Response Behavior

31 Pages Posted: 22 Feb 2014 Last revised: 11 Mar 2014

See all articles by Carsten Sauer

Carsten Sauer

Bielefeld University

Katrin Auspurg

University of Konstanz

Thomas Hinz

University of Konstanz

Stefan Liebig

Bielefeld University - Department of Sociology; German Institute for Economic Research (DIW Berlin) - German Socio Economic Panel

Jürgen Schupp

German Institute for Economic Research (DIW Berlin); Institute for Sociology, Freie Universität; The Institute for the Study of Labor (IZA)

Date Written: February 1, 2014

Abstract

This paper describes the implementation of a factorial survey within the SOEP-Pretest of 2008 and investigates (1) respondents' comments about the vignettes, (2) interviewers' assessments of respondents comprehension and willingness to answer, and (3) response behavior regarding response time, use of the answering scale, and consistency of evaluations by different age groups and educational background. The respondents evaluated 24 vignettes consisting of ten dimensions that described full-time employees and their gross earnings. The evaluation task was to assess whether the given earnings were just or unjust, and if they were rated as unjust, respondents had to specify the amount of injustice on an 100-point rating scale. In regard to respondents' comments, the critique mentioned most frequently by respondents referred to the content (unrealistic descriptions) and the number of the vignettes. The analysis of the interviewers' assessments of respondents' comprehension and willingness to answer revealed less comprehension and willingness to answer for older and less well educated respondents, although these differences were similar to those found for the complete questionnaire. The analysis of the response behavior revealed no differences of response time between the groups. Analyses of response consistency show that one should consider hints for simplifying heuristics: such heuristics can lead to an artificially high response consistency. The implications of the findings are discussed.

JEL Classification: C81, D63, J31

Suggested Citation

Sauer, Carsten and Auspurg, Katrin and Hinz, Thomas and Liebig, Stefan and Schupp, Juergen, Method Effects in Factorial Surveys: An Analysis of Respondents' Comments, Interviewers' Assessments, and Response Behavior (February 1, 2014). SOEPpaper No. 629. Available at SSRN: https://ssrn.com/abstract=2399404 or http://dx.doi.org/10.2139/ssrn.2399404

Carsten Sauer (Contact Author)

Bielefeld University ( email )

Universitätsstraße 25
Bielefeld, 33615
Germany

Katrin Auspurg

University of Konstanz ( email )

Fach D-144
Universitätsstraße 10
Konstanz, D-78457
Germany

Thomas Hinz

University of Konstanz ( email )

Fach D-144
Universitätsstraße 10
Konstanz, D-78457
Germany

Stefan Liebig

Bielefeld University - Department of Sociology ( email )

Fakultaet fuer Soziologie
Universitaetsstr. 25
33613 Bielefeld, DE 33613
Germany

HOME PAGE: http://www.uni-bielefeld.de/soz/personen/liebig/

German Institute for Economic Research (DIW Berlin) - German Socio Economic Panel ( email )

DIW Berlin
Berlin, Berlin 14191
Germany

Juergen Schupp

German Institute for Economic Research (DIW Berlin) ( email )

Mohrenstraße 58
Berlin, D-10117
Germany
+49 30 8 978 9238 (Phone)
+49 30 8 978 9109 (Fax)

HOME PAGE: http://www.diw.de/deutsch/das_institut/mitarbeiter/innen/26652.html?uid=jschupp

Institute for Sociology, Freie Universität ( email )

Garystr. 55
I
Berlin, D-14195
Germany

HOME PAGE: http://www.polsoz.fu-berlin.de/soziologie/mitarbeiter/b_hochschullehrer/schupp.html

The Institute for the Study of Labor (IZA) ( email )

P.O. Box 7240
Bonn, D-53072
Germany

HOME PAGE: http://www.iza.org/index_html?lang=de&mainframe=http%3A//www.iza.org/de/webcontent/personnel/fellows

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
63
Abstract Views
616
rank
364,382
PlumX Metrics