Anchoring Vignettes and Response Consistency

Abstract

The use of anchoring vignettes to correct for differential item functioning rests upon two identifying assumptions: vignette equivalence and response consistency. To test the second assumption the authors conduct an experiment in which respondents in an Internet panel are asked to both describe their health in a number of domains and rate their health in these domains. In a subsequent interview respondents are shown vignettes that are in fact descriptions of their own health. Under response consistency and some auxiliary assumptions with regard to the validity of the experiment, there should be no systematic differences between the evaluation of these vignettes in the second interview and the self-evaluations in the first interview. They analyze data for five health domains: sleep, mobility, concentration, breathing and affect. Although descriptively the vignettes and the self-evaluations are similar for a number of domains, their nonparametric analysis suggests that response consistency is satisfied for the domain of sleep, while it indicates rejection of either the auxiliary assumptions or response consistency for the other domains of health. Parametric analysis suggests that the auxiliary assumptions may be most problematic. The analysis points at the need for a systematic experimental approach to the design of anchoring vignettes before using them in practice.