Estimating Latent Traits from Expert Surveys: An Analysis of Sensitivity to Data Generating Process

28 Pages Posted: 18 Dec 2018

See all articles by Kyle L. Marquardt

Kyle L. Marquardt

V-Dem Institute, University of Gothenburg

Daniel Pemstein

North Dakota State University; Göteborg University - V-Dem Institute

Date Written: December 2018

Abstract

Models for converting expert-coded data to point estimates of latent concepts assume different data-generating processes. In this paper, we simulate ecologically-valid data according to different assumptions, and examine the degree to which common methods for aggregating expert-coded data can recover true values and construct appropriate coverage intervals from these data. We find that hierarchical latent variable models and the bootstrapped mean perform similarly when variation in reliability and scale perception is low; latent variable techniques outperform the mean when variation is high. Hierarchical A-M and IRT models generally perform similarly, though IRT models are often more likely to include true values within their coverage intervals. The median and non-hierarchical latent variable modeling techniques perform poorly under most assumed data generating processes.

Suggested Citation

Marquardt, Kyle L. and Pemstein, Daniel, Estimating Latent Traits from Expert Surveys: An Analysis of Sensitivity to Data Generating Process (December 2018). V-Dem Working Paper 2018:83. Available at SSRN: https://ssrn.com/abstract=3302459 or http://dx.doi.org/10.2139/ssrn.3302459

Kyle L. Marquardt (Contact Author)

V-Dem Institute, University of Gothenburg ( email )

Department of Political Science
Sprängkullsgatan 19, PO 711
Gothenburg, SE 40530
Sweden

HOME PAGE: http://v-dem.net

Daniel Pemstein

North Dakota State University ( email )

Fargo, ND 58105
United States

Göteborg University - V-Dem Institute ( email )

United States

Register to save articles to
your library

Register

Paper statistics

Downloads
2
Abstract Views
70
PlumX Metrics