Estimating Latent Traits from Expert Surveys: An Analysis of Sensitivity to Data Generating Process

28 Pages Posted: 18 Dec 2018

See all articles by Kyle L. Marquardt

Kyle L. Marquardt

National Research University Higher School of Economics - School of Political Science

Daniel Pemstein

North Dakota State University; University of Gothenburg - V-Dem Institute

Date Written: December 2018

Abstract

Models for converting expert-coded data to point estimates of latent concepts assume different data-generating processes. In this paper, we simulate ecologically-valid data according to different assumptions, and examine the degree to which common methods for aggregating expert-coded data can recover true values and construct appropriate coverage intervals from these data. We find that hierarchical latent variable models and the bootstrapped mean perform similarly when variation in reliability and scale perception is low; latent variable techniques outperform the mean when variation is high. Hierarchical A-M and IRT models generally perform similarly, though IRT models are often more likely to include true values within their coverage intervals. The median and non-hierarchical latent variable modeling techniques perform poorly under most assumed data generating processes.

Suggested Citation

Marquardt, Kyle L. and Pemstein, Daniel, Estimating Latent Traits from Expert Surveys: An Analysis of Sensitivity to Data Generating Process (December 2018). V-Dem Working Paper 2018:83, Available at SSRN: https://ssrn.com/abstract=3302459 or http://dx.doi.org/10.2139/ssrn.3302459

Kyle L. Marquardt (Contact Author)

National Research University Higher School of Economics - School of Political Science ( email )

Moscow
Russia

Daniel Pemstein

North Dakota State University ( email )

Fargo, ND 58105
United States

University of Gothenburg - V-Dem Institute ( email )

United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
6
Abstract Views
217
PlumX Metrics