IRT Models for Expert-Coded Panel Data

62 Pages Posted: 12 Jan 2017

See all articles by Kyle L. Marquardt

Kyle L. Marquardt

V-Dem Institute, University of Gothenburg

Daniel Pemstein

North Dakota State University; Göteborg University - V-Dem Institute

Date Written: January 2017

Abstract

Data sets quantifying phenomena of social-scientific interest often use multiple experts to code latent concepts. While it remains standard practice to report the average score across experts, experts likely vary in both their expertise and their interpretation of question scales. As a result, the mean may be an inaccurate statistic. Item-response theory (IRT) models provide an intuitive method for taking these forms of expert disagreement into account when aggregating ordinal ratings produced by experts, but they have rarely been applied to cross-national expert-coded panel data. In this article, we investigate the utility of IRT models for aggregating expert-coded data by comparing the performance of various IRT models to the standard practice of reporting average expert codes, using both real and simulated data. Specifically, we use expert-coded cross-national panel data from the V–Dem data set to both conduct real-data comparisons and inform ecologically-motivated simulation studies. We find that IRT approaches outperform simple averages when experts vary in reliability and exhibit differential item functioning (DIF). IRT models are also generally robust even in the absence of simulated DIF or varying expert reliability. Our findings suggest that producers of cross-national data sets should adopt IRT techniques to aggregate expert-coded data of latent concepts.

Suggested Citation

Marquardt, Kyle L. and Pemstein, Daniel, IRT Models for Expert-Coded Panel Data (January 2017). V-Dem Working Paper 2017:41. Available at SSRN: https://ssrn.com/abstract=2897442 or http://dx.doi.org/10.2139/ssrn.2897442

Kyle L. Marquardt (Contact Author)

V-Dem Institute, University of Gothenburg ( email )

Department of Political Science
Sprängkullsgatan 19, PO 711
Gothenburg, SE 40530
Sweden

HOME PAGE: http://v-dem.net

Daniel Pemstein

North Dakota State University ( email )

Fargo, ND 58105
United States

Göteborg University - V-Dem Institute ( email )

United States

Register to save articles to
your library

Register

Paper statistics

Downloads
34
Abstract Views
216
PlumX Metrics