How and How Much Does Expert Error Matter? Implications for Quantitative Peace Research

35 Pages Posted: 28 Feb 2019

See all articles by Kyle L. Marquardt

Kyle L. Marquardt

National Research University Higher School of Economics - School of Political Science

Date Written: February 2019

Abstract

Expert-coded datasets provide scholars with otherwise unavailable cross-national longitudinal data on important concepts. However, expert coders vary in their reliability and scale perception, potentially resulting in substantial measurement error; this variation may correlate with outcomes of interest, biasing results in analyses that use these data. This latter concern is particularly acute for key concepts in peace research. In this article, I describe potential sources of expert error, focusing on the measurement of identity-based discrimination. I then use expert-coded data on identity-based discrimination to examine:

1) the implications of measurement error for quantitative analyses that use expert-coded data, and

2) the degree to which different techniques for aggregating these data ameliorate these issues.

To do so, I simulate data with different forms and levels of expert error and regress conflict onset on different aggregations of these data. These analyses yield two important results. First, almost all aggregations show a positive relationship between identity-based discrimination and conflict onset consistently across simulations, in line with the assumed true relationship between the concept and outcome. Second, different aggregation techniques vary in their substantive robustness beyond directionality. A structural equation model provides the most consistently robust estimates, while both the point estimates from an Item Response Theory (IRT) model and the average over expert codings provide similar and relatively robust estimates in most simulations. The median over expert codings and a naive multiple imputation technique yield the least robust estimates.

Suggested Citation

Marquardt, Kyle L., How and How Much Does Expert Error Matter? Implications for Quantitative Peace Research (February 2019). V-Dem Working Paper 2019:84. Available at SSRN: https://ssrn.com/abstract=3344190 or http://dx.doi.org/10.2139/ssrn.3344190

Kyle L. Marquardt (Contact Author)

National Research University Higher School of Economics - School of Political Science ( email )

Moscow
Russia

Register to save articles to
your library

Register

Paper statistics

Downloads
10
Abstract Views
110
PlumX Metrics