What Makes Experts Reliable?

28 Pages Posted: 5 Jun 2018

See all articles by Kyle L. Marquardt

Kyle L. Marquardt

V-Dem Institute, University of Gothenburg

Daniel Pemstein

North Dakota State University; Göteborg University - V-Dem Institute

Brigitte Seim

University of North Carolina (UNC) at Chapel Hill

Yi-ting Wang

National Cheng Kung University

Date Written: June 2018

Abstract

Many datasets use experts to code latent quantities of interest. However, scholars have not explored either the factors affecting expert reliability or the degree to which these factors influence estimates of latent concepts. Here we systematically analyze potential correlates of expert reliability using six randomly selected variables from a cross-national panel dataset, V-Dem v8. The V-Dem project includes a diverse group of over 3,000 experts and uses an IRT model to incorporate variation in both expert reliability and scale perception into its data aggregation process. In the process, the IRT model produces an estimate of expert reliability, which affects the relative contribution of an expert to the model. We examine a variety of factors that could correlate with reliability, and find little evidence of theoretically-untenable bias due to expert characteristics. On the other hand, there is evidence that attentive and condent experts who have a basic contextual knowledge of the concept of democracy are more reliable.

Suggested Citation

Marquardt, Kyle L. and Pemstein, Daniel and Seim, Brigitte and Wang, Yi-ting, What Makes Experts Reliable? (June 2018). V-Dem Working Paper 2018:68. Available at SSRN: https://ssrn.com/abstract=3190946 or http://dx.doi.org/10.2139/ssrn.3190946

Kyle L. Marquardt (Contact Author)

V-Dem Institute, University of Gothenburg ( email )

Department of Political Science
Sprängkullsgatan 19, PO 711
Gothenburg, SE 40530
Sweden

HOME PAGE: http://v-dem.net

Daniel Pemstein

North Dakota State University ( email )

Fargo, ND 58105
United States

Göteborg University - V-Dem Institute ( email )

United States

Brigitte Seim

University of North Carolina (UNC) at Chapel Hill ( email )

102 Ridge Road
Chapel Hill, NC 27514
United States

Yi-ting Wang

National Cheng Kung University ( email )

No.1, University Road
Tainan
Taiwan

Register to save articles to
your library

Register

Paper statistics

Downloads
17
Abstract Views
157
PlumX Metrics