Evaluating and Improving Item Response Theory Models for Cross-National Expert Surveys

55 Pages Posted: 3 Jun 2015 Last revised: 24 May 2016

See all articles by Daniel Pemstein

Daniel Pemstein

North Dakota State University; University of Gothenburg - V-Dem Institute

Eitan Tzelgov

University of Gothenburg - V-Dem Institute; University of Gothenburg

Yi-ting Wang

National Cheng Kung University

Date Written: March 1, 2015

Abstract

The data produced by the Varieties of Democracy (V-Dem) project contains ordinal ratings of a multitude of country-level indicators across space and time, with multiple experts providing judgments for each country-year observation. We use an ordinal item response theory (O-IRT) model to aggregate multiple experts' ratings. The V-Dem data provide a challenging domain for such models because they exhibit little cross-national bridging. That is, few coders provide ratings for multiple countries, making it difficult to calibrate the scales of estimates cross-nationally. In this paper, we provide a systematic analysis of the issue of bridging. We first use simulations to explore how much bridging one needs to achieve scale identification when coders' thresholds vary across countries and when the latent traits of some countries lack variation. We then examine how posterior predictive checks can be used to check cases of extent of scale non-comparability. Finally, we develop and evaluate search algorithms designed to select bridges that are most likely allow one to correct scale incompatibility problems.

Suggested Citation

Pemstein, Daniel and Tzelgov, Eitan and Wang, Yi-ting, Evaluating and Improving Item Response Theory Models for Cross-National Expert Surveys (March 1, 2015). V-Dem Working Paper 2015:1, Available at SSRN: https://ssrn.com/abstract=2613421 or http://dx.doi.org/10.2139/ssrn.2613421

Daniel Pemstein (Contact Author)

North Dakota State University ( email )

Fargo, ND 58105
United States

University of Gothenburg - V-Dem Institute ( email )

United States

Eitan Tzelgov

University of Gothenburg - V-Dem Institute ( email )

United States

University of Gothenburg ( email )

Viktoriagatan 30
Göteborg, 405 30
Sweden

Yi-ting Wang

National Cheng Kung University ( email )

No.1, University Road
Tainan
Taiwan

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
99
Abstract Views
718
rank
314,397
PlumX Metrics