Aggregating Experts: Reducing Error and Assessing Reliability
16 Pages Posted: 30 Apr 2010
Date Written: July 28, 2008
Abstract
This paper lays out a simple two-step approach for aggregating individual level data from expert evaluators to create improved quality measures of U.S. House incumbents that are reliable both within and across districts. The two-step procedure is designed to minimize both systematic biases and random error in evaluator opinions to increase the validity and reliability of the aggregated measure. The result is a measure that has higher intra-district agreement than simple mean aggregations of opinions while retaining high between-district reliability. The approach is based on knitting together ideas from several different fields including organizational theory and marketing. Taken together, these steps make a substantial improvement in the construction of aggregated measures of candidate personal quality in U.S. House elections and are generally applicable to any circumstance in which unit level measures are created from evaluators ratings of a common target.
Keywords: reliability, expert opinions, congressional elections, opinion aggregation, inter-rater agreement
Suggested Citation: Suggested Citation
Do you have negative results from your research you’d like to share?
Recommended Papers
-
Informants in Organizational Marketing Research
By G.h. Van Bruggen, Gary L. Lilien, ...
-
Impact of Co-Opetition on Firm Competitive Behavior: An Empirical Examination
By Devi R. Gnyawali, Jinyu He, ...
-
Cross-Functional 'Coopetition': The Simultaneous Role of Cooperation and Competition within Firms
By Xueming Luo, Rebecca J. Slotegraaf, ...