Aggregating Experts: Reducing Error and Assessing Reliability

Abstract

This paper lays out a simple two-step approach for aggregating individual level data from expert evaluators to create improved quality measures of U.S. House incumbents that are reliable both within and across districts. The two-step procedure is designed to minimize both systematic biases and random error in evaluator opinions to increase the validity and reliability of the aggregated measure. The result is a measure that has higher intra-district agreement than simple mean aggregations of opinions while retaining high between-district reliability. The approach is based on knitting together ideas from several different fields including organizational theory and marketing. Taken together, these steps make a substantial improvement in the construction of aggregated measures of candidate personal quality in U.S. House elections and are generally applicable to any circumstance in which unit level measures are created from evaluators ratings of a common target.

Keywords: reliability, expert opinions, congressional elections, opinion aggregation, inter-rater agreement