Aggregating Experts: Reducing Error and Assessing Reliability

16 Pages Posted: 30 Apr 2010

See all articles by Cherie D Maestas

Cherie D Maestas

University of North Carolina Charlotte

Date Written: July 28, 2008


This paper lays out a simple two-step approach for aggregating individual level data from expert evaluators to create improved quality measures of U.S. House incumbents that are reliable both within and across districts. The two-step procedure is designed to minimize both systematic biases and random error in evaluator opinions to increase the validity and reliability of the aggregated measure. The result is a measure that has higher intra-district agreement than simple mean aggregations of opinions while retaining high between-district reliability. The approach is based on knitting together ideas from several different fields including organizational theory and marketing. Taken together, these steps make a substantial improvement in the construction of aggregated measures of candidate personal quality in U.S. House elections and are generally applicable to any circumstance in which unit level measures are created from evaluators ratings of a common target.

Keywords: reliability, expert opinions, congressional elections, opinion aggregation, inter-rater agreement

Suggested Citation

Maestas, Cherie D, Aggregating Experts: Reducing Error and Assessing Reliability (July 28, 2008). Available at SSRN: or

Cherie D Maestas (Contact Author)

University of North Carolina Charlotte ( email )

Dept. of Political Science & Public Administration
9201 University City Blvd
Charlotte, NC 28223
United States

Register to save articles to
your library


Paper statistics

Abstract Views
PlumX Metrics