Distinguishing between Consistency and Error in Reliability Coefficients: Improving the Estimation and Interpretation of Information on Measurement Precision

59 Pages Posted: 1 Oct 2010

See all articles by William P. Fisher

William P. Fisher

University of California, Berkeley - BEAR Center

Date Written: September 30, 2009

Abstract

Reliability coefficients indicate the proportion of total variance due to differences not attributable to error among measures separated along a quantitative continuum by a testing, survey, or assessment instrument. Person score or measure reliability is influenced by both the internal consistency of a data set and the number of items, though textbooks and research papers rarely evaluate the extent to which these factors independently affect the data in question. Probabilistic formulations of the requirements for unidimensional measurement separate consistency from error by modeling individual response processes instead of group-level variation. The utility of this separation is illustrated via analyses of small sets of simulated data, and of subsets of data from a 78-item survey of over 2,500 parents of children with disabilities. Measurement reliability ultimately concerns the structural invariance specified in models requiring sufficient statistics, parameter separation, unidimensionality, and other qualities that historically have made quantification simple, practical, and convenient for end users. The paper concludes with suggestions for a research program aimed at focusing measurement research more on the calibration and wide dissemination of tools applicable to individuals, and less on the statistical study of inter-variable relations in large data sets.

Keywords: reliability, unidimensionality, consistency, error, statistics, measurement, individual behavior models, parameter, Rasch models separation; model fit; survey research; rating scales;

JEL Classification: C42, C44, C51, C52, C91

Suggested Citation

Fisher, William P., Distinguishing between Consistency and Error in Reliability Coefficients: Improving the Estimation and Interpretation of Information on Measurement Precision (September 30, 2009). Available at SSRN: https://ssrn.com/abstract=1685556 or http://dx.doi.org/10.2139/ssrn.1685556

William P. Fisher (Contact Author)

University of California, Berkeley - BEAR Center ( email )

Berkeley, CA 94704
United States

HOME PAGE: http://www.LivingCapitalMetrics.com

Register to save articles to
your library

Register

Paper statistics

Downloads
139
Abstract Views
726
rank
207,813
PlumX Metrics