Measurement Error Obfuscates Scientific Knowledge: Path to Cumulative Knowledge Requires Corrections for Unreliability and Psychometric Meta-Analyses

Industrial and Organizational Psychology (7,4), 2014, pp. 507-518.

12 Pages Posted: 17 Jan 2015

See all articles by Chockalingam Viswesvaran

Chockalingam Viswesvaran

Florida International University

Deniz S. Ones

University of Minnesota

Frank L. Schmidt

University of Iowa - Henry B. Tippie College of Business

Huy Le

University of Texas at San Antonio

In‐Sue Oh

Temple University - Department of Human Resource Management

Date Written: December 1, 2014

Abstract

All measurements must contend with unreliability. No measure is free of measurement error. More attention must be paid to measurement error in all psychological research. The problem of reliability is more severe when rating scales are involved. Many of the constructs in industrial-organizational (I-O) psychology and organizational behavior/human resource management research are assessed using ratings. Most notably the organizationally central construct of job performance is often assessed using ratings (Austin & Villanova, 1992; Borman & Brush, 1993; Campbell, Gasser, & Oswald, 1996; Viswesvaran, Ones, & Schmidt, 1996; Viswesvaran, Schmidt, & Ones, 2005). The reliability of its assessment is a critical issue with consequences for (a) validation and (b) decision making. For over a century now, it has been known that measurement error obfuscates relationships among variables that scientists assess. Again for over a century, it has been known that statistical corrections for unreliability can help reveal the true magnitudes of relationships being examined. However, until mid-1970s, corrections for attenuation were hampered by the fact that the effect of sampling error is magnified in corrected correlations (Schmidt & Hunter, 1977). Only with the advent of psychometric meta-analysis was, it possible to fully reap the benefits of corrections for attenuation because the problem of sampling error was diminished by averaging across many samples and thereby increasing sample sizes. Since the advent of psychometric meta-analysis 38 years ago, scientific knowledge in the field of I-O psychology has greatly increased. Hundreds of meta-analyses have established basic scientific principles and tested theories.

Against this backdrop, LeBreton, Scherer, and James (2014) have written a focal article that distrusts corrections for unreliability in psychometric meta-analyses. They question the appropriateness of using interrater reliabilities of job performance ratings for corrections for attenuation in validity generalization studies. Because of length limitations on comments in this journal, we will address only major errors, not all errors in LeBreton et al.'s strident article. The focal article is unfortunately more emotional than rational in tone and conceptually and statistically confused. In our comment, we address only the two latter problems.

We have organized our comment in five major sections: (a) purpose of validation and logic of correction for attenuation, (b) reliability of overall job performance ratings, (c) validity estimation versus administrative decision use of criteria, (d) accurate validity estimates for predictors used in employee selection, and (e) correct modeling of job performance determinants.

Suggested Citation

Viswesvaran, Chockalingam (Vish) and Ones, Deniz S. and Schmidt, Frank L. and Le, Huy and Oh, In-Sue, Measurement Error Obfuscates Scientific Knowledge: Path to Cumulative Knowledge Requires Corrections for Unreliability and Psychometric Meta-Analyses (December 1, 2014). Industrial and Organizational Psychology (7,4), 2014, pp. 507-518.. Available at SSRN: https://ssrn.com/abstract=2550813

Chockalingam (Vish) Viswesvaran

Florida International University ( email )

University Park
11200 SW 8th Street
Miami, FL 33199
United States

Deniz S. Ones

University of Minnesota ( email )

Department of Psychology
75 East River Road
Minneapolis, MN 55455
United States

HOME PAGE: http://www.psych.umn.edy/faculty/ones

Frank L. Schmidt

University of Iowa - Henry B. Tippie College of Business ( email )

Acquisitions
5020 Main Library
Iowa City, IA 52242-1000
United States

Huy Le

University of Texas at San Antonio

One UTSA Circle
San Antonio, TX 78249
United States

In-Sue Oh (Contact Author)

Temple University - Department of Human Resource Management ( email )

1801 Liacouras Walk
Alter Hall 343
Philadelphia, PA 19122
United States

HOME PAGE: http://www.fox.temple.edu/mcm_people/in-sue-oh/

Register to save articles to
your library

Register

Paper statistics

Downloads
52
rank
374,731
Abstract Views
413
PlumX Metrics
!

Under construction: SSRN citations while be offline until July when we will launch a brand new and improved citations service, check here for more details.

For more information