Who R U?: On the (In)Accuracy of Incumbent Based Estimates of Range Restriction in Criterion-Related and Differential Validity Research

Journal of Applied Psychology (Forthcoming)

Fox School of Business Research Paper No. 17-007

69 Pages Posted: 20 Jan 2017 Last revised: 15 Mar 2017

See all articles by Philip L. Roth

Philip L. Roth

Clemson University - Department of Management

Huy Le

University of Texas at San Antonio

In‐Sue Oh

Temple University - Department of Human Resource Management

Chad H. Van Iddekinge

Florida State University - College of Business

Steven Robbins

Independent

Date Written: January 1, 2017

Abstract

Correcting validity estimates for selection procedures for range restriction typically involves comparing variance in predictor scores between all job applicants and applicants who were selected. However, some research on criterion-related and differential validity of cognitive ability tests has relied on range restriction corrections based on data from job incumbents. Unfortunately, there remains ambiguity concerning the accuracy of this incumbent based approach vis-à-vis the applicant based approach. To address this issue, we conduct Monte Carlo simulations and an analysis of data from an actual selection process. Our first simulation study showed that incumbent based range restriction corrections result in downwardly biased estimates of criterion-related validity, whereas applicant based corrections were quite accurate. Our second set of simulations showed that incumbent based range restriction corrections can produce evidence of differential validity when there is no differential validity in the population. In contrast, applicant based corrections tended to accurately estimate population parameters and show little, if any, evidence of differential validity when there is no differential validity in the population. Analysis of data for the ACT as a predictor of academic performance revealed similar patterns of bias for incumbent based corrections in an academic setting. Overall, the present findings raise serious concerns regarding the use of incumbent based range restriction corrections in lieu of applicant based corrections. They also cast doubt on recent evidence for differential validity of predictors of job performance.

Keywords: criterion-related validity, differential validity, range restriction, personnel selection, cognitive ability tests, simulations, meta-analysis

Suggested Citation

Roth, Philip L. and Le, Huy and Oh, In-Sue and Van Iddekinge, Chad H. and Robbins, Steven, Who R U?: On the (In)Accuracy of Incumbent Based Estimates of Range Restriction in Criterion-Related and Differential Validity Research (January 1, 2017). Journal of Applied Psychology (Forthcoming); Fox School of Business Research Paper No. 17-007. Available at SSRN: https://ssrn.com/abstract=2902033

Philip L. Roth

Clemson University - Department of Management ( email )

101 Sirrine Hall
Clemson, SC 29634
United States
864-656-2015 (Phone)

HOME PAGE: http://business.clemson.edu/Managemt/faculty/l3_fac_Roth.html

Huy Le

University of Texas at San Antonio

One UTSA Circle
San Antonio, TX 78249
United States

In-Sue Oh (Contact Author)

Temple University - Department of Human Resource Management ( email )

1801 Liacouras Walk
Alter Hall 343
Philadelphia, PA 19122
United States

HOME PAGE: http://www.fox.temple.edu/mcm_people/in-sue-oh/

Chad H. Van Iddekinge

Florida State University - College of Business ( email )

423 Rovetta Business Building
Tallahassee, FL 32306-1110
United States

Steven Robbins

Independent ( email )

No Address Available

Register to save articles to
your library

Register

Paper statistics

Downloads
13
Abstract Views
205
PlumX Metrics