Who R U?: On the (In)Accuracy of Incumbent Based Estimates of Range Restriction in Criterion-Related and Differential Validity Research
Journal of Applied Psychology (Forthcoming)
69 Pages Posted: 20 Jan 2017 Last revised: 15 Mar 2017
Date Written: January 1, 2017
Correcting validity estimates for selection procedures for range restriction typically involves comparing variance in predictor scores between all job applicants and applicants who were selected. However, some research on criterion-related and differential validity of cognitive ability tests has relied on range restriction corrections based on data from job incumbents. Unfortunately, there remains ambiguity concerning the accuracy of this incumbent based approach vis-à-vis the applicant based approach. To address this issue, we conduct Monte Carlo simulations and an analysis of data from an actual selection process. Our first simulation study showed that incumbent based range restriction corrections result in downwardly biased estimates of criterion-related validity, whereas applicant based corrections were quite accurate. Our second set of simulations showed that incumbent based range restriction corrections can produce evidence of differential validity when there is no differential validity in the population. In contrast, applicant based corrections tended to accurately estimate population parameters and show little, if any, evidence of differential validity when there is no differential validity in the population. Analysis of data for the ACT as a predictor of academic performance revealed similar patterns of bias for incumbent based corrections in an academic setting. Overall, the present findings raise serious concerns regarding the use of incumbent based range restriction corrections in lieu of applicant based corrections. They also cast doubt on recent evidence for differential validity of predictors of job performance.
Keywords: criterion-related validity, differential validity, range restriction, personnel selection, cognitive ability tests, simulations, meta-analysis
Suggested Citation: Suggested Citation