Use of ROC Curve Analysis Gives Fallacious Results for Prediction: Use Predictivity-Based Indices

Abstract

The area under the ROC curve is frequently used for assessing the predictive efficacy of a model and the Youden index is commonly used to provide the optimal cut-off. Both are misleading tools for predictions. A ROC curve is drawn for the sensitivity of a quantitative test against its (1 – specificity) at different values of the test. Both sensitivity and specificity are retrospective in nature since these are indicators of correct classification of already known conditions. They are not indicators of future events and not valid for predictions because predictivity intimately depends on the prevalence which is ignored by sensitivity and specificity. We explain this fallacy in detail and illustrate with several examples that the actual predictivity could be very different from the ROC curve-based predictivity reported by many authors. The predictive efficacy of a test or a model is best assessed by the percentage correctly predicted in a prospective framework. We propose predictivity-based ROC curves as tools for providing predictivities at varying prevalence in different populations. For optimal cutoff for prediction, we propose a P-index where the sum of positive and negative predictivities is maximum after subtracting 1.

Note:

Funding Information: None.

Conflict of Interests: None.

Keywords: Area under the ROC curve, C-index, P-index, Prediction models, Predictivity, Predictivity-based ROC curve