Evaluating Risk Assessments Using Receiver Operating Characteristic (ROC) Analysis: Rationale, Advantages, Insights, and Limitations

Abstract

The last two decades have witnessed major changes in the way that mental health professionals assess, describe, and think about persons’ risk for future violence. Psychiatrists and psychologists have gone from believing that they could not predict violence to feeling certain they can assess violence risk with well-above-chance accuracy. Receiver operating characteristic (ROC) analysis has played a central role in changing this view. This article reviews the key concepts underlying ROC methods, the meaning of the area under the ROC curve (AUC), the relationship between AUC and effect size d, and what these two indices tell us about evaluations of violence risk.

AUC and d provide succinct but incomplete descriptions of discrimination capacity. These indices do not provide details about sensitivity-specificity trade-offs; they do not tell us how to balance false positive and false negative errors; and they do not determine whether a diagnostic system is accurate enough to make practically useful distinctions between violent and nonviolent subject groups. Justifying choices or clinical practices requires a contextual investigation of outcomes, a process that takes us beyond simply knowing global indices of accuracy.

Keywords: violence, risk assessment, receiver operating characteristic analysis, ROC, accuracy

JEL Classification: D81, C19