Quantifying the Accuracy of Forensic Examiners in the Absence of a Diagnostic 'Gold Standard'

Abstract:     
Background: In medicine, diagnostic accuracy is usually evaluated against a near-infallible criterion - a "gold standard" - for true disease status. Most mental health classifications have no gold standard, however, and absence of agreed-upon truth is common for psycholegal assessments.

Aims: To show that even without a gold standard, accuracy of forensic assessments can be quantified using latent class methods and receiver operating characteristic (ROC) analysis.

Method: Using redacted court reports, clinicians rated 156 hospitalized criminal defendants on their Dusky-defined competence to stand trial and on their understanding, appreciation, and reasoning about criminal proceedings. Multiple ratings per evaluee permitted estimation of ROC parameters using maximum likelihood and Bayesian approaches.

Results: Raters were highly accurate, with ROC areas averaging 0.967 or more.

Conclusions: Accuracy of forensic examinations can be quantified numerically despite the absence of a diagnostic gold standard. Methods used in this study should be applicable to many other psycholegal questions where quantifying accuracy would have scientific and evidentiary value.

Keywords: competence to stand trial, adjudicative competence, ROC analysis, diagnostic accuracy, maximum likelihood, Bayesian, gold standard

JEL Classifications: C11, C14, C15, C63, K14

Working Paper Series