Signal Detection Theory Fails to Account for Real-World Consequences of Inconclusive Decisions

Abstract

This response to Arkes and Koehler’s important paper, Inconclusives and Error Rates In Forensic Science: A Signal Detection Theory Approach, explains that the authors’ approach: (1) neglects to account for known differences in how inconclusive decisions are deployed by examiners across various feature-comparison methods and (2) leaves readers with the misimpression that blind proficiency testing may serve as a near-complete solution to the complex problems associated with assessing the validity of feature-comparison methods.

Arkes and Koehler describe three categories of feature-comparison examiners: conservative (focused on minimizing error); confident (focused on drawing a firm source conclusion); and strategic (acting conservatively or confidently depending on whether they are in a test situation or examining case work). While it may apply to some disciplines under certain scenarios, firearms examiners routinely apply an approach that falls outside of this scheme. The published literature demonstrates that firearms examiners do not draw inconclusive decisions evenly between same-source and different-source comparisons. Rather, they consistently render inconclusive decisions more frequently when data supports exclusion rather than identification.

This approach is best described by a fourth category, which we term, the Biased Examiner. As this response explains, the Biased Examiner problem results in significant prejudice to the accused and may contribute to miscarriages of justice. And, because it stems from a number of sources, the Biased Examiner problem cannot be fully resolved by blind proficiency testing.

Keywords: forensic evidence, friction ridge, firearms examiners, wrongful convictions, comparison algorithms, systemic bias