Confidence Bands for Roc Curves

Abstract

In this paper we study techniques for generating and evaluatingconfidence bands on ROC curves. ROC curve evaluation israpidly becoming a commonly used evaluation metric in machinelearning, although evaluating ROC curves has thus far been limitedto studying the area under the curve (AUC) or generation ofone-dimensional confidence intervals by freezing one variableâ€"the false-positive rate, or threshold on the classification scoringfunction. Researchers in the medical field have long been usingROC curves and have many well-studied methods for analyzingsuch curves, including generating confidence intervals aswell as simultaneous confidence bands. In this paper we introducethese techniques to the machine learning community andshow their empirical fitness on the Covertype data setâ€"a standardmachine learning benchmark from the UCI repository. Weshow how some of these methods work remarkably well, othersare too loose, and that existing machine learning methods for generationof 1-dimensional confidence intervals do not translate wellto generation of simultaneous bandsâ€"their bands are too tight.