Confidence Bands for Roc Curves

9 Pages Posted: 9 Oct 2008

See all articles by Sofus Macskassy

Sofus Macskassy

Fetch Technologies, Inc

Foster Provost

New York University

Multiple version iconThere are 2 versions of this paper

Date Written: 2004


In this paper we study techniques for generating and evaluatingconfidence bands on ROC curves. ROC curve evaluation israpidly becoming a commonly used evaluation metric in machinelearning, although evaluating ROC curves has thus far been limitedto studying the area under the curve (AUC) or generation ofone-dimensional confidence intervals by freezing one variableâ€"the false-positive rate, or threshold on the classification scoringfunction. Researchers in the medical field have long been usingROC curves and have many well-studied methods for analyzingsuch curves, including generating confidence intervals aswell as simultaneous confidence bands. In this paper we introducethese techniques to the machine learning community andshow their empirical fitness on the Covertype data setâ€"a standardmachine learning benchmark from the UCI repository. Weshow how some of these methods work remarkably well, othersare too loose, and that existing machine learning methods for generationof 1-dimensional confidence intervals do not translate wellto generation of simultaneous bandsâ€"their bands are too tight.

Suggested Citation

Macskassy, Sofus and Provost, Foster, Confidence Bands for Roc Curves (2004). NYU Working Paper No. 2451/14116, Available at SSRN:

Sofus Macskassy (Contact Author)

Fetch Technologies, Inc ( email )

2041 Rosecrans Ave
Suite 245
El Segundo, CA 90245
United States


Foster Provost

New York University ( email )

44 West Fourth Street
New York, NY 10012
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
PlumX Metrics