Confidence Bands for Roc Curves

9 Pages Posted: 9 Oct 2008

See all articles by Sofus Macskassy

Sofus Macskassy

Fetch Technologies, Inc

Foster Provost

New York University

Multiple version iconThere are 2 versions of this paper

Date Written: 2004

Abstract

In this paper we study techniques for generating and evaluatingconfidence bands on ROC curves. ROC curve evaluation israpidly becoming a commonly used evaluation metric in machinelearning, although evaluating ROC curves has thus far been limitedto studying the area under the curve (AUC) or generation ofone-dimensional confidence intervals by freezing one variableâ€"the false-positive rate, or threshold on the classification scoringfunction. Researchers in the medical field have long been usingROC curves and have many well-studied methods for analyzingsuch curves, including generating confidence intervals aswell as simultaneous confidence bands. In this paper we introducethese techniques to the machine learning community andshow their empirical fitness on the Covertype data setâ€"a standardmachine learning benchmark from the UCI repository. Weshow how some of these methods work remarkably well, othersare too loose, and that existing machine learning methods for generationof 1-dimensional confidence intervals do not translate wellto generation of simultaneous bandsâ€"their bands are too tight.

Suggested Citation

Macskassy, Sofus and Provost, Foster, Confidence Bands for Roc Curves (2004). Information Systems Working Papers Series, Vol. , pp. -, 2004. Available at SSRN: https://ssrn.com/abstract=1281313

Sofus Macskassy (Contact Author)

Fetch Technologies, Inc ( email )

2041 Rosecrans Ave
Suite 245
El Segundo, CA 90245
United States

HOME PAGE: http://www.fetch.com

Foster Provost

New York University ( email )

44 West Fourth Street
New York, NY 10012
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
97
Abstract Views
786
rank
203,428
PlumX Metrics