Roc Confidence Bands: An Empirical Study

10 Pages Posted: 9 Oct 2008

See all articles by Sofus Mcskassy

Sofus Mcskassy

affiliation not provided to SSRN

Foster Provost

New York University

Saharon Rosset

affiliation not provided to SSRN

Date Written: 2005

Abstract

This paper is about constructing confidence bands around an ROCcurve such that (1 - \delta)% of the ROC curves traced by data setsof size r will fall completely within the bands. We introduce tothe machine learning community three methods from the medicalfield that are applicable to generate such bands. We then evaluatethese methods on the simple case of Ã'Â'à ¢Ã' à ¬Ã' à ÂSbinormalÃ'Â'à ¢Ã' à ¬Ã' à Â? distributionsÃ'Â'à ¢Ã' à ¬Ã' à  the scores for positive and the score for negative instances aredrawn from two normal distributions. We show that none of themethods generate appropriate bands and investigate two types ofvariances problems. We show that widening the bands does notproduce the proper bandwidths but that fitting a normal distributionto the observed drawn samples and drawing samples fromthis distribution (parametric bootstrap) does generate bands thatare much closer to the desired coverage although still not perfect.We tested the original methods as well as parametric bootstrap onthe covertype data set from the UCI ML-repority. The originalmethods perform the same as in the synthetic case, whereas theparametric bootstrap technique did not yield the expected results.This is primarily due to not being able to generate a good fit forthe score distributions. Whether it is possible to fit well-behavingparametric distribution to learned models is an open question weleave to the machine learning community to answer.

Suggested Citation

Mcskassy, Sofus and Provost, Foster and Rosset, Saharon, Roc Confidence Bands: An Empirical Study (2005). NYU Working Paper No. 2451/14107, Available at SSRN: https://ssrn.com/abstract=1281304

Sofus Mcskassy (Contact Author)

affiliation not provided to SSRN

No Address Available

Foster Provost

New York University ( email )

44 West Fourth Street
New York, NY 10012
United States

Saharon Rosset

affiliation not provided to SSRN

No Address Available

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
93
Abstract Views
1,275
Rank
502,806
PlumX Metrics