Evaluating Discrete Choice Prediction Models When the Evaluation Data is Corrupted: Analytic Results and Bias Corrections for the Area Under the ROC

Data Mining and Knowledge Discovery, Forthcoming

Posted: 20 Dec 2015

See all articles by Roger Stein

Roger Stein

Sloan School of Management, MIT

Date Written: September 15, 2015

Abstract

There has been a growing recognition that issues of data quality, which are routine in practice, can materially affect the assessment of learned model performance. In this paper, we develop some analytic results that are useful in sizing the biases associated with tests of discriminatory model power when these are performed using corrupt (“noisy”) data. As it is sometimes unavoidable to test models with data that are known to be corrupt, we also provide some guidance on interpreting results of such tests. In some cases, with appropriate knowledge of the corruption mechanism, the true values of the performance statistics such as the area under the ROC curve may be recovered (in expectation), even when the underlying data have been corrupted. We also provide estimators of the standard errors of such recovered performance statistics. An analysis of the estimators reveals interesting behavior including the observation that “noisy” data does not “cancel out” across models even when the same corrupt data set is used to test multiple candidate models. Because our results are analytic, they may be applied in a broad range of settings and this can be done without the need for simulation.

Keywords: ROC Model validation, Prediction, Data corruption, Bias correction, Misclassification, Credit models, Machine learning

Suggested Citation

Stein, Roger, Evaluating Discrete Choice Prediction Models When the Evaluation Data is Corrupted: Analytic Results and Bias Corrections for the Area Under the ROC (September 15, 2015). Data Mining and Knowledge Discovery, Forthcoming. Available at SSRN: https://ssrn.com/abstract=2705816

Roger Stein (Contact Author)

Sloan School of Management, MIT ( email )

100 Main Street
E62-416
Cambridge, MA 02142
United States

HOME PAGE: http://www.rogermstein.com

Register to save articles to
your library

Register

Paper statistics

Abstract Views
119
PlumX Metrics