Debiasing In-Sample Policy Performance for Small-Data, Large-Scale Optimization

95 Pages Posted: 3 Jun 2021 Last revised: 1 Apr 2022

See all articles by Vishal Gupta

Vishal Gupta

Data Science and Operations, Marshall School of Business

Michael Huang

University of Southern California - Data Sciences and Operations

Paat Rusmevichientong

University of Southern California - Marshall School of Business

Date Written: June 2, 2021

Abstract

Motivated by the poor performance of cross-validation in settings where data are scarce,
we propose a novel estimator of the out-of-sample performance of a policy in data-driven optimization. Our approach exploits the optimization problem's sensitivity analysis to estimate the gradient of the optimal objective value with respect to the amount of noise in the data and uses the estimated gradient to debias the policy's in-sample performance. Importantly, unlike cross-validation techniques, our approach avoids sacrificing data for a test set, utilizes all data when training and, hence, is well-suited to settings where data are scarce. We prove bounds on the bias and variance of our estimator for optimization problems with uncertain objectives but known, potentially non-convex, feasible regions. For more specialized optimization problems where the feasible region is ``weakly-coupled" in a certain sense, we prove stronger results. Specifically, we provide explicit high-probability bounds on the error of our estimator that holds uniformly over a policy class and depends on the problem's dimension and policy class's complexity. Importantly, all of our bounds show that under mild conditions, the error of our estimator vanishes as the dimension of the optimization problem grows, even if the amount of available data remains small and constant. Said differently, we prove our estimator performs well in the small-data, large-scale regime. Finally, we numerically compare our proposed method to state-of-the-art approaches through a case-study on dispatching emergency medical response services using real data. Our method provides more accurate estimates of out-of-sample performance and learns better-performing policies.

Keywords: Data-driven optimization. Small-data, large-scale regime. Cross-validation. Large-scale optimization.

Suggested Citation

Gupta, Vishal and Huang, Michael and Rusmevichientong, Paat, Debiasing In-Sample Policy Performance for Small-Data, Large-Scale Optimization (June 2, 2021). Available at SSRN: https://ssrn.com/abstract=3858984 or http://dx.doi.org/10.2139/ssrn.3858984

Vishal Gupta

Data Science and Operations, Marshall School of Business ( email )

Marshall School of Business
BRI 401, 3670 Trousdale Parkway
Los Angeles, CA 90089
United States

HOME PAGE: http://www-bcf.usc.edu/~guptavis/

Michael Huang (Contact Author)

University of Southern California - Data Sciences and Operations ( email )

701 Exposition Blvd
Los Angeles, CA
United States

Paat Rusmevichientong

University of Southern California - Marshall School of Business ( email )

701 Exposition Blvd
Los Angeles, CA California 90089
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
124
Abstract Views
521
rank
309,396
PlumX Metrics