Tractable Robust Supervised Learning Models

Abstract

At the heart of supervised learning is a minimization problem with an objective function that evaluates a set of training data over a loss function that penalizes poor fitting and a regularization function that penalizes over-fitting to the training data. More recently, data-driven robust optimization based learning models provide an intuitive robustness perspective of regularization. However, when the loss function is not Lipschitz continuous, solving the robust learning models exactly can be computationally challenging. We focus on providing tractable approximations for robust regression and classification problems for loss functions derived from Lipschitz continuous functions raised to the power of p. We also show the equivalence of the type-p robust learning models to the pth-root regularization problems when the underlying support sets are unbounded. Inspired by Long et al. (2021), we also propose tractable type-p robust satisficing learning models that are specified by target loss parameters. We illustrate that the robust satisficing regression and classification models can be tractably solved for a large class of problems, and we also establish finite sample probabilistic guarantees for limiting losses beyond the specified target. While the family of solutions generated by regularization and robust satisficing can be the same, from empirical studies on popular datasets, the relative targets for reasonably good out-of-sample performance can be found within a narrow range. We also demonstrate in the numerical study that the target-based hyper-parameter is easier to determine via cross-validation and can improve out-of-sample performance compared to standard regularization approaches.

Keywords: Regression, classification, regularization, robust optimization, robust satisficing