Visualizing The Implicit Model Selection Tradeoff

He, Zezhen, and Yaron Shaposhnik. "Visualizing the Implicit Model Selection Tradeoff." Journal of Artificial Intelligence Research 76 (2023): 829-881.

56 Pages Posted: 25 Oct 2021 Last revised: 4 Apr 2023

See all articles by Zezhen (Dawn) He

Zezhen (Dawn) He

University of Rochester - Simon Business School

Yaron Shaposhnik

University of Rochester - Simon Business School

Date Written: October 21, 2021

Abstract

The recent rise of machine learning (ML) has been leveraged by practitioners and researchers to provide new solutions to an ever growing number of business problems. As with other ML applications, these solutions rely on model selection, which is typically achieved by evaluating certain metrics on models separately and selecting the model whose evaluations (i.e., accuracy-related loss and/or certain interpretability measures) are optimal. However, empirical evidence suggests that, in practice, multiple models often attain competitive results. Therefore, while models' overall performance could be similar, they could operate quite differently. This results in an implicit tradeoff in models' performance throughout the feature space which resolving requires new model selection tools.

This paper explores methods for comparing predictive models in an interpretable manner to uncover the tradeoff and help resolve it. To this end, we propose various methods that synthesize ideas from supervised learning, unsupervised learning, dimensionality reduction, and visualization to demonstrate how they can be used to inform model developers about the model selection process. Using various datasets and a simple Python interface, we demonstrate how practitioners and researchers could benefit from applying these approaches to better understand the broader impact of their model selection choices.

Keywords: Model selection, Interpretability, Machine learning, Visualization

Suggested Citation

He, Zezhen and Shaposhnik, Yaron, Visualizing The Implicit Model Selection Tradeoff (October 21, 2021). He, Zezhen, and Yaron Shaposhnik. "Visualizing the Implicit Model Selection Tradeoff." Journal of Artificial Intelligence Research 76 (2023): 829-881., Available at SSRN: https://ssrn.com/abstract=3946701 or http://dx.doi.org/10.2139/ssrn.3946701

Zezhen He (Contact Author)

University of Rochester - Simon Business School ( email )

Rochester, NY 14627
United States

Yaron Shaposhnik

University of Rochester - Simon Business School ( email )

Rochester, NY 14627
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
92
Abstract Views
493
Rank
493,900
PlumX Metrics