The Best Way to Select Features? Comparing MDA, LIME, and SHAP

The Journal of Financial Data Science Winter 2021, 3 (1) 127-139; DOI: https://doi.org/10.3905/jfds.2020.1.047

Posted: 14 Jul 2021

Date Written: September 15, 2020

Abstract

Feature selection in machine learning is subject to the intrinsic randomness of the feature selection algorithms (e.g., random permutations during MDA). The stability of selected features with respect to such randomness is essential to the human interpretability of a machine learning algorithm. The authors propose a rank-based stability metric called the instability index to compare the stabilities of three feature selection algorithms—MDA, LIME, and SHAP—as applied to random forests. Typically, features are selected by averaging many random iterations of a selection algorithm. Although the variability of the selected features does decrease as the number of iterations increases, it does not go to zero, and the features selected by the three algorithms do not necessarily converge to the same set. LIME and SHAP are found to be more stable than MDA, and LIME is at least as stable as SHAP for the top-ranked features. Hence, overall, LIME is best suited for human interpretability. However, the selected set of features from all three algorithms significantly improves various predictive metrics out of sample, and their predictive performance does not differ significantly. Experiments were conducted on synthetic datasets, two public benchmark datasets, an S&P 500 dataset, and on proprietary data from an active investment strategy.

Keywords: Feature selection, instability index, importance score, trading strategy

JEL Classification: C58, G17

Suggested Citation

Man, Xin and Chan, Ernest, The Best Way to Select Features? Comparing MDA, LIME, and SHAP (September 15, 2020). The Journal of Financial Data Science Winter 2021, 3 (1) 127-139; DOI: https://doi.org/10.3905/jfds.2020.1.047, Available at SSRN: https://ssrn.com/abstract=3880618

Xin Man

PredictNow.ai ( email )

56 Niagara on the Green Blvd
Niagara-on-the-Lake, L0S 1J0
Canada

Ernest Chan (Contact Author)

PredictNow.ai ( email )

56 Niagara on the Green Blvd
Niagara-on-the-Lake, L0S 1J0
Canada

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
415
PlumX Metrics