Anti-discrimination Laws, AI, and Gender Bias: A Case Study in Non-mortgage Fintech Lending

Stephanie Kelley, Anton Ovchinnikov, David R. Hardoon, Adrienne Heinrich (2022) Antidiscrimination Laws, Artificial Intelligence, and Gender Bias: A Case Study in Nonmortgage Fintech Lending. Manufacturing & Service Operations Management 0(0). https://doi.org/10.1287/msom.2022.1108

Posted: 23 Nov 2020 Last revised: 15 Nov 2022

See all articles by Stephanie Kelley

Stephanie Kelley

Saint Mary's University, Canada - Department of Finance, Information Systems & Management Science

Anton Ovchinnikov

Smith School of Business - Queen's University; INSEAD - Decision Sciences

David R. Hardoon

Aboitiz Data Innovation; Union Bank of Philippines

Adrienne Heinrich

UnionBank of the Philippines; Aboitiz Data Innovation

Date Written: November 1, 2021

Abstract

Problem definition: We use a realistically large, publicly available dataset from a global fintech lender to simulate the impact of different anti-discrimination laws, and their corresponding data management and model building regimes, on gender-based discrimination in the non-mortgage fintech lending setting.

Academic/Practical Relevance: Our paper extends the conceptual understanding of model-based discrimination from computer science to a realistic context that simulates the situations faced by fintech lenders in practice, where advanced machine learning (ML) techniques are used with high-dimensional, feature rich, highly multi-collinear data. We provide technically and legally permissible approaches for firms to reduce discrimination across different anti-discrimination regimes, whilst managing profitability.

Methodology: We train statistical and ML models on a large and realistically rich, publicly available dataset to simulate different anti-discrimination regimes, and measure their impact on model discrimination, predictive quality, and firm profitability. We use ML explainability techniques to understand the drivers of ML discrimination.

Results: We find that regimes that prohibit the use of gender (like those in the United States) substantially increase discrimination, and slightly decrease firm profitability. We observe that ML models are less discriminatory, of better predictive quality, and more profitable compared to traditional statistical models like logistic regression. Unlike omitted variable bias—which drives discrimination in statistical models—ML discrimination is driven by changes in the model training procedure, including feature engineering and feature selection, when gender is excluded. We observe that down-sampling the training data to rebalance gender, gender-aware hyperparameter selection, and up-sampling the training data to rebalance gender, all reduce discrimination, with varying trade-offs in predictive quality and firm profitability. Probabilistic gender proxy modeling (imputing applicant gender) further reduces discrimination with negligible impact on predictive quality and a slight increase in firm profitability.

Managerial Implications: A rethink is required of the anti-discrimination laws, specifically with respect to the collection and use of protected attributes for machine learning models. Firms should be able to collect protected attributes to, at minimum, measure discrimination, and, ideally, take steps to reduce it. Increased data access should come with greater accountability for firms.



Keywords: discrimination, bias, ethics, law, fintech, artificial intelligence, machine learning, gender

Suggested Citation

Kelley, Stephanie and Ovchinnikov, Anton and Hardoon, David and Heinrich, Adrienne, Anti-discrimination Laws, AI, and Gender Bias: A Case Study in Non-mortgage Fintech Lending (November 1, 2021). Stephanie Kelley, Anton Ovchinnikov, David R. Hardoon, Adrienne Heinrich (2022) Antidiscrimination Laws, Artificial Intelligence, and Gender Bias: A Case Study in Nonmortgage Fintech Lending. Manufacturing & Service Operations Management 0(0). https://doi.org/10.1287/msom.2022.1108, Available at SSRN: https://ssrn.com/abstract=3719577 or http://dx.doi.org/10.2139/ssrn.3719577

Stephanie Kelley (Contact Author)

Saint Mary's University, Canada - Department of Finance, Information Systems & Management Science ( email )

Halifax, Nova Scotia B3H 3C3
Canada

Anton Ovchinnikov

Smith School of Business - Queen's University ( email )

143 Union Str. West
Kingston, ON K7L3N6
Canada

INSEAD - Decision Sciences ( email )

United States

David Hardoon

Aboitiz Data Innovation ( email )

Singapore
Singapore

Union Bank of Philippines ( email )

Pasig
Philippines

Adrienne Heinrich

UnionBank of the Philippines ( email )

Metro Manila
Philippines

Aboitiz Data Innovation ( email )

Singapore
Singapore

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
2,707
PlumX Metrics