Biased Programmers? Or Biased Data? A Field Experiment in Operationalizing AI Ethics
In Proceedings of the 21st ACM Conference on Economics and Computation (pp. 679-681).
Posted: 24 Jun 2020
Date Written: June 1, 2020
Why do biased beliefs arise, and what interventions can prevent them? We study this topic in a field experiment about using machine learning to predict human capital. We randomly assign ~400 AI engineers to predict standardized test scores of OECD residents under different experimental conditions. We then assess the resulting predictive algorithms using the realized test performance and through randomized audit-like manipulations of algorithmic inputs. We find that biased beliefs are mostly caused by biased training data. However, simple reminders about bias are almost half as effective at fully de-biasing training data. We find mixed results on technical education and programmer demographics. Programmers who understand technical guidance successfully reduce bias. However, many do not follow the advice, resulting in algorithms that are worse than programmers given a simple reminder. Predictions by female and minority AI programmers do not exhibit less bias or discrimination. However prediction errors are correlated within engineering demographics, creating bias reductions from cross-demographic averaging. We also find no effects of our incentive treatments and no evidence that programmers' implicit associations between gender and math (measured through an IAT) are correlated with bias in code.
Suggested Citation: Suggested Citation