Using Sensitive Data to Prevent Discrimination by AI: Does the GDPR Need a New Exception?

40 Pages Posted: 9 May 2022 Last revised: 11 May 2022

See all articles by Marvin van Bekkum

Marvin van Bekkum

iHub, Radboud University, Nijmegen

Frederik Zuiderveen Borgesius

iHub, Radboud University, Nijmegen

Abstract

Organisations can use artificial intelligence to make decisions about people for a variety of reasons, for instance, to select the best candidates from many job applications. However, AI systems can have discriminatory effects when used for decision-making. To illustrate, an AI system could reject applications of people with a certain ethnicity, while the organisation did not plan such ethnicity discrimination. But in Europe, an organisation runs into a problem when it wants to assess whether its AI system accidentally leads to ethnicity discrimination: the organisation may not know the applicants’ ethnicity. In principle, the GDPR bans the use of certain ‘special categories of data’ (sometimes called ‘sensitive data’), which include data on ethnicity, religion, and sexual preference. The proposal for an AI Act of the European Commission includes a provision that would enable organisations to use of special categories of data for their auditing AI systems. This paper asks whether the GDPR’s rules on special categories of personal data hinder the prevention of AIdriven discrimination. We argue that the GDPR does prohibit such use of special category data in many circumstances. We also map out the arguments for and against creating an exception to the GDPR’s ban on using special categories of personal data, to enable preventing discrimination by AI systems. The paper discusses European law, but the paper can be relevant outside Europe too, as many policymakers in the world grapple with the tension between privacy and non-discrimination policy.

Keywords: Non-discrimination, Data Protection, AI, Automated Decision-Making, Artificial Intelligence, Special Categories of Data, AI fairness testing, AI auditing, testing an AI system for discrimination

Suggested Citation

Van Bekkum, Marvin and Zuiderveen Borgesius, Frederik, Using Sensitive Data to Prevent Discrimination by AI: Does the GDPR Need a New Exception?. Available at SSRN: https://ssrn.com/abstract=4104823 or http://dx.doi.org/10.2139/ssrn.4104823

Marvin Van Bekkum (Contact Author)

iHub, Radboud University, Nijmegen ( email )

Postbus 9108
Nijmegen, 6500 HK
Netherlands

Frederik Zuiderveen Borgesius

iHub, Radboud University, Nijmegen ( email )

Nijmegen
Netherlands

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
137
Abstract Views
501
rank
294,393
PlumX Metrics