Human Intervention in Automated Decision-Making: Toward the Construction of Contestable Systems

Abstract

Concerns about “black box” machine learning algorithms have led many data protection laws and regulations to establish a right to human intervention on decision-making supported by artificial intelligence. Such interventions provide data subjects with means to protect their rights, freedoms, and legitimate interests, either as a bare minimum requirement for data processing or as a central norm governing decision-aiding artificial intelligence. In this paper, I present contestability by design as an approach to two kinds of issues with current legal implementations of the right to human intervention. The first kind is the uncertainty about what kind of decision should be covered by this right: while a narrow reading of rules such as GDPR Article 22(3) would include all sorts of fully-automated decisions, I show how a broader interpretation can provide more effective protection for data subjects against side-effects of automated decision-making. The second class of issues ensues from practical effects, or the lack thereof, of this right to intervention: even within a clear conceptual framework, data subjects might still lack the information they need to the concrete exercise of their right, or the human intervention itself might introduce biases and limitations that result in undesirable outcomes. After discussing how those effects can be identified and measured, I then explore how the proper protection of the rights of data subjects is possible only if the possibility of contesting decisions based solely on automated processing is not an afterthought, but instead a requirement at each stage of an artificial intelligence system’s lifecycle.

Keywords: Automated decision-making, algorithmic bias, machine learning regulation, contestability by design, privacy by design