Beyond Automation: Machine Learning-Based Systems and Human Behavior in the Personalization Economy

Abstract

Corporations seek to exploit the enormous economic potential of personalization beyond the confinements of the online space. They increasingly rely on machine learning-based systems to select and evaluate their customers. These systems define how we are perceived and tailor the way in which are treated. The implications of these systems are already immense, and they foreshadow a larger transformation. Over the course of the 21st century, ubiquitous, machine learning-based personalization will become a fundamental condition of human existence, shaping our behavior and constraining our freedom.

Legal scholarship needs a conceptual foundation to address urgent questions about how personalization, driven by machine learning-based decision-making, affects liberty and other liberal democratic values. Drawing on surveillance theory, this article develops that foundation and examines the normative and constitutional implications of ubiquitous personalization. The article builds on the concepts of panopticism and the surveillance assemblage to analyze how machine learning-based decision-making amplifies corporate power and governs human behavior. It argues that existing legal responses, focusing on rights, explainability and transparency, fail to prevent the already fragile balance of power between individuals and corporations from tipping in corporations’ interest. To adequately respond to machine learning-based decision-making, legal scholarship must overcome its individualistic focus and engage in a debate on the legitimacy of corporate surveillance.

Keywords: Big Data, AI, Automated Decision-Making, Corporate Surveillance, Panopticism, Surveillance Assemblage, Legitimacy, machine-learning, transparency, fundamental rights