The Siren Call of Content Moderation Formalism
NEW TECHNOLOGIES OF COMMUNICATION AND THE FIRST AMENDMENT: THE INTERNET, SOCIAL MEDIA AND CENSORSHIP (Lee Bollinger & Geoffrey Stone eds., 2022 Forthcoming)
24 Pages Posted: 12 Jan 2022
Date Written: January 10, 2022
Systems of online content moderation governance are becoming some of the most elaborate and extensive bureaucracies in history, and they are deeply imperfect and need reform. Would-be reformers of content moderation systems are drawn to a highly rule-bound and formalistic vision of how these bureaucracies should operate, but the sprawling chaos of online speech is too vast, ever-changing, and varied to be brought into consistent compliance with rigid rules. This essay argues that the quest to make content moderation systems ever more formalistic will not fix public and regulatory concerns about the legitimacy and accountability of how platforms moderate content on their services. The largest social media platforms operate massive unelected, unaccountable, and increasingly complex bureaucracies that decide to act or not act on millions of pieces of content uploaded to their platforms every day. A formalistic model, invoking judicial-style norms of reasoning and precedent, is doomed to fail at this scale and complexity. As these governance systems mature, it is time to be content moderation realists about the task ahead.
Keywords: content moderation, formalism, realism, transparency, platform governance
Suggested Citation: Suggested Citation