The Algorithmic Management of Polarization and Violence on Social Media

Knight First Amendment Institute. KnightColumbia.Org, forthcoming

36 Pages Posted: 6 May 2023 Last revised: 25 May 2023

See all articles by Jonathan Stray

Jonathan Stray

University of California, Berkeley - Center for Human-Compatible AI

Ravi Iyer

University of Southern California

Helena Puig Larrauri

Build Up

Date Written: May 25, 2023

Abstract

Social media platforms are involved in all aspects of social life, including in conflict settings. Incidental choices about how they are designed can have profound effects on people when conflict has the potential to escalate to violence. We review theories of conflict escalation and the practice of professional peacebuilders, and distinguish between constructive conflict, which can be part of important societal changes, and destructive conflict where positions become more identity based and intractable. Platforms have largely responded to conflict through content moderation thus far, yet moderation will never affect more than a small amount of objectively policy-violating content, and expanding those efforts will only lead to more backtracking, biased enforcement, and controversy. Instead we draw on recently-published platform experiments, the reports of content creators, international peacebuilding practitioners, and the experiences of those in conflict settings to argue that platforms often incentivize conflict actors toward more divisive and potentially violence-inducing speech, while also facilitating mass harassment and manipulation. We propose that platforms monitor for the conflict relevant side effects of prioritizing distribution based on engagement, such as the incentivization of divisive content, and that they deprecate certain engagement signals (such as comments, shares or time spent) in sensitive contexts. It may also be possible for platforms to support the transformation from destructive to constructive conflict by drawing attention to cross-cutting content, and supporting the on-platform efforts of conflict transformation professionals. To produce widespread legitimacy for these efforts, and overcome the problem of business incentives, we recommend the public creation of clear guidelines for conflict-sensitive platform design, including new kinds of practical conflict metrics.

Keywords: social media, conflict, polarization, violence, algorithms

Suggested Citation

Stray, Jonathan and Iyer, Ravi and Puig Larrauri, Helena, The Algorithmic Management of Polarization and Violence on Social Media (May 25, 2023). Knight First Amendment Institute. KnightColumbia.Org, forthcoming, Available at SSRN: https://ssrn.com/abstract=4429558 or http://dx.doi.org/10.2139/ssrn.4429558

Jonathan Stray

University of California, Berkeley - Center for Human-Compatible AI

Ravi Iyer (Contact Author)

University of Southern California ( email )

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
595
Abstract Views
3,210
Rank
97,669
PlumX Metrics