Auditing Human-Machine Communication Systems Using Simulated Humans

Handbook of Human-Machine Communication

20 Pages Posted: 2 Sep 2021

See all articles by Nicholas Diakopoulos

Nicholas Diakopoulos

Northwestern University - School of Communication

Jack Bandy

Northwestern University

Henry Dambanemuya

Northwestern University

Date Written: August 31, 2021

Abstract

Audit methods have emerged as a powerful approach for helping to describe and understand the outputs of algorithmic systems. In this chapter we examine the methodological considerations and limitations associated with simulating humans in order to systematically audit the behavior of Human-Machine Communication (HMC) systems. We first detail two recent audits we undertook—one of the Amazon Alexa voice assistant and the other of the Twitter algorithmic timeline—where we developed techniques for simulating users to systematically gather data and assess various aspects of each system. We then elaborate methodological considerations in terms of system access, audit scope, simulation fidelity, and other ethical issues, drawing on our case studies and other experiences to offer concrete illustrations and advice for researchers interested in applying audit methods to HMC systems. New opportunities for methodological development are also discussed.

Keywords: Auditing, Algorithm Auditing, Algorithmic Curation, Twitter, Alexa, Simulation, Algorithms

Suggested Citation

Diakopoulos, Nicholas and Bandy, John and Dambanemuya, Henry, Auditing Human-Machine Communication Systems Using Simulated Humans (August 31, 2021). Handbook of Human-Machine Communication, Available at SSRN: https://ssrn.com/abstract=3915192 or http://dx.doi.org/10.2139/ssrn.3915192

Nicholas Diakopoulos (Contact Author)

Northwestern University - School of Communication ( email )

70 Arts Circle Drive
Office of the Dean
Evanston, IL 60208
United States

HOME PAGE: http://www.nickdiakopoulos.com/

John Bandy

Northwestern University ( email )

2001 Sheridan Road
Evanston, IL 60208
United States

Henry Dambanemuya

Northwestern University ( email )

2001 Sheridan Road
Evanston, IL 60208
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
7
Abstract Views
131
PlumX Metrics