How the Machine 'Thinks:' Understanding Opacity in Machine Learning Algorithms

12 Pages Posted: 15 Sep 2015

See all articles by Jenna Burrell

Jenna Burrell

Data & Society Research Institute; University of California, Berkeley - School of Information

Date Written: September 15, 2015


This article considers the issue of opacity as a problem for socially consequential mechanisms of classification and ranking, such as spam filters, credit card fraud detection, search engines, news trends, market segmentation and advertising, insurance or loan qualification, and credit scoring. These mechanisms of classification all frequently rely on computational algorithms, and in many cases on machine learning algorithms to do this work. In this article, I draw a distinction between three forms of opacity: (1) opacity as intentional corporate or state secrecy (2) opacity as technical illiteracy, and (3) an opacity that arises from the characteristics of machine learning algorithms and the scale required to apply them usefully. The analysis in this article gets inside the algorithms themselves. I cite existing literatures in computer science, known industry practices (as they are publicly presented), and do some testing and manipulation of code as a form of lightweight code audit. I argue that recognizing the distinct forms of opacity that may be coming into play in a given application is key to determining which of a variety of technical and non-technical solutions could help to prevent harm.

Keywords: opacity, machine learning, classification, inequality, discrimination, spam filtering

Suggested Citation

Burrell, Jenna, How the Machine 'Thinks:' Understanding Opacity in Machine Learning Algorithms (September 15, 2015). Available at SSRN: or

Jenna Burrell (Contact Author)

Data & Society Research Institute ( email )

36 West 20th Street
11th Floor
New York,, NY 10011
United States

University of California, Berkeley - School of Information ( email )

102 South Hall
Berkeley, CA 94720-4600
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Abstract Views
PlumX Metrics