Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis

37 Pages Posted: 19 Nov 2014 Last revised: 11 Mar 2016

See all articles by Ceren Budak

Ceren Budak

University of Michigan at Ann Arbor

Sharad Goel

Stanford University

Justin M. Rao

Microsoft Research; Microsoft Corporation - Microsoft Research - Redmond

Date Written: November 17, 2014


It is widely thought that news organizations exhibit ideological bias, but rigorously quantifying such slant has proven methodologically challenging. Through a combination of machine learning and crowdsourcing techniques, we investigate the selection and framing of political issues in 15 major U.S. news outlets. Starting with 803,146 news stories published over 12 months, we first used supervised learning algorithms to identify the 14% of articles pertaining to political events. We then recruited 749 online human judges to classify a random subset of 10,950 of these political articles according to topic and ideological position. Our analysis yields an ideological ordering of outlets consistent with prior work. We find, however, that news outlets are considerably more similar than generally believed. Specifically, with the exception of political scandals, we find that major news organizations present topics in a largely non-partisan manner, casting neither Democrats nor Republicans in a particularly favorable or unfavorable light. Moreover, again with the exception of political scandals, there is little evidence of systematic differences in story selection, with all major news outlets covering a wide variety of topics with frequency largely unrelated to the outlet's ideological position. Finally, we find that news organizations express their ideological bias not by directly advocating for a preferred political party, but rather by disproportionately criticizing one side, a convention that further moderates overall differences.

Keywords: media bias, text analysis, crowdsourcing, big data

JEL Classification: D83, L10, L82

Suggested Citation

Budak, Ceren and Goel, Sharad and Rao, Justin M., Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis (November 17, 2014). Available at SSRN: https://ssrn.com/abstract=2526461 or http://dx.doi.org/10.2139/ssrn.2526461

Ceren Budak (Contact Author)

University of Michigan at Ann Arbor ( email )

110 Tappan Hall
855 S. University Ave
Ann Arbor, MI 48109
United States

Sharad Goel

Stanford University ( email )

475 Via Ortega
Stanford, CA 94305
United States

HOME PAGE: http://5harad.com

Justin M. Rao

Microsoft Research ( email )

641 Avenue of Americas
7th Floor
New York, NY 11249
United States

Microsoft Corporation - Microsoft Research - Redmond ( email )

Building 99
Redmond, WA
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Abstract Views
PlumX Metrics