Download This Paper Open PDF in Browser

Crowdsourcing Judgments of News Source Quality

21 Pages Posted: 17 Feb 2018  

Gordon Pennycook

Yale University

David G. Rand

Yale University

Date Written: February 7, 2018

Abstract

The spread of misinformation and disinformation, especially on social media, is a major societal challenge. Here, we assess whether crowdsourced ratings of trust in news sources can effectively differentiate between more and less reliable sources. To do so, we ran a preregistered experiment (N = 1,010 from Amazon Mechanical Turk) in which individuals rated familiarity with, and trust in, 60 news sources from three categories: 1) Mainstream media outlets, 2) Websites that produce hyper-partisan coverage of actual facts, and 3) Websites that produce blatantly false content (“fake news”). Our results indicate that, despite substantial partisan bias, laypeople across the political spectrum rate mainstream media outlets as far more trustworthy than either hyper-partisan or fake news sources (all but 1 mainstream source, Salon, was rated as more trustworthy than every hyper-partisan or fake news source when equally weighting ratings of Democrats and Republicans). Critically, however, excluding ratings from participants who are not familiar with a given news source dramatically reduces the difference between mainstream media sources and hyper-partisan or fake news sites. For example, 30% of the mainstream media websites (Salon, the Guardian, Fox News, Politico, Huffington Post, and Newsweek) received lower trust scores than the most trusted fake news site (news4ktla.com) when excluding unfamiliar ratings. This suggests that rather than being initially agnostic about unfamiliar sources, people are initially skeptical – and thus a lack of familiarity is an important cue for untrustworthiness. Overall, our findings indicate that crowdsourcing media trustworthiness judgments is a promising approach for fighting misinformation and disinformation online, but that trustworthiness ratings from participants who are unfamiliar with a given source should not be ignored.

Keywords: fake news, news media, social media, media trust, misinformation

Suggested Citation

Pennycook, Gordon and Rand, David G., Crowdsourcing Judgments of News Source Quality (February 7, 2018). Available at SSRN: https://ssrn.com/abstract=3118471

Gordon Pennycook (Contact Author)

Yale University ( email )

New Haven, CT 06520
United States

David G. Rand

Yale University ( email )

New Haven, CT 06520
United States

HOME PAGE: http://www.DaveRand.org

Paper statistics

Downloads
881
rank
22,358
Abstract Views
4,113
PlumX