Crowdsourcing Judgments of News Source Quality

26 Pages Posted: 17 Feb 2018 Last revised: 21 Apr 2018

See all articles by Gordon Pennycook

Gordon Pennycook

University of Regina

David G. Rand

Massachusetts Institute of Technology (MIT)

Date Written: April 13, 2018


The spread of misinformation and disinformation, especially on social media, is a major societal challenge. Here, we assess whether crowdsourced ratings of trust in news sources can effectively differentiate between more and less reliable sources. To do so, we ran a preregistered experiment (N = 1,010 from Amazon’s Mechanical Turk) in which individuals rated familiarity with, and trust in, 60 news sources from three categories: 1) Mainstream media outlets, 2) Websites that produce hyper-partisan coverage, and 3) Websites that produce blatantly false content (“fake news”). Our results indicate that, despite substantial partisan differences, laypeople across the political spectrum rate mainstream media outlets as far more trustworthy than either hyper-partisan or fake news sources (every mainstream source was rated as more trustworthy than every hyper-partisan or fake news source when equally weighting ratings of Democrats and Republicans). Critically, however, excluding ratings from participants who were not familiar with a given news source dramatically reduced the difference between mainstream media sources and hyper-partisan or fake news sites. For example, 26% of the mainstream media websites (the Guardian, Fox News, Politico, Huffington Post, and Newsweek) received lower trust scores than the most trusted fake news site ( when excluding unfamiliar ratings. This suggests that rather than being initially agnostic about unfamiliar sources, people are initially skeptical – and thus a lack of familiarity is an important cue for untrustworthiness. Our findings indicate that crowdsourcing media trustworthiness judgments is a promising approach for fighting misinformation and disinformation online, but that trustworthiness ratings from participants who are unfamiliar with a given source should not be ignored.

Keywords: fake news, news media, social media, media trust, misinformation

Suggested Citation

Pennycook, Gordon and Rand, David G., Crowdsourcing Judgments of News Source Quality (April 13, 2018). Available at SSRN: or

Gordon Pennycook (Contact Author)

University of Regina ( email )

3737 Wascana Parkway
Regina, Saskatchewan S4S OA2 S4S 0A1

David G. Rand

Massachusetts Institute of Technology (MIT) ( email )

77 Massachusetts Avenue
50 Memorial Drive
Cambridge, MA 02139-4307
United States


Register to save articles to
your library


Paper statistics

Abstract Views
PlumX Metrics