Crowdsourcing Judgments of News Source Quality
26 Pages Posted: 17 Feb 2018 Last revised: 21 Apr 2018
Date Written: April 13, 2018
The spread of misinformation and disinformation, especially on social media, is a major societal challenge. Here, we assess whether crowdsourced ratings of trust in news sources can effectively differentiate between more and less reliable sources. To do so, we ran a preregistered experiment (N = 1,010 from Amazon’s Mechanical Turk) in which individuals rated familiarity with, and trust in, 60 news sources from three categories: 1) Mainstream media outlets, 2) Websites that produce hyper-partisan coverage, and 3) Websites that produce blatantly false content (“fake news”). Our results indicate that, despite substantial partisan differences, laypeople across the political spectrum rate mainstream media outlets as far more trustworthy than either hyper-partisan or fake news sources (every mainstream source was rated as more trustworthy than every hyper-partisan or fake news source when equally weighting ratings of Democrats and Republicans). Critically, however, excluding ratings from participants who were not familiar with a given news source dramatically reduced the difference between mainstream media sources and hyper-partisan or fake news sites. For example, 26% of the mainstream media websites (the Guardian, Fox News, Politico, Huffington Post, and Newsweek) received lower trust scores than the most trusted fake news site (news4ktla.com) when excluding unfamiliar ratings. This suggests that rather than being initially agnostic about unfamiliar sources, people are initially skeptical – and thus a lack of familiarity is an important cue for untrustworthiness. Our findings indicate that crowdsourcing media trustworthiness judgments is a promising approach for fighting misinformation and disinformation online, but that trustworthiness ratings from participants who are unfamiliar with a given source should not be ignored.
Keywords: fake news, news media, social media, media trust, misinformation
Suggested Citation: Suggested Citation