Signals of Public Opinion in Online Communication: A Comparison of Methods and Data Sources

The Annals of the American Academy of Political and Social Science, Forthcoming

23 Pages Posted: 2 Feb 2015  

Sandra Gonzalez-Bailon

University of Pennsylvania - Annenberg School for Communication

Georgios Paltoglou

University of Wolverhampton

Date Written: February 1, 2015

Abstract

This study offers a systematic comparison of automated content analysis tools by assessing their ability to correctly identify affective tone (e.g., positive vs. negative) in different data contexts and social media environments. Our comparisons assess the reliability and validity of publicly available, off-the-shelf classifiers. We use datasets from a range of online sources that vary in the diversity and formality of the language used, and we apply different classifiers to extract information about the affective tone in these datasets. We first measure agreement (reliability test) and then compare their classifications with the benchmark of human coding (validity test). Our analyses show that validity and reliability vary with the formality and diversity of the text; we also show that ready-to-use methods leave much space for improvement in domain-specific content and that a machine-learning approach offers more accurate predictions.

Keywords: content analysis; text mining; sentiment analysis; information diversity; lexicon-based methods; machine learning

Suggested Citation

Gonzalez-Bailon, Sandra and Paltoglou, Georgios, Signals of Public Opinion in Online Communication: A Comparison of Methods and Data Sources (February 1, 2015). The Annals of the American Academy of Political and Social Science, Forthcoming. Available at SSRN: https://ssrn.com/abstract=2558788

Sandra Gonzalez-Bailon (Contact Author)

University of Pennsylvania - Annenberg School for Communication ( email )

Philadelphia, PA
United States

HOME PAGE: http://www.asc.upenn.edu/sgonzalezbailon/

Georgios Paltoglou

University of Wolverhampton ( email )

Wulfruna Street
Wolverhampton, West Midlands WV1 1SB
United States

Paper statistics

Downloads
89
Rank
239,598
Abstract Views
510