More than a Feeling: Accuracy and Application of Sentiment Analysis

Hartmann, J., Heitmann, M., Siebert, C., & Schamp, C. (2022). More than a Feeling: Accuracy and Application of Sentiment Analysis. International Journal of Research in Marketing, Forthcoming.

32 Pages Posted: 5 Dec 2019 Last revised: 30 Sep 2022

See all articles by Jochen Hartmann

Jochen Hartmann

TUM School of Management,Technical University of Munich

Mark Heitmann

University of Hamburg

Christian Siebert

University of Hamburg

Christina Schamp

University of Mannheim

Date Written: May 12, 2022

Abstract

Sentiment is fundamental to human communication. Countless marketing applications mine opinions from social media communication, news articles, customer feedback, or corporate communication. Various sentiment analysis methods are available and new ones have recently been proposed. Lexicons can relate individual words and expressions to sentiment scores. In contrast, machine learning methods are more complex to interpret, but promise higher accuracy, i.e., fewer false classifications. We propose an empirical framework and quantify these trade-offs for different types of research questions, data characteristics, and analytical resources to enable informed method decisions contingent on the application context. Based on a meta-analysis of 272 datasets and 12 million sentiment-labeled text documents, we find that the recently proposed transfer learning models indeed perform best, but can perform worse than popular leaderboard benchmarks suggest. We quantify the accuracy-interpretability trade-off, showing that, compared to widely established lexicons, transfer learning models on average classify more than 20 percentage points more documents correctly. To form realistic performance expectations, additional context variables, most importantly the desired number of sentiment classes and the text length, should be taken into account. We provide a pre-trained sentiment analysis model (called SiEBERT) with open-source scripts that can be applied as easily as an off-the-shelf lexicon.

Keywords: Sentiment Analysis; Meta-Analysis; Natural Language Processing; Machine Learning; Transfer Learning; Deep Contextual Language Models; Text Mining

Suggested Citation

Hartmann, Jochen and Heitmann, Mark and Siebert, Christian and Schamp, Christina, More than a Feeling: Accuracy and Application of Sentiment Analysis (May 12, 2022). Hartmann, J., Heitmann, M., Siebert, C., & Schamp, C. (2022). More than a Feeling: Accuracy and Application of Sentiment Analysis. International Journal of Research in Marketing, Forthcoming., Available at SSRN: https://ssrn.com/abstract=3489963 or http://dx.doi.org/10.2139/ssrn.3489963

Jochen Hartmann (Contact Author)

TUM School of Management,Technical University of Munich ( email )

Arcisstrasse 21
Munchen, 80333
Germany

Mark Heitmann

University of Hamburg ( email )

Allende-Platz 1
Hamburg, 20146
Germany

Christian Siebert

University of Hamburg ( email )

Allende-Platz 1
Hamburg, 20146
Germany

Christina Schamp

University of Mannheim ( email )

L 7, 3-5
Mannheim, 68161
Germany

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
2,206
Abstract Views
7,597
Rank
12,782
PlumX Metrics