More than a Feeling: Accuracy and Application of Sentiment Analysis

32 Pages Posted: 5 Dec 2019 Last revised: 12 May 2022

See all articles by Jochen Hartmann

Jochen Hartmann

University of Groningen - Faculty of Economics and Business

Mark Heitmann

University of Hamburg

Christian Siebert

University of Hamburg

Christina Schamp

University of Mannheim

Date Written: May 12, 2022

Abstract

Sentiment is fundamental to human communication. Countless marketing applications mine opinions from social media communication, news articles, customer feedback, or corporate communication. Various sentiment analysis methods are available and new ones have recently been proposed. Lexicons can relate individual words and expressions to sentiment scores. In contrast, machine learning methods are more complex to interpret, but promise higher accuracy, i.e., fewer false classifications. We propose an empirical framework and quantify these trade-offs for different types of research questions, data characteristics, and analytical resources to enable informed method decisions contingent on the application context. Based on a meta-analysis of 272 datasets and 12 million sentiment-labeled text documents, we find that the recently proposed transfer learning models indeed perform best, but can perform worse than popular leaderboard benchmarks suggest. We quantify the accuracy-interpretability trade-off, showing that, compared to widely established lexicons, transfer learning models on average classify more than 20 percentage points more documents correctly. To form realistic performance expectations, additional context variables, most importantly the desired number of sentiment classes and the text length, should be taken into account. We provide a pre-trained sentiment analysis model (called SiEBERT) with open-source scripts that can be applied as easily as an off-the-shelf lexicon.

Keywords: Sentiment Analysis; Meta-Analysis; Natural Language Processing; Machine Learning; Transfer Learning; Deep Contextual Language Models; Text Mining

Suggested Citation

Hartmann, Jochen and Heitmann, Mark and Siebert, Christian and Schamp, Christina, More than a Feeling: Accuracy and Application of Sentiment Analysis (May 12, 2022). Available at SSRN: https://ssrn.com/abstract=3489963 or http://dx.doi.org/10.2139/ssrn.3489963

Jochen Hartmann (Contact Author)

University of Groningen - Faculty of Economics and Business ( email )

Postbus 72
9700 AB Groningen
Netherlands

Mark Heitmann

University of Hamburg ( email )

Allende-Platz 1
Hamburg, 20146
Germany

Christian Siebert

University of Hamburg ( email )

Allende-Platz 1
Hamburg, 20146
Germany

Christina Schamp

University of Mannheim ( email )

L 7, 3-5
Mannheim, 68161
Germany

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
1,731
Abstract Views
5,672
rank
14,080
PlumX Metrics