Uncertainty in Sentiment Analysis with LLMs using QCM (Quantiles of Correlation Matrices) - Distance

Abstract

In this paper, we investigate how to better understand uncertainty in sentiment score when
using Natural Language Processing (NLP) and Large Language Models (LLMs) to derive from
a sentiment score from a text. After explaining why uncertainty on sentiment score is challenging,
especially in the case of news sentiment analysis, we present multiple prompts and an
on-going research of a novel way to identify uncertainty in sentiment analysis. Our method
computes the correlation between the cumulative sentiment score over different time windows
with important influenced factors and validate that a similar proportion of patterns exists,
using a measure based on the distance of Quantiles of Correlation Matrices, called QCMDistance.
We apply this approach to see how sentiments in financial news are connected with
changes in stock markets and to assess that a common pattern exists with the produced sentiment
score, establishing a moderate uncertainty of the resulted sentiment score.

Keywords: LLMs, Quantile distance

JEL Classification: G11, G13