A Taxonomy for Understanding and Identifying Uncertainty in AI-Generated Responses

Abstract

People search for information to meet their personal, business, and civic goals and increasingly do so with AI-based tools. We present a taxonomy of uncertainties in responses generated by Large Language Models (LLMs). It identifies three main types of uncertainty: outcome variability, model uncertainty, and prompt ambiguity. Outcome variability encompasses the unpredictability in data, world as aleatory uncertainty. Model uncertainty arises from insufficient knowledge available to the model, termed epistemic uncertainty. Prompt ambiguity involves unclear user inputs leading to multiple valid interpretations. The study explores detection methods for these uncertainties, employing strategies such as token probability analysis and temperature sampling. This taxonomy aims to enable researchers and regulators to identify, track, and remediate uncertainty from LLM-based tools that may bias or otherwise impair decision making.

Keywords: Taxonomy, LLM, Uncertainty, Error

JEL Classification: D81, D82, D83, D84