Executives vs. Chatbots: Unmasking Insights through Human-AI Differences in Earnings Conference Q&A

Abstract

A significant portion of information shared in earnings calls is conveyed through verbal communication by corporate managers. However, quantifying the extent of new information provided by managers poses challenges due to the unstructured nature of human language and the difficulty in gauging the market’s existing knowledge. In this study, we introduce a novel measure of information content (Human-AI Differences, HAID) by exploiting the discrepancy between answers to questions at earnings calls provided by corporate executives and those given by several context-preserving Large Language Models (LLM) such as ChatGPT, Google Bard, and an open source LLM. HAID strongly predicts stock liquidity, abnormal returns, number of analysts’ forecast revisions, analyst forecast accuracy following these calls, and propensity of managers to provide management guidance, consistent with HAID capturing new information conveyed by managers. Overall, our results highlight the importance of using LLM as a tool to help investors unveil the veiled – penetrating the information layers and unearthing hidden insights.

Keywords: ChatGPT; Bard; Large Language Model; AI; Conference Call; Chatbot; Information Content

JEL Classification: C45, C88, D80, G3; G11, G12, G14, M41