LLMs Outperform Outsourced Human Coders on Complex Textual Analysis

33 Pages Posted: 23 Dec 2024 Last revised: 24 Dec 2024

See all articles by Vicente J. Bermejo

Vicente J. Bermejo

ESADE Business School

Andres Gago

Universidad Torcuato Di Tella

Ramiro H. Gálvez

Universidad Torcuato Di Tella

Nicolás Harari

Boston University

Date Written: November 13, 2024

Abstract

This paper evaluates the effectiveness of large language models (LLMs) in extracting complex information from textual sources. We compare the performance of various LLMs with that of outsourced human coders on five natural language processing tasks, ranging from named entity recognition to identifying nuanced political criticism in news articles. Using a corpus of Spanish news articles, we find that LLMs consistently outperform outsourced human coders, especially in tasks requiring deep contextual understanding. These findings suggest that current LLM technology provides researchers without programming expertise a cost-effective alternative for sophisticated text analysis.

Keywords: Large Language Models, Text Analysis, Human Annotation, Natural Language Processing, News, Media

Suggested Citation

Bermejo, Vicente J. and Gago, Andres and Gálvez, Ramiro H. and Harari, Nicolás, LLMs Outperform Outsourced Human Coders on Complex Textual Analysis (November 13, 2024). Available at SSRN: https://ssrn.com/abstract=5020034 or http://dx.doi.org/10.2139/ssrn.5020034

Vicente J. Bermejo (Contact Author)

ESADE Business School ( email )

Av. de Pedralbes, 60-62
Barcelona, 08034
Spain

Andres Gago

Universidad Torcuato Di Tella ( email )

Saenz Valiente 1010
C1428BIJ Buenos Aires
Argentina
C1428BCW (Fax)

Ramiro H. Gálvez

Universidad Torcuato Di Tella ( email )

Minones 2159
C1428ATG Buenos Aires, 1428
Argentina

Nicolás Harari

Boston University ( email )

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
150
Abstract Views
496
Rank
401,164
PlumX Metrics