The Use of LLMs to Annotate Data in Management Research: Foundational Guidelines and Warnings

Abstract

The emergence of large language models (LLMs) has opened new avenues for integrating artificial intelligence into research, particularly for data annotation and text classification. However, the benefits and risks of using LLMs in research remain poorly understood, such that researchers lack guidance on how best to implement this tool. We address this gap by developing a foundational framework for implementing LLMs for annotation in management research, providing structured guidance on key implementation decisions and best practices. We illustrate the implementation of this framework through an empirical application: classifying sustainability claims in crowdfunding projects to assess the performance relationships of these claims. We demonstrate that while LLMs can match or exceed traditional methods' performance at lower cost, variations in prompt design can significantly affect results and downstream analyses. We thus develop procedures for sensitivity analysis and provide documentation to help researchers implement these robustness checks while maintaining methodological integrity.

Keywords: Artificial Intelligence, Research Methods, NLP, Classification, Crowdfunding