The Use of LLMs to Annotate Data in Management Research: Guidelines and Warnings

Abstract

The emergence of large language models (LLMs) has opened new avenues for integrating artificial intelligence into research, particularly for data annotation and text classification. However, the benefits and risks of using LLMs in research remain poorly understood, such that researchers lack guidance on how best to implement this tool. We address this gap by developing a methodological framework for implementing LLMs in management research, providing structured guidance on key implementation decisions and best practices. We illustrate the implementation of this framework through an empirical application classifying sustainability claims in crowdfunding projects to assess the performance effects of these claims. We demonstrate that while LLMs can match or exceed traditional methods' performance at lower cost, variations in prompt design can significantly affect results and downstream analyses. We thus develop procedures for sensitivity analysis and provide documentation to help researchers implement these robustness checks while maintaining methodological integrity.

Keywords: Artificial Intelligence, Research Methods, NLP, Classification, Crowdfunding