The Rise of Generative AI for Tax Research

Abstract

ChatGPT has exploded into the popular consciousness. Generative artificial intelligence, in the form of natural language chatbots, is newly poised to significantly shift how tax professionals conduct research. These emerging advanced AI systems can analyze vast amounts of text and generate coherent, contextually relevant responses.

As various industries continue to adopt AI-powered tools, lawyers need to understand and adapt to these new technologies. As Jason Chen has written in Tax Notes, “It is only by incorporating these types of technological advances into daily business practices that modern tax professionals can gain valuable advantages over others in the increasingly competitive world of taxation.”

In this article, we will consider the benefits of using generative AI, examine the concerns, and discuss ways that it can be improved. To illustrate these considerations, we submitted a series of tax law questions to three different generative AI models: ChatGPT (based on GPT-3.5), ChatGPT Plus (based on GPT-4), and Ask Blue J. The questions were selected to represent a variety of tax law areas and to present the types of research questions that a typical tax law practitioner might encounter in their day-to-day practice.

For this experiment we posed identical questions to ChatGPT and Ask Blue J and recorded their responses. We tested the questions against the GPT-3.5 and GPT-4 versions of ChatGPT. We primed each conversation with ChatGPT using the following instruction: “I want you to answer all questions as they apply in the context of U.S. federal income tax law. Please provide IRS resources along with your answers.” This additional instruction was provided to ChatGPT only because Ask Blue J is already fine-tuned to focus on tax and to provide sources with its responses. Providing the priming instruction to the GPT models ensures a fair comparison. Other than this introductory prompt, our questions to each chatbot were the same. The results highlight the importance of developers and users engaging with this technology thoughtfully when harnessing it for tax research purposes.

Keywords: ChatGPT, generative AI, artificial intelligence, tax research

JEL Classification: H0