ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience
31 Pages Posted: 5 Jul 2023 Last revised: 11 Dec 2023
Date Written: July 1, 2023
Abstract
The emergence of ChatGPT, a large language model-powered chatbot, has the potential to revolutionize the way users search for information by shifting from keyword-based queries to natural language interactions. This study investigates user behavior differences when using search engines and chatbot tools for various information retrieval tasks and explores the combined use of both tools. We conduct two randomized online experiments to achieve this goal. Our findings reveal that ChatGPT users consistently spend less time on all tasks, with no significant difference in overall task performance between the ChatGPT and Google Search groups. Notably, ChatGPT excels at addressing common information look-ups and providing summarized solutions, but struggles with fact-checking tasks which require users to take extensive exploration to resolve conflicting information. The impact of cognitive fit between task requirements and search tools’ design features on performance is task-dependent. Our results also indicate that a strong belief in search result accuracy can diminish the effectiveness of using ChatGPT. When both tools are accessible, the outcomes lie between the results of using each tool individually. Participants strategically employ both tools, often selecting the most suitable one for a specific task. Our study offers valuable insights for future development and integration of chatbot tools with search engines, suggesting potential improvements to enhance user experience and information-seeking efficiency.
Keywords: generative AI, large language model, search engines, chatbot, online experiment
Suggested Citation: Suggested Citation