ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience

31 Pages Posted: 5 Jul 2023 Last revised: 11 Dec 2023

See all articles by Ruiyun Rayna Xu

Ruiyun Rayna Xu

Farmer School of Business, Miami University

Yue (Katherine) Feng

Hong Kong Polytechnic University - Department of Management and Marketing

Hailiang Chen

HKU Business School, The University of Hong Kong

Date Written: July 1, 2023

Abstract

The emergence of ChatGPT, a large language model-powered chatbot, has the potential to revolutionize the way users search for information by shifting from keyword-based queries to natural language interactions. This study investigates user behavior differences when using search engines and chatbot tools for various information retrieval tasks and explores the combined use of both tools. We conduct two randomized online experiments to achieve this goal. Our findings reveal that ChatGPT users consistently spend less time on all tasks, with no significant difference in overall task performance between the ChatGPT and Google Search groups. Notably, ChatGPT excels at addressing common information look-ups and providing summarized solutions, but struggles with fact-checking tasks which require users to take extensive exploration to resolve conflicting information. The impact of cognitive fit between task requirements and search tools’ design features on performance is task-dependent. Our results also indicate that a strong belief in search result accuracy can diminish the effectiveness of using ChatGPT. When both tools are accessible, the outcomes lie between the results of using each tool individually. Participants strategically employ both tools, often selecting the most suitable one for a specific task. Our study offers valuable insights for future development and integration of chatbot tools with search engines, suggesting potential improvements to enhance user experience and information-seeking efficiency.

Keywords: generative AI, large language model, search engines, chatbot, online experiment

Suggested Citation

Xu, Ruiyun and Feng, Yue (Katherine) and Chen, Hailiang, ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience (July 1, 2023). Available at SSRN: https://ssrn.com/abstract=4498671 or http://dx.doi.org/10.2139/ssrn.4498671

Ruiyun Xu

Farmer School of Business, Miami University

Department of Information Systems and Analytics
3095 Farmer School of Business, 800 E. High St.
Oxford, OH 45056
United States

Yue (Katherine) Feng

Hong Kong Polytechnic University - Department of Management and Marketing ( email )

Li Ka Shing Tower
The Hong Kong Polytechnic University
Hung Hom, Kowloon, Hong Kong
China

Hailiang Chen (Contact Author)

HKU Business School, The University of Hong Kong ( email )

Hong Kong
China

HOME PAGE: http://www.hkubs.hku.hk/people/hailiang-chen

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
600
Abstract Views
2,506
Rank
93,981
PlumX Metrics