ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience

31 Pages Posted: 5 Jul 2023

See all articles by Ruiyun Rayna Xu

Ruiyun Rayna Xu

Farmer School of Business, Miami University

Yue (Katherine) Feng

Hong Kong Polytechnic University - Department of Management and Marketing

Hailiang Chen

HKU Business School, The University of Hong Kong

Date Written: July 2023

Abstract

The advent of ChatGPT, a large language model-powered chatbot, has prompted questions about its potential implications for traditional search engines. In this study, we investigate the differences in user behavior when employing search engines and chatbot tools for information-seeking tasks. We carry out a randomized online experiment, dividing participants into two groups: one using a ChatGPT-like tool and the other using a Google Search-like tool. Our findings reveal that the ChatGPT group consistently spends less time on all tasks, with no significant difference in overall task performance between the groups. Notably, ChatGPT levels user search performance across different education levels and excels in answering straightforward questions and providing general solutions but falls short in fact-checking tasks. Users perceive ChatGPT’s responses as having higher information quality compared to Google Search, despite displaying a similar level of trust in both tools. Furthermore, participants using ChatGPT report significantly better user experiences in terms of usefulness, enjoyment, and satisfaction, while perceived ease of use remains comparable between the two tools. However, ChatGPT may also lead to overreliance and generate or replicate misinformation, yielding inconsistent results. Our study offers valuable insights for search engine management and highlights opportunities for integrating chatbot technologies into search engine designs.

Keywords: ChatGPT, generative AI, Google, search engines, chatbot, online experiment

Suggested Citation

Xu, Ruiyun and Feng, Yue (Katherine) and Chen, Hailiang, ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience (July 2023). Available at SSRN: https://ssrn.com/abstract=4498671 or http://dx.doi.org/10.2139/ssrn.4498671

Ruiyun Xu

Farmer School of Business, Miami University

Department of Information Systems and Analytics
3095 Farmer School of Business, 800 E. High St.
Oxford, OH 45056
United States

Yue (Katherine) Feng

Hong Kong Polytechnic University - Department of Management and Marketing ( email )

Li Ka Shing Tower
The Hong Kong Polytechnic University
Hung Hom, Kowloon, Hong Kong
China

Hailiang Chen (Contact Author)

HKU Business School, The University of Hong Kong ( email )

K. K. Leung Building
University of Hong Kong
Hong Kong

HOME PAGE: http://www.fbe.hku.hk/people/academic/hailiang-chen

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
72
Abstract Views
275
Rank
532,600
PlumX Metrics