Experimental Evidence of the Effects of Large Language Models versus Web Search on Depth of Learning

20 Pages Posted: 21 Jan 2025

See all articles by Shiri Melumad

Shiri Melumad

University of Pennsylvania - The Wharton School

Jin Ho Yun

University of Pennsylvania - The Wharton School

Date Written: January 20, 2025

Abstract

The effects of using large language models (LLMs) versus traditional web search on depth of learning are explored. Results from four online and laboratory experiments (N = 4,591) lend support for the predictions that when individuals learn about a topic from LLMs, they tend to develop shallower knowledge than when they learn through standard web search, even when the core information in the results is the same. This shallower knowledge accrues from an inherent feature of LLMs—the presentation of results as syntheses of information rather than individual search links—which makes learning more passive than in standard web search, where users actively discover and synthesize information sources themselves. In turn, when subsequently forming advice on the topic based on what they learned, those who learned from LLM syntheses (vs. standard search results) feel less invested in forming their advice and, more importantly, create advice that is sparser, less original—and ultimately less likely to be adopted by recipients. Implications of the findings for recent research on the benefits and risks of LLMs are discussed.

Suggested Citation

Melumad, Shiri and Yun, Jin Ho, Experimental Evidence of the Effects of Large Language Models versus Web Search on Depth of Learning (January 20, 2025). The Wharton School Research Paper, Available at SSRN: https://ssrn.com/abstract=5104064 or http://dx.doi.org/10.2139/ssrn.5104064

Shiri Melumad (Contact Author)

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Jin Ho Yun

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
63
Abstract Views
204
Rank
727,462
PlumX Metrics