Experimental Evidence of the Effects of Large Language Models versus Web Search on Depth of Learning
20 Pages Posted: 21 Jan 2025
Date Written: January 20, 2025
Abstract
The effects of using large language models (LLMs) versus traditional web search on depth of learning are explored. Results from four online and laboratory experiments (N = 4,591) lend support for the predictions that when individuals learn about a topic from LLMs, they tend to develop shallower knowledge than when they learn through standard web search, even when the core information in the results is the same. This shallower knowledge accrues from an inherent feature of LLMs—the presentation of results as syntheses of information rather than individual search links—which makes learning more passive than in standard web search, where users actively discover and synthesize information sources themselves. In turn, when subsequently forming advice on the topic based on what they learned, those who learned from LLM syntheses (vs. standard search results) feel less invested in forming their advice and, more importantly, create advice that is sparser, less original—and ultimately less likely to be adopted by recipients. Implications of the findings for recent research on the benefits and risks of LLMs are discussed.
Suggested Citation: Suggested Citation