Effects of LLM Use and Note-Taking On Reading Comprehension and Memory: A Randomised Experiment in Secondary Schools

48 Pages Posted: 3 Feb 2025

See all articles by Pia Kreijkes

Pia Kreijkes

University of Cambridge

Viktor Kewenig

University College London

Martina Kuvalja

University of Cambridge

Mina Lee

Microsoft Research

Sylvia Vitello

University of Cambridge

Jake M. Hofman

Microsoft Research, New York City

Abigail Sellen

Microsoft Research

Sean Rintel

Microsoft Research

Daniel G. Goldstein

Microsoft Corporation - Microsoft Research, New York City

David M. Rothschild

Microsoft Research

Lev Tankelevitch

Microsoft Research

Tim Oates

University of Cambridge

Date Written: January 13, 2025

Abstract

The rapid uptake of Generative AI, particularly large language models (LLMs), by students raises urgent questions about their effects on learning. We compared the impact of LLM use to that of traditional note-taking, or a combination of both, on secondary school students' reading comprehension and retention. We conducted a pre-registered, randomised controlled experiment with within-and between-participant design elements in schools. 405 students aged 14-15 studied two text passages and completed comprehension and retention tests three days later. Quantitative results demonstrated that both note-taking alone and combined with the LLM had significant positive effects on retention and comprehension compared to the LLM alone. Yet, most students preferred using the LLM over note-taking, and perceived it as more helpful. Qualitative results revealed that many students valued LLMs for making complex material more accessible and reducing cognitive load, while they appreciated note-taking for promoting deeper engagement and aiding memory. Additionally, we identified "archetypes" of prompting behaviour, offering insights into the different ways students interacted with the LLM. Overall, our findings suggest that, while note-taking promotes cognitive engagement and long-term comprehension and retention, LLMs may facilitate initial understanding and student interest. The study reveals the continued importance of traditional learning approaches, the benefits of combining AI use with traditional learning over using AI alone, and the AI skills that students need to maximise those benefits.

Keywords: Large language models, Learning, Memory, Reading comprehension, Note-taking

Suggested Citation

Kreijkes, Pia and Kewenig, Viktor and Kuvalja, Martina and Lee, Mina and Vitello, Sylvia and Hofman, Jake and Sellen, Abigail and Rintel, Sean and Goldstein, Daniel G. and Rothschild, David M. and Tankelevitch, Lev and Oates, Tim, Effects of LLM Use and Note-Taking On Reading Comprehension and Memory: A Randomised Experiment in Secondary Schools (January 13, 2025). Available at SSRN: https://ssrn.com/abstract=5095149 or http://dx.doi.org/10.2139/ssrn.5095149

Pia Kreijkes (Contact Author)

University of Cambridge ( email )

Shaftesbury Road
Cambridge, CB2 8EA
United Kingdom

Viktor Kewenig

University College London ( email )

Gower Street
London, WC1E 6BT
United Kingdom

Martina Kuvalja

University of Cambridge ( email )

Mina Lee

Microsoft Research ( email )

Sylvia Vitello

University of Cambridge ( email )

Jake Hofman

Microsoft Research, New York City ( email )

300 Lafayette St
New York, NY 10012
United States

Abigail Sellen

Microsoft Research ( email )

Sean Rintel

Microsoft Research ( email )

Daniel G. Goldstein

Microsoft Corporation - Microsoft Research, New York City ( email )

300 Lafayette St
New York, NY NY 10012
United States

David M. Rothschild

Microsoft Research ( email )

New York City, NY NY 10011
United States

Lev Tankelevitch

Microsoft Research ( email )

Tim Oates

University of Cambridge ( email )

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
85
Abstract Views
461
Rank
612,332
PlumX Metrics