AI Meets the Classroom: When Does ChatGPT Harm Learning?

40 Pages Posted: 1 Oct 2024

See all articles by Matthias Lehmann

Matthias Lehmann

University of Cologne

Philipp B. Cornelius

Erasmus University Rotterdam (EUR) - Department of Technology and Operations Management; Erasmus University Rotterdam (EUR) - Rotterdam School of Management (RSM)

Fabian J. Sting

University of Cologne; Erasmus University - Rotterdam School of Management

Date Written: August 29, 2024

Abstract

In this paper, we study how generative AI and specifically large language models (LLMs) impact learning in coding classes. We show across three studies that LLM usage can have positive and negative effects on learning outcomes. Using observational data from universitylevel programming courses, we establish such effects in the field. We replicate these findings in subsequent experimental studies, which closely resemble typical learning scenarios, to show causality. We find evidence for two contrasting mechanisms that determine the overall effect of LLM usage on learning. Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to solve practice exercises for them and thus do not invest sufficient own mental effort. Those who never used LLMs before are particularly prone to such adverse behavior. Students without prior domain knowledge gain more from having access to LLMs. Finally, we show that the self-perceived benefits of using LLMs for learning exceed the actual benefits, potentially resulting in an overestimation of one's own abilities. Overall, our findings show promising potential of LLMs as learning support, however also that students have to be very cautious of possible pitfalls.

Keywords: Generative AI, Artificial Intelligence, Learning, Education, ChatGPT, Coding, Learning-by-Doing

Suggested Citation

Lehmann, Matthias and Cornelius, Philipp B. and Sting, Fabian J., AI Meets the Classroom: When Does ChatGPT Harm Learning? (August 29, 2024). Available at SSRN: https://ssrn.com/abstract=4941259 or http://dx.doi.org/10.2139/ssrn.4941259

Matthias Lehmann (Contact Author)

University of Cologne ( email )

Philipp B. Cornelius

Erasmus University Rotterdam (EUR) - Department of Technology and Operations Management ( email )

RSM Erasmus University
PO Box 1738
3000 DR Rotterdam
Netherlands

Erasmus University Rotterdam (EUR) - Rotterdam School of Management (RSM) ( email )

P.O. Box 1738
Room T08-21
3000 DR Rotterdam, 3000 DR
Netherlands

Fabian J. Sting

University of Cologne ( email )

Albertus Magnus Platz
Cologne, NRW 50923
Germany

HOME PAGE: http://https://scm.uni-koeln.de/en/sting-f

Erasmus University - Rotterdam School of Management ( email )

Erasmus University
PO Box 1738
3000 DR Rotterdam, Zuid-Holland
Netherlands

HOME PAGE: http://www.rsm.nl/fsting

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
319
Abstract Views
1,356
Rank
197,345
PlumX Metrics