Generative AI Can Harm Learning

59 Pages Posted: 18 Jul 2024

See all articles by Hamsa Bastani

Hamsa Bastani

University of Pennsylvania - The Wharton School

Osbert Bastani

University of Pennsylvania - Department of Computer and Information Science

Alp Sungu

University of Pennsylvania - The Wharton School

Haosen Ge

University of Pennsylvania - The Wharton School

Özge Kabakcı

Budapest British International School

Rei Mariman

Independent; Independent

Date Written: July 15, 2024

Abstract

Generative artificial intelligence (AI) is poised to revolutionize how humans work, and has already demonstrated promise in significantly improving human productivity. However, a key remaining question is how generative AI affects learning, namely, how humans acquire new skills as they perform tasks. This kind of skill learning is critical to long-term productivity gains, especially in domains where generative AI is fallible and human experts must check its outputs. We study the impact of generative AI, specifically OpenAI's GPT-4, on human learning in the context of math classes at a high school. In a field experiment involving nearly a thousand students, we have deployed and evaluated two GPT based tutors, one that mimics a standard ChatGPT interface (called GPT Base) and one with prompts designed to safeguard learning (called GPT Tutor). These tutors comprise about 15% of the curriculum in each of three grades. Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base). That is, access to GPT-4 can harm educational outcomes. These negative learning effects are largely mitigated by the safeguards included in GPT Tutor. Our results suggest that students attempt to use GPT-4 as a "crutch" during practice problem sessions, and when successful, perform worse on their own. Thus, to maintain long-term productivity, we must be cautious when deploying generative AI to ensure humans continue to learn critical skills.
* HB, OB, and AS contributed equally

Keywords: Generative AI, Human Capital Development, Education, Human-AI Collaboration, Large Language Models

Suggested Citation

Bastani, Hamsa and Bastani, Osbert and Sungu, Alp and Ge, Haosen and Kabakcı, Özge and Mariman, Rei, Generative AI Can Harm Learning (July 15, 2024). The Wharton School Research Paper, Available at SSRN: https://ssrn.com/abstract=4895486 or http://dx.doi.org/10.2139/ssrn.4895486

Hamsa Bastani

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Osbert Bastani

University of Pennsylvania - Department of Computer and Information Science ( email )

3330 Walnut Street
Philadelphia, PA 19104
United States

Alp Sungu (Contact Author)

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Haosen Ge

University of Pennsylvania - The Wharton School ( email )

Özge Kabakcı

Budapest British International School ( email )

Rei Mariman

Independent ( email )

Independent ( email )

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
13,861
Abstract Views
45,571
Rank
640
PlumX Metrics