Is ChatGPT Racially Biased? The Case of Evaluating Student Writing

Abstract

We present experimental proof of racial bias in ChatGPT's evaluation of student writing. By manipulating racial descriptors in prompts, we assessed differences in scores given by two ChatGPT models. Our findings indicate that descriptions of students as Black or White lead to significantly higher scores compared to race-neutral or Hispanic descriptors. This suggests that ChatGPT's outputs are influenced by racial information, which raises concerns about its application in educational settings. The study highlights the need for transparent and bias-tested AI tools in education to prevent the perpetuation of existing inequities and suggests implications for educators, administrators, and policy makers.

Keywords: Generative AI, Large Language Models, Racial Bias