30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity

47 Pages Posted: 17 Oct 2023 Last revised: 10 Mar 2024

See all articles by Zhihan (Helen) Wang

Zhihan (Helen) Wang

University of Michigan, Stephen M. Ross School of Business

Jiaxin Pei

University of Michigan at Ann Arbor - School of Information

Jun Li

University of Michigan, Stephen M. Ross School of Business

Date Written: October 16, 2023

Abstract

The widespread adoption of learning management systems in educational institutions has yielded numerous benefits for teaching staff but also introduced the risk of unequal treatment towards students. We present an analysis of over 30 million Canvas grading records from a large public university, revealing a significant bias in sequential grading tasks. We find that assignments graded later in the sequence tend to (1) receive lower grades, (2) receive comments that are notably more negative and less polite, and (3) exhibit lower grading quality measured by post-grade complaints from students.

Furthermore, we show that the system design of Canvas, which pre-orders submissions by student surnames, transforms the sequential bias into a significant disadvantage for students with alphabetically lower-ranked surname initials. These students consistently receive lower grades, more negative and impolite comments, and raise more post-grade complaints as a result of their disadvantaged position in the grading sequence. This surname initial disparity is observed across a wide range of subjects, and is more prominent in social science and humanities as compared to engineering, science and medicine. The assignment-level surname disparity aggregates to a course-level surname disparity of students' GPA and can potentially lead to inequitable job opportunities. For platforms and education institutions, the system-induced surname grading disparity can be mitigated by randomizing student submissions in grading tasks. Education institutions should keep the workload of graders at a reasonable level to reduce fatigue and/or have multiple graders as a cross validation to enhance grading quality.

Keywords: EdTech, education operations, platform design, behavioral bias, service operations management, people-centric operations

Suggested Citation

Wang, Zhihan (Helen) and Pei, Jiaxin and Li, Jun, 30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity (October 16, 2023). Available at SSRN: https://ssrn.com/abstract=4603146

Zhihan (Helen) Wang (Contact Author)

University of Michigan, Stephen M. Ross School of Business ( email )

701 Tappan Street
Ann Arbor, MI MI 48109
United States

Jiaxin Pei

University of Michigan at Ann Arbor - School of Information ( email )

304 West Hall
550 East University
Ann Arbor, MI 48109-1092
United States

Jun Li

University of Michigan, Stephen M. Ross School of Business ( email )

701 Tappan Street
Ann Arbor, MI MI 48109
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
1,296
Abstract Views
7,096
Rank
30,433
PlumX Metrics