30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity

Abstract

The widespread adoption of learning management systems in educational institutions has yielded numerous benefits for teaching staff but also introduced the risk of unequal treatment towards students. We present an analysis of over 30 million Canvas grading records from a large public university, revealing a significant bias in sequential grading tasks. We find that assignments graded later in the sequence tend to (1) receive lower grades, (2) receive comments that are notably more negative and less polite, and (3) exhibit lower grading quality measured by post-grade complaints from students.

Furthermore, we show that the system design of Canvas, which pre-orders submissions by student surnames, transforms the sequential bias into a significant disadvantage for students with alphabetically lower-ranked surname initials. These students consistently receive lower grades, more negative and impolite comments, and raise more post-grade complaints as a result of their disadvantaged position in the grading sequence. This surname initial disparity is observed across a wide range of subjects, and is more prominent in social science and humanities as compared to engineering, science and medicine. The assignment-level surname disparity aggregates to a course-level surname disparity of students' GPA and can potentially lead to inequitable job opportunities. For platforms and education institutions, the system-induced surname grading disparity can be mitigated by randomizing student submissions in grading tasks. Education institutions should keep the workload of graders at a reasonable level to reduce fatigue and/or have multiple graders as a cross validation to enhance grading quality.

Keywords: EdTech, education operations, platform design, behavioral bias, service operations management, people-centric operations