Beyond the Numbers: Statistical and Data Literacy, Domain Literacy and Supreme Court of Canada Data Analytics

Abstract

There is growing interest in using data analytics to assess the work of judges and courts in Canada. Generally, analytics outputs highlight patterns or trends in the work or behaviour of a court or a single judge. Existing literature on judicial analytics often focuses on what the tools can do – that is, the technical capabilities and/or predictive power of analytics technology. However, it is equally important to engage critically with the practical and normative dimensions of judicial analytics. Which observations produced by judicial analytics tools are statistically significant? What background information about the court and/or the law is necessary to appropriately interpret a given judicial analytics output? Which features of judicial decision-making can be empirically tracked in a meaningful way? How are those features best measured? What are the standards or benchmarks against which observed judicial behaviour should be assessed?

Failing to account for such questions can give rise to at least two kinds of harm. First, indiscriminately pumping into public discourse data about judges and courts can lead to “data overload”. The public may become overwhelmed by the amount of information available such that it becomes difficult to assess what information is truly significant for advancing one’s understanding of the work of judges and courts. Second, where judicial analytics outputs are not appropriately contextualized, there is risk of proliferating misleading data about courts and judges. This can undermine the public’s ability to properly understand and evaluate judicial work. It may even lead to calls for reforms based on partial or incorrect understandings of the “problem” supposedly in need of fixing.

In this article, we identify two related but distinct kinds of literacy required for meaningful and responsible engagement with judicial analytics outputs: (1) data and statistical literacy and (2) domain literacy. We explore these two kinds of literacy using data from and about the Supreme Court of Canada (SCC) as a case study. The SCC is especially amenable to analytics work because it has a comprehensive public dataset (i.e. all pleadings, written decisions and audio-video recordings of its oral hearings are published) and its work is often of special interest to the public given its stature as the highest court in Canada.

Keywords: Judges, courts, technology, analytics, judicial analytics, data science, quantitative research, Supreme Court of Canada