Good AI Legal Help, Bad AI Legal Help: Establishing quality standards for responses to people’s legal problem stories

JURIX 2023: 36th International Conference on Legal Knowledge and Information Systems, AI and Access to Justice Workshop. December 2023.

12 Pages Posted: 20 Jan 2024

See all articles by Margaret Hagan

Margaret Hagan

Stanford Legal Design Lab; Stanford Law School

Date Written: December 18, 2023

Abstract

Much has been made of generative AI models’ ability to perform legal tasks or pass legal exams, but a more important question for public policy is whether AI platforms can help the millions of people who are in need of legal help around their housing, family, domestic violence, debt, criminal records, and other important problems. When a person comes to a well-known, general generative AI platform to ask about their legal problem, what is the quality of the platform’s response? Measuring quality is difficult in the legal domain, because there are few standardized sets of rubrics to judge things like the quality of a professional’s response to a person’s request for advice. This study presents a proposed set of 22 specific criteria to evaluate the quality of a system’s answers to a person’s request for legal help for a civil justice problem. It also presents the review of these evaluation criteria by legal domain experts like legal aid lawyers, courthouse self help center staff, and legal help website administrators. The result is a set of standards, context, and proposals that technologists and policymakers can use to evaluate the quality of this specific legal help task in future benchmark efforts.

Keywords: AI, justice, access to justice, technology policy

Suggested Citation

Hagan, Margaret, Good AI Legal Help, Bad AI Legal Help: Establishing quality standards for responses to people’s legal problem stories (December 18, 2023). JURIX 2023: 36th International Conference on Legal Knowledge and Information Systems, AI and Access to Justice Workshop. December 2023., Available at SSRN: https://ssrn.com/abstract=4696936

Margaret Hagan (Contact Author)

Stanford Legal Design Lab ( email )

559 Nathan Abbott Way
Stanford, CA 94305
United States

HOME PAGE: http://margarethagan.com

Stanford Law School ( email )

559 Nathan Abbott Way
Stanford, CA 94305-8610
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
114
Abstract Views
318
Rank
446,450
PlumX Metrics