Validity Assessment of Legal Will Statements as Natural Language Inference

Findings of the Association for Computational Linguistics: EMNLP 2022 (forthcoming 2022)

Arizona Legal Studies Discussion Paper No. 22-21

11 Pages Posted: 17 Nov 2022 Last revised: 21 Nov 2022

See all articles by Alice Kwak

Alice Kwak

The University of Arizona - Department of Linguistics

Jacob Israelsen

University of Arizona - James E. Rogers College of Law

Clayton Morrison

University of Arizona - School of Information

Derek E. Bambauer

University of Arizona - James E. Rogers College of Law

Mihai Surdeanu

University of Arizona - Department of Computer Science

Date Written: November 15, 2022

Abstract

This work introduces a natural language inference (NLI) dataset that focuses on the validity of statements in legal wills. This dataset is unique because: (a) each entailment decision requires three inputs: the statement from the will, the law, and the conditions that hold at the time of the testator’s death; and (b) the included texts are longer than the ones in current NLI datasets. We trained eight neural NLI models in this dataset. All the models achieve more than 80% macro F1 and accuracy, which indicates that neural approaches can handle this task reasonably well. However, group accuracy, a stricter evaluation measure that is calculated with a group of positive and negative examples generated from the same statement as a unit, is in mid 80s at best, which suggests that the models’ understanding of the task remains superficial. Further ablative analyses and explanation experiments indicate that all three text segments are used for prediction, but some decisions rely on semantically irrelevant tokens. This indicates that overfitting on these longer texts likely happens, and that additional research is required for this task to be solved.

Keywords: natural language, natural language inference, NLI, machine learning, smart contracts, blockchain, AI, artificial intelligence, algorithm, automation, texts, legal wills

Suggested Citation

Kwak, Alice and Israelsen, Jacob and Morrison, Clayton and Bambauer, Derek E. and Surdeanu, Mihai, Validity Assessment of Legal Will Statements as Natural Language Inference (November 15, 2022). Findings of the Association for Computational Linguistics: EMNLP 2022 (forthcoming 2022), Arizona Legal Studies Discussion Paper No. 22-21, Available at SSRN: https://ssrn.com/abstract=4279728

Alice Kwak

The University of Arizona - Department of Linguistics ( email )

United States

Jacob Israelsen

University of Arizona - James E. Rogers College of Law ( email )

P.O. Box 210176
Tucson, AZ 85721-0176
United States

Clayton Morrison

University of Arizona - School of Information ( email )

United States

Derek E. Bambauer (Contact Author)

University of Arizona - James E. Rogers College of Law ( email )

P.O. Box 210176
Tucson, AZ 85721-0176
United States

Mihai Surdeanu

University of Arizona - Department of Computer Science ( email )

United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
22
Abstract Views
81
PlumX Metrics