Journal DOI 10.68956/journal.legal-engineering-technology

Creative Commons source reading · Legal language models

Mixed-domain Language Modeling for Processing Long Legal Documents

Wenyue Hua; Yuchen Zhang; Zhe Chen; Josie Li; Melanie Weber

Originally published in Proceedings of the Natural Legal Language Processing Workshop 2023 · 2023-12

From the article

Selected source text. JournalOS removes markup and shortens excerpts where indicated. The full article provides the complete context.

Abstract

The application of Natural Language Processing (NLP) to specialized domains, such as the law, has recently received a surge of interest. As many legal services rely on processing and analyzing large collections of documents, automating such tasks with NLP tools such as language models emerges as a key challenge since legal documents may contain specialized vocabulary from other domains, such as medical terminology in personal injury text. However, most language models are general-purpose models, which either have limited reasoning capabilities on highly specialized legal terminology and syntax, such as BERT or ROBERTA, or are expensive to run and tune, such as GPT-3.5 and Claude. Thus, in this paper, we propose a specialized language model for personal injury text, LEGALRELECTRA, which is trained on mixed-domain legal and medical corpora. We show that as a small language model, our model improves over general-domain and single-domain medical and legal language models when processing mixed-domain (personal injury) text.

Editorial reading guide

  • Which legal corpus, jurisdiction and evaluation task does the article use?
  • What evidence supports the reported performance, and where could the method fail in practice?