Journal DOI 10.68956/journal.legal-engineering-technology

Creative Commons source reading · Legal language models

Effectively Leveraging BERT for Legal Document Classification

Nut Limsopatham

Originally published in Proceedings of the Natural Legal Language Processing Workshop 2021 · 2021-11

From the article

Selected source text. JournalOS removes markup and shortens excerpts where indicated. The full article provides the complete context.

Abstract

Bidirectional Encoder Representations from Transformers (BERT) has achieved state-of-the-art performances on several text classification tasks, such as GLUE and sentiment analysis. Recent work in the legal domain started to use BERT on tasks, such as legal judgement prediction and violation prediction. A common practise in using BERT is to fine-tune a pre-trained model on a target task and truncate the input texts to the size of the BERT input (e.g. at most 512 tokens). However, due to the unique characteristics of legal documents, it is not clear how to effectively adapt BERT in the legal domain. In this work, we investigate how to deal with long documents, and how is the importance of pre-training on documents from the same domain as the target task. We conduct experiments on the two recent datasets: ECHR Violation Dataset and the Overruling Task Dataset, which are multi-label and binary classification tasks, respectively. Importantly, on average the number of tokens in a document from the ECHR Violation Dataset is more than 1,600.

Editorial reading guide

  • Which legal corpus, jurisdiction and evaluation task does the article use?
  • What evidence supports the reported performance, and where could the method fail in practice?