Creative Commons source reading · Information extraction and summarization
Do Language Models Learn about Legal Entity Types during Pretraining?
Claire Barale; Michael Rovatsos; Nehal Bhuta
Originally published in Proceedings of the Natural Legal Language Processing Workshop 2023 · 2023-12
From the article
Selected source text. JournalOS removes markup and shortens excerpts where indicated. The full article provides the complete context.
Abstract
Language Models (LMs) have proven their ability to acquire diverse linguistic knowledge during the pretraining phase, potentially serving as a valuable source of incidental supervision for downstream tasks. However, there has been limited research conducted on the retrieval of domain-specific knowledge, and specifically legal knowledge. We propose to explore the task of Entity Typing, serving as a proxy for evaluating legal knowledge as an essential aspect of text comprehension, and a foundational task to numerous downstream legal NLP applications. Through systematic evaluation and analysis and two types of prompting (cloze sentences and QA-based templates) and to clarify the nature of these acquired cues, we compare diverse types and lengths of entities both general and domain-specific entities, semantics or syntax signals, and different LM pretraining corpus (generic and legal-oriented) and architectures (encoder BERT-based and decoder-only with Llama2).
Editorial reading guide
- Which legal corpus, jurisdiction and evaluation task does the article use?
- What evidence supports the reported performance, and where could the method fail in practice?
