Creative Commons source reading · Judgment prediction and analysis
Legal Judgment Prediction: If You Are Going to Do It, Do It Right
Masha Medvedeva; Pauline Mcbride
Originally published in Proceedings of the Natural Legal Language Processing Workshop 2023 · 2023-12
From the article
Selected source text. JournalOS removes markup and shortens excerpts where indicated. The full article provides the complete context.
Abstract
The field of Legal Judgment Prediction (LJP) has witnessed significant growth in the past decade, with over 100 papers published in the past three years alone. Our comprehensive survey of over 150 papers reveals a stark reality: only ~7% of published papers are doing what they set out to do - predict court decisions. We delve into the reasons behind the flawed and unreliable nature of the remaining experiments, emphasising their limited utility in the legal domain. We examine the distinctions between predicting court decisions and the practices of legal professionals in their daily work. We explore how a lack of attention to the identity and needs of end-users has fostered the misconception that LJP is a near-solved challenge suitable for practical application, and contributed to the surge in academic research in the field. To address these issues, we examine three different dimensions of ‘doing LJP right’: using data appropriate for the task; tackling explainability; and adopting an application-centric approach to model reporting and evaluation.
Editorial reading guide
- Which legal corpus, jurisdiction and evaluation task does the article use?
- What evidence supports the reported performance, and where could the method fail in practice?
