Creative Commons source reading · Judgment prediction and analysis
Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark
Joel Niklaus; Ilias Chalkidis; Matthias Stürmer
Originally published in Proceedings of the Natural Legal Language Processing Workshop 2021 · 2021-11
From the article
Selected source text. JournalOS removes markup and shortens excerpts where indicated. The full article provides the complete context.
Abstract
In many jurisdictions, the excessive workload of courts leads to high delays. Suitable predictive AI models can assist legal professionals in their work, and thus enhance and speed up the process. So far, Legal Judgment Prediction (LJP) datasets have been released in English, French, and Chinese. We publicly release a multilingual (German, French, and Italian), diachronic (2000-2020) corpus of 85K cases from the Federal Supreme Court of Switzer- land (FSCS). We evaluate state-of-the-art BERT-based methods including two variants of BERT that overcome the BERT input (text) length limitation (up to 512 tokens). Hierarchical BERT has the best performance (approx. 68-70% Macro-F1-Score in German and French). Furthermore, we study how several factors (canton of origin, year of publication, text length, legal area) affect performance. We release both the benchmark dataset and our code to accelerate future research and ensure reproducibility.
Editorial reading guide
- Which legal corpus, jurisdiction and evaluation task does the article use?
- What evidence supports the reported performance, and where could the method fail in practice?
