Creative Commons source reading · Evaluation, bias and explainability
ContractEval: Benchmarking LLMs for Clause-Level Legal Risk Identification in Commercial Contracts
Shuang Liu; Zelong Li; Ruoyun Ma; Haiyan Zhao; Mengnan Du
Originally published in Proceedings of the Natural Legal Language Processing Workshop 2025 · 2025-11
From the article
Selected source text. JournalOS removes markup and shortens excerpts where indicated. The full article provides the complete context.
Abstract
The potential of large language models (LLMs) in contract legal risk analysis remains underexplored. In response, this paper introduces ContractEval, the first benchmark to thoroughly evaluate whether open-source LLMs could match proprietary LLMs in identifying clause-level legal risks in commercial contracts. Using the Contract Understanding Atticus Dataset (CUAD), we assess 4 proprietary and 15 open-source LLMs. Our results highlight five key findings: (1) Proprietary models outperform open-source models in both correctness and output effectiveness. (2) Larger open-source models generally perform better, though the improvement slows down as models get bigger. (3) Reasoning (“thinking”) mode improves output effectiveness but reduces correctness, likely due to over-complicating simpler tasks. (4) Open-source models generate “no related clause” responses more frequently even when relevant clauses are present. (5) Model quantization speed up inference but at the cost of performance drop, showing the tradeoff between efficiency and accuracy.
Editorial reading guide
- Which legal corpus, jurisdiction and evaluation task does the article use?
- What evidence supports the reported performance, and where could the method fail in practice?
