Journal DOI 10.68956/journal.legal-engineering-technology

Creative Commons source reading · Evaluation, bias and explainability

LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation

Joseph Enguehard; Morgane Van Ermengem; Kate Atkinson; Sujeong Cha; Arijit Ghosh Chowdhury; Prashanth Kallur Ramaswamy; Jeremy Roghair; Hannah R Marlowe; Carina Suzana Negreanu; Kitty Boxall; Diana Mincu

Originally published in Proceedings of the Natural Legal Language Processing Workshop 2025 · 2025-11

From the article

Selected source text. JournalOS removes markup and shortens excerpts where indicated. The full article provides the complete context.

Abstract

Evaluating large language model (LLM) outputs in the legal domain presents unique challenges due to the complex and nuanced nature of legal analysis. Current evaluation approaches either depend on reference data, which is costly to produce, or use standardized assessment methods, both of which have significant limitations for legal applications.Although LLM-as-a-Judge has emerged as a promising evaluation technique, its reliability and effectiveness in legal contexts depend heavily on evaluation processes unique to the legal industry and how trustworthy the evaluation appears to the human legal expert.

Editorial reading guide

  • Which legal corpus, jurisdiction and evaluation task does the article use?
  • What evidence supports the reported performance, and where could the method fail in practice?