Teaching AI to Read Judicial Reasoning: A New Sentence-Level Benchmark for the German Constitutional Court

When Large Language Models Meet Judicial Reasoning

Judicial reasoning has long been one of the most challenging frontiers for large language models (LLMs). Interpreting legal text goes far beyond surface-level meaning — it involves legislative purpose, structural logic, and historical context across multiple dimensions. A new paper on arXiv (arXiv:2609.26945v1) tackles this challenge by proposing a sentence-level evaluation benchmark specifically designed to test how well LLMs can identify interpretive canons — the principled methods judges use to construe legal text.

The theoretical foundation of this work draws from the German legal tradition — rooted in Savigny and systematized by Larenz — which categorizes legal interpretation into grammatical, systematic, historical, teleological, and related approaches. This framework is a cornerstone of civil law judicial reasoning. The researchers' goal is to translate these abstract jurisprudential categories into classification criteria that machines can actually evaluate.

Friedrich Carl von Savigny, the 19th-century founder of the German Historical School of jurisprudence, first proposed a systematic four-element framework for legal interpretation in his System of the Modern Roman Law: the grammatical element (word meaning), the logical element (internal structure of provisions), the historical element (legislative background), and the systematic element (position within the legal order). In the 20th century, Karl Larenz refined this framework by adding teleological interpretation, forming the five-element system widely used in civil law legal education. Unlike the common law tradition, which relies heavily on case-based reasoning, civil law places greater emphasis on deductive reasoning from statutory text. Identifying which interpretive method a judge employs in their argumentation is therefore critical to understanding the logic behind a ruling — and this is precisely what makes operationalizing this tradition as a machine learning task so valuable.

Three Core Contributions

The paper identifies three main contributions. First, the authors operationalize Larenz's conception of legal interpretation into an executable set of classification criteria. This step alone is non-trivial: translating the flexible, context-sensitive methods described by legal scholars into well-bounded, annotatable categories forms the entire foundation of the study.

Second, the researchers built and publicly released a dataset of decisions from the German Federal Constitutional Court, annotated at the sentence level. Choosing constitutional court rulings as the corpus means the data carries an exceptionally high density of legal reasoning and domain-specific complexity.

Third, the paper provides baseline evaluation results across four LLMs from three model families, comparing two prompting strategies: expert-written prompts and prompts automatically optimized using a Genetic-Pareto Algorithm (GEPA).

The Genetic-Pareto Algorithm (GEPA) is an automatic prompt optimization method that combines evolutionary computation with multi-objective optimization. Its core idea is to treat prompt texts as evolvable individuals, generating candidate prompt populations through genetic operations such as crossover and mutation, then selecting optimal solutions that perform well across multiple evaluation metrics using the Pareto front criterion — avoiding the overfitting that single-objective optimization can produce. The advantage of such automated prompt optimization methods is that they do not rely on human domain expertise and are theoretically transferable across domains. However, the results of this study show that on tasks requiring deep domain understanding — such as legal interpretation — algorithmically generated prompts did not outperform those written by legal experts, providing a concrete empirical case study of the limits of such approaches.

Evaluation Results: Expert Prompts Remain Robust

The data shows that average F1 scores across seven binary classification subtasks cluster between 70.4 and 79.2 across all tested models. This range indicates that LLMs have developed a meaningful capacity to recognize judicial interpretation methods — but a notable gap remains before they can be reliably deployed in practice.

Which Canon Is Easiest to Identify — and Which Is Hardest?

The study reveals an interesting pattern: grammatical interpretation is consistently the easiest canon for models to identify, while systematic interpretation is consistently the hardest. This aligns with intuition — grammatical interpretation focuses on the literal wording of text, making its features relatively explicit. Systematic interpretation, by contrast, requires the model to understand a provision's position and logical relationships within the broader legal system, demanding cross-sentence and cross-provision reasoning that is inherently more complex.

Systematic interpretation requires the reader to situate a single provision within the context of an entire statute or even the whole legal system — for example, whether a concept defined in the general provisions of civil law should extend to administrative regulations, or what logical tension exists between a particular clause and constitutional fundamental rights. This type of reasoning spans individual sentences, demands a holistic grasp of the macro-structure of legal texts, and is essentially a multi-hop reasoning task. For LLMs, this requires not only understanding the semantics of the current sentence, but also maintaining cross-provision reference relationships within the context window while drawing on implicit knowledge of legal system structure. This explains why systematic interpretation consistently emerges as the most difficult category across all subtasks.

GEPA-Optimized Prompts Fail to Systematically Outperform Expert Prompts

One thought-provoking finding: under the test configurations used in this study, GEPA-optimized prompts did not systematically outperform expert-written prompts. This finding carries a dual message. On one hand, it demonstrates that prompts crafted by domain experts drawing on legal knowledge constitute a meaningful, competitive baseline in their own right. On the other hand, it suggests that automated prompt optimization methods cannot straightforwardly replace human expert judgment on highly specialized tasks that require deep domain knowledge.

Value and Limitations of This Research

The significance of this work lies in pushing legal AI evaluation down from high-level tasks like verdict prediction and statute retrieval to the more granular level of reasoning method identification. Understanding which interpretive canon a passage of judicial argument employs is a key step toward understanding why a judge ruled the way they did. For developers looking to integrate LLMs into legal assistance workflows, a sentence-level benchmark like this provides a far more precise measure of model capability.

At the same time, the limitations deserve acknowledgment: F1 scores in the 70–80 range imply error rates that remain non-trivial, especially on complex tasks like systematic interpretation. Legal reasoning demands extremely high accuracy, and current model performance is better suited as a research reference point than as a production-ready judicial assistance tool.

With the public release of domain-specific datasets and the establishment of evaluation standards, future research can build iteratively on this benchmark, advancing LLMs' understanding of civil law judicial reasoning toward greater depth and reliability.