Fine-Tuning LLMs for Legal Headnote Generation: Why It Fails and How to Fix It

A Real-World Legal AI Dilemma

Recently, a developer posted a highly representative technical challenge on Reddit: can you fine-tune an open-source LLM to automatically generate the complex "legal principles" or "headnotes" (headnote/ratio decidendi) found in court judgments?

The developer had access to a massive local database of court decisions — roughly 500,000 rulings, about 20% of which included a "ratio" (a summary of the legal reasoning). He attempted to fine-tune models in the Gemma family (the "gemma4 26B A4B" mentioned in the post appears to describe a Gemma-architecture MoE model with ~26B total parameters but a much smaller number of activated parameters). Neither the base nor the instruction-tuned (IT) versions could outperform simply prompting the base model directly on his self-built evaluation set.

This case distills nearly every classic pain point in vertical-domain LLM deployment, and it's worth unpacking in depth.

Why Fine-Tuning Failed to Beat Direct Prompting

The Task Itself Operates at Too High an Abstraction Level

Extracting verbatim passages from a judgment and "generating a concise legal principle" are two fundamentally different tasks in terms of difficulty. The former is essentially an extractive task — the model just needs to locate and copy. The latter is a task of highly abstract reasoning and synthesis, requiring the model to understand the case facts, the logic of statutory application, and then re-express that understanding in precise legal language.

The developer had already observed this: the extraction task could "match the gold" standard answer, but the output was full of padding and generic phrasing, making it practically unusable. This is exactly the "same form, different substance" problem — the model learned to mimic the surface structure of the output without grasping the underlying legal reasoning. And that kind of deep reasoning is precisely what small-scale fine-tuning struggles most to inject.

The Ceiling of Activated Parameters in MoE Architecture

If the model is indeed a MoE architecture (the "A4B" label suggests roughly 4B activated parameters), then for complex reasoning tasks, the number of parameters actually participating in computation is the critical bottleneck. Generating legal headnotes requires logical synthesis across long stretches of text, which places extremely high demands on the model's effective "thinking capacity." Lightweight fine-tuning methods like LoRA are simply hard-pressed to reshape this capability at such a small activation scale.

Common Pitfalls When Fine-Tuning Legal LLMs

Data Quality Matters Far More Than Data Quantity

The developer mentioned filtering for the "highest quality" training data. But in legal headnote generation, consistency within the training data is often more critical than volume. Among 500,000 rulings, the style, granularity, and level of abstraction of ratios written by different judges across different eras can vary enormously. If training samples are stylistically inconsistent, the model picks up contradictory signals and ultimately produces vague, "averaged-out" output — which neatly explains the padding and generalizations he observed.

Recommended approaches:

• Cluster ratios by length, structure, and abstraction level; prioritize stylistically consistent subsets
• Use a small number of extremely high-quality (human-verified) samples rather than blindly scaling up volume
• Clearly distinguish between "factual summaries" and "legal principles" to avoid mixing task signals

Is the Evaluation Standard Itself Reliable?

The developer built his own LLM-based evaluation. This harbors a subtle trap: if the evaluation metric is biased, fine-tuning amounts to optimizing in the wrong direction. The quality of a legal headnote is extremely difficult to measure with text similarity metrics like BLEU or ROUGE, because a legally correct formulation may share almost no wording with the gold standard.

If evaluation relies too heavily on surface text matching, a general-purpose LLM that's directly prompted may actually appear "better" simply because its language is more fluent. It's worth considering blind evaluations by legal experts, or designing evaluation dimensions based on "coverage of legal elements."

Hidden Technical Errors from Vibe Coding

The developer admitted the entire project was built by "vibe coding" with Claude — essentially having AI write the code by feel. This carries real risk in a fine-tuning pipeline. Issues like prompt template concatenation in data preprocessing, loss masking (label masking), and mismatches between instruction format and the model's chat template are exactly the kind of "silent errors" that are easy to introduce and hard to detect.

In particular, the prompt format used during training must be strictly consistent with the format used at inference time, and the conversation template for IT models is frequently processed incorrectly. Any single mismatch can severely undermine fine-tuning effectiveness — with no error thrown.

Is Automated Legal Headnote Generation Actually Feasible?

A More Pragmatic Technical Path

The conclusion: difficult, but not impossible — the path just needs adjustment. For a mid-sized model directly generating high-quality legal headnotes, the marginal returns of pure supervised fine-tuning are limited. More pragmatic alternatives include:

1. Task decomposition (pipeline): First extract key statutes and facts, then synthesize in steps — rather than end-to-end generation in one shot
2. RAG + fine-tuning: Retrieve similar precedents as context, reducing the pressure on the model to "generate from thin air"
3. Choose a larger base or reasoning-focused model: Complex legal reasoning benefits significantly from model scale and reasoning capacity
4. DPO preference optimization: After SFT, use preference pairs of "good headnote vs. bad headnote" to further align output quality

Lessons for Vertical-Domain AI Deployment

This case is a reminder: fine-tuning is not a magic key. When even a directly prompted general-purpose LLM struggles with a highly abstract task, hoping that lightweight fine-tuning on a small-to-medium model will "turn lead into gold" is usually a dead end. The real leverage points are: task decomposition, data consistency, rigorous evaluation methodology, and a clear-eyed understanding of the model's capability boundaries.

For anyone deploying AI in a vertical domain, the more efficient path is typically to first validate the task ceiling using the strongest available general-purpose model (with carefully crafted prompts and few-shot examples), and only then decide whether to invest in fine-tuning.