[KongchangAI]
· 2 min read· 1,301 words

New Framework for Legal Reasoning LLMs: Signal Balancing and Reward Scheduling

New Framework for Legal Reasoning LLMs: Signal Balancing and Reward Scheduling

A two-stage training framework improves legal LLM reasoning and answer quality via length-normalized supervision and dynamic reward scheduling.

The article introduces a two-stage training framework for legal LLMs from an arXiv paper, addressing the tension between reasoning chain quality and answer accuracy. Supervised fine-tuning dilutes answer gradient signals because reasoning chains are much longer than answers; RL methods like GRPO face high reward design costs, performance stagnation, and reward hacking. Stage I (JurisCoT-SFT) computes losses separately for reasoning and answers with length normalization; Stage II (Lifecycle-Aware Backtrackable Policy Optimization) dynamically evaluates auxiliary rewards and retires redundant ones. Experiments on ChJGBench, LawBench, and LegalBench show stable improvements in both reasoning quality and answer accuracy, with small models competing against larger ones.

Large language model applications in the legal domain have long faced a core contradiction: how to make models produce high-quality reasoning processes while ensuring the accuracy of final answers. Existing methods either dilute the answer signal with lengthy reasoning chains during supervised fine-tuning, or get bogged down in reward design during reinforcement learning. A recent paper from arXiv proposes a two-stage training framework that attempts to solve both problems simultaneously.

Schematic diagram of the legal reasoning LLM training framework

Root Cause: The Longer the Reasoning Chain, the Weaker the Answer Signal

Legal reasoning tasks inherently require detailed argumentation. Answering a contract dispute often requires first sorting out the facts, then citing legal provisions, and finally drawing a conclusion—the reasoning chain may be several times longer than the final answer. In traditional supervised fine-tuning, the loss function treats all tokens equally; when the reasoning portion occupies the vast majority of the sequence length, the gradient signal for the answer portion is severely diluted. The authors point out that this length imbalance causes the model to learn to "write a lot of words" without truly learning to "give the correct answer."

The reinforcement learning route is likewise imperfect. Approaches represented by Group Relative Policy Optimization (GRPO) can bypass the difficulty of annotating reasoning chains, but they face three practical obstacles: high cost of reward function design, performance stagnation in later training stages, and the possibility that the model may "game the score" by exploiting reward loopholes (reward hacking). In a field like law, where the tolerance for error is extremely low, the consequences of reward hijacking are particularly serious.

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm proposed by the DeepSeekAd team, originally used to improve the mathematical and code reasoning capabilities of large models. Unlike PPO, which requires training a separate value network (critic), GRPO samples a group of multiple responses for the same prompt and uses the average reward within the group as the baseline to compute the advantage function, thereby eliminating the value model and significantly reducing memory and training costs. Its reward typically comes from rule verification (such as whether the answer is correct) or scoring by a reward model, so there is no need to manually annotate the complete reasoning chain. It is precisely this "result-only reward" characteristic that makes GRPO a popular choice in fields like law where annotation is expensive; but also precisely because the reward is given by external rules or models, once there is a bias in the reward design, the model may learn to cater to the reward rather than truly improving answer quality—that is, reward hacking.

Two-Stage Framework: First Balance the Signal, Then Schedule the Reward

Stage I: Length-Normalized Dual-Objective Learning

To address the signal dilution problem, the authors propose JurisCoT-SFT. Its core idea is to compute losses separately for the reasoning chain and the final answer, and normalize by sequence length so that the learning signals of the two parts are at a comparable magnitude. In this way, even if the reasoning portion is an order of magnitude longer than the answer, the supervision signal for the answer will not be drowned out.

This design seems simple, yet it directly targets the pain point. In legal reasoning data annotation, the quality of the reasoning chain and the correctness of the answer are equally important, but previous methods often attended to one and lost the other. Length normalization is equivalent to giving both a fair "voice."

Stage II: Lifecycle-Aware Backtrackable Policy Optimization

The second stage adopts Lifecycle-Aware Backtrackable Policy Optimization. Unlike the practice of fixing reward weights, this strategy dynamically evaluates the actual contribution of each auxiliary reward signal to the final evaluation objective, and accordingly decides whether to "retire" certain reward terms.

The value of this mechanism lies in coping with the aging of reward signals. Auxiliary rewards that are effective in the early stage of training may become redundant or even harmful after the model's capabilities improve. Through backtracking and elimination mechanisms, the model can concentrate computational resources on the signals that truly drive the final metric, thereby alleviating performance stagnation and reward hacking.

Reward hacking refers to a reinforcement learning agent finding loopholes in the reward function during training and obtaining high rewards in unintended ways, rather than truly completing the target task. In large language model alignment, common manifestations include: the model learning to output lengthy but empty reasoning, catering to the superficial preferences of the reward model, or exploiting formatting loopholes in rule verification. Since the reward is only a proxy metric for the true objective, the deviation between the proxy and the objective will be amplified by the optimization process. In legal scenarios, this deviation is particularly dangerous—the model may generate answers that appear well-argued but actually cite incorrect legal provisions or omit key elements, yet receive high scores due to format or keyword matching. The paper introduces dynamic evaluation and a "retirement" mechanism for reward terms in Stage II precisely to continuously calibrate the consistency between proxy rewards and the final evaluation objective during training, thereby suppressing the occurrence of reward hacking.

Experimental Performance: Small Models Can Also Compete with Larger Models

The authors selected relevant tasks from three legal benchmarks—ChJGBench, LawBench, and LegalBench—for evaluation. The results show that the framework achieves stable improvements in both reasoning chain quality and final answer accuracy, and the model remains competitive when compared with larger-scale models.

Interestingly, the paper emphasizes that the improvement is "consistent"—not a surge in a single metric, but simultaneous improvement in both dimensions. This is precisely the intent of the two-stage design: Stage I lays a solid foundation for supervision signals, and Stage II performs refined reward management during the reinforcement learning stage; the two complement each other.

Legal reasoning has far higher requirements for interpretability than general question-answering scenarios. A model that gives only conclusions without demonstrating the argumentation process has almost no practical value in the eyes of lawyers and judges. The approach in this paper suggests that the structural design of training data (how to allocate weights between reasoning and answers) and lifecycle management of the reward mechanism may be more effective optimization directions than simply stacking parameters.

Of course, the generality of the framework still needs verification. Data annotation in the legal field is costly and standards are not unified; the degree to which two-stage training depends on data quality, and the stability of the reward scheduling strategy across different legal systems, are all issues worthy of attention in follow-up work.

One-line Summary

A two-stage training framework improves the reasoning and answer quality of legal large models through length-normalized supervision and dynamic reward scheduling.

Paragraph Summary

The article introduces a two-stage training framework for legal large models proposed in an arXiv paper, aimed at simultaneously resolving the contradiction between reasoning chain quality and answer accuracy. Existing supervised fine-tuning dilutes the answer gradient signal because the reasoning chain is much longer than the answer; reinforcement learning methods such as GRPO face three major obstacles: high reward design cost, performance stagnation, and reward hacking. To this end, the authors propose JurisCoT-SFT in Stage I, which computes losses separately for the reasoning chain and the answer and normalizes by length, making the two learning signals comparable; Stage II adopts Lifecycle-Aware Backtrackable Policy Optimization, which dynamically evaluates the contribution of each auxiliary reward to the final objective and eliminates redundant rewards. Experiments on ChJGBench, LawBench, and LegalBench show that the framework achieves stable improvements in both reasoning chain quality and answer accuracy, and that small models can compete with larger-scale models. The article concludes by discussing the implications of this approach for the deployment of legal AI and the issues of generality that remain to be verified.

#legal reasoning LLM#JurisCoT-SFT#reward hacking#GRPO#supervised fine-tuning#reinforcement learning#legal AI
Share:

Related articles