MLR: Enhancing Small Models' Long-Chain Reasoning with a Multi-Level Reasoning Architecture

When Long-Chain Reasoning Goes Off Track: The Plan Drift Problem for Small Models

Inference-time scaling enhances model reasoning by extending the chain of thought (CoT), an approach that has proven highly effective in large language models. However, existing methods typically rely on a single policy and are trained with outcome-reward RL. The problem is that long-horizon planning failures occur—the model's implicit reasoning plan gradually deviates from an effective strategy, which is especially severe for small models with limited capacity.

The essence of this "plan drift" is that in a reasoning trajectory spanning thousands of tokens, intermediate steps lack explicit structural constraints, so the model easily produces locally plausible content that globally drifts from the goal. Outcome rewards provide feedback only at the end of the trajectory, and errors in the intermediate process cannot be corrected in time.

MLR's Core Idea: Splitting the Long Chain into "Planning—Execution" Layers

To address the above problem, researchers propose Multi-Level Reasoning (MLR), which reframes long CoT generation as a two-level stochastic process.

High-Level Planner and Low-Level Executor

The high-level planner generates structured step descriptors, each specifying two things simultaneously: a reasoning mode and a semantic subgoal. The low-level executor generates detailed reasoning content under the constraints of these descriptors, forming a "planning—execution" alternating loop.

The cleverness of this design lies in introducing explicit intermediate structure into long-chain reasoning. Each step is no longer freely generated but anchored by the high-level plan to a specific mode and subgoal, thereby suppressing plan drift.

Minimal Implementation: Base Model + Lightweight LoRA

To maintain scalability, MLR adopts a minimalist design: the base model itself serves as the low-level policy, and a lightweight LoRA module implements the high-level policy. This means no additional large model is introduced, training and deployment costs remain controllable, and it is especially suitable for resource-constrained small-model scenarios.

A two-level stochastic process is a hierarchical generative framework in probabilistic modeling: upper-level variables are sampled first, determining the conditional distribution of the lower level, which then generates concrete outputs under that condition. In MLR, the high-level planner samples a sequence of step descriptors, and the low-level executor generates detailed reasoning content according to the conditional distribution of each descriptor. This hierarchical structure gives the generation of reasoning trajectories a decomposable probabilistic form, allowing credit assignment and intervention at the step granularity rather than black-box sampling by a single policy over the entire trajectory.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: a pair of low-rank decomposition matrices is attached in parallel to the weight matrices of a pretrained model, and only these small parameters are updated during training while the backbone weights remain frozen, greatly reducing the number of trainable parameters and memory overhead. MLR thereby lets the base model play both the low-level execution and high-level planning roles—the backbone handles detailed reasoning generation, and the LoRA adapter outputs step descriptors. The two share the same base, which avoids introducing a second large model and facilitates training and deployment in resource-constrained scenarios.

The Training Challenge: Credit Assignment under Sparse Rewards

Even with a sound architecture, training remains a bottleneck. Researchers observed that outcome-reward RL provides sparse and delayed feedback for long trajectories (e.g., thousands of tokens), severely hindering credit assignment—the model struggles to determine which step caused the final result to be good or bad.

Iterative Step-DPO and TSMC

To this end, MLR introduces iterative Step-DPO, a process-level preference optimization scheme. It leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable step-by-step supervision. Compared with relying solely on final outcomes, this method provides denser training signals at the step granularity, leading to more effective training, higher stability, and better accuracy.

From an engineering perspective, process-level supervision combined with TSMC's sampling mechanism essentially estimates "future potential" for each key decision point in a long trajectory, so that credit assignment no longer relies only on terminal rewards.

Credit assignment is a classic problem in reinforcement learning: when a trajectory ultimately receives a reward, one must determine which actions in the sequence contributed most to the outcome. In long CoT scenarios, trajectories can span thousands of tokens, while the outcome reward provides only a single scalar at the end, so the contributions of intermediate steps are severely diluted. Common countermeasures include introducing process reward models, step-level preference annotation, or step-by-step estimation based on value functions. MLR's choice of process-level preference optimization follows the latter line of thought, using scalable sampling to estimate the relative quality of each step and thereby distributing the terminal signal to the step granularity.

Sequential Monte Carlo (SMC) approximates complex distributions by progressively sampling a set of weighted particles, and is suitable for probabilistic inference over long sequences. Twisted SMC introduces a "twisting function" at each step as a look-ahead estimate that evaluates the future potential of the current partial trajectory in advance, thereby more efficiently guiding particles toward high-reward regions. In MLR, TSMC is used to provide step-by-step supervision signals for each key decision point in long reasoning trajectories: the twisting function approximates future returns, and particle weights reflect the relative contribution of each step, so that the construction of preference pairs for Step-DPO no longer depends on a single terminal reward.

Experimental Results: Comprehensive Leadership under a Small Data Budget

Researchers conducted extensive experiments on challenging mathematical, scientific, and logical reasoning benchmarks. The key results are noteworthy:

• Outstanding data efficiency: Under the same and reduced data budget (using only 10% of the SFT data and 5% of the preference data relative to the DeepSeek^Ad-R1 distillation setting), MLR outperforms SFT-based distillation methods and strong RL/preference optimization baselines across multiple base models and tasks.
• Stronger long-horizon robustness: MLR exhibits slower performance degradation in long-horizon reasoning, indicating greater robustness when scaling CoT generation.

Taken together, MLR's value lies not only in "achieving better results with less data," but also in mitigating the performance decay that worsens with length in long-chain reasoning—precisely the most realistic obstacle when deploying small models on reasoning tasks.

Why It's Worth Watching

The insight from MLR is that rather than letting a single policy "freely improvise" over ultra-long trajectories, it is better to explicitly introduce a hierarchical structure that decouples planning from execution. The design in which lightweight LoRA handles high-level planning while the base model handles execution also offers a low-cost engineering path for enhancing small models' capabilities.

Of course, coordination between the two-level policies, the choice of granularity for step descriptors, and the extra computational overhead of TSMC sampling are all factors to weigh in practical deployment. But from the perspective of methodological completeness, MLR combines the two directions of "structural constraints" and "process supervision," offering a new paradigm for inference-time scaling that is worth following.