MoE Routing Signals: A New Paradigm for Process Supervision

Process Reward Models (PRMs) provide fine-grained supervision for multi-step reasoning, but obtaining reliable process labels has always been a core bottleneck. Existing methods rely on human annotation, rollout-based estimation, or auxiliary verification—all costly and difficult to scale. A new study explores another source of process supervision natively exposed by Mixture-of-Experts (MoE) models: expert routing. The research shows that simple routing statistics can provide an intrinsic process confidence signal whose effectiveness is competitive with conventional output logit confidence.

Routing Confidence: From Computational Scheduling to Supervision Signal

The research team found that when an MoE model processes each reasoning step, the distribution pattern of expert routing implicitly encodes the model's confidence in that step. They constructed a routing confidence metric that combines three dimensions: overall routing quality, top routing quality, and intra-step routing variation. Together, these three dimensions characterize the model's degree of "hesitation" during reasoning—when the routing distribution is dispersed and changes sharply, it often means the model lacks confidence in that step.

The significance of this finding lies in redefining routing decisions—originally regarded as a pure computational scheduling mechanism in MoE architectures—as an architecture-native source of process supervision. Compared with methods that require additional verifier training or rely on human annotation, the routing signal comes entirely from a natural byproduct of model inference at no extra cost.

Mixture-of-Experts (MoE) architecture: MoE is a sparse-activation neural network design consisting of multiple "expert" subnetworks and a router (gating network). For each input token, the router computes a weight distribution over experts and typically activates only the top-k experts for computation. This design allows the model's parameter count to grow substantially while keeping computation manageable. Crucially, the routing distribution itself is a decision result of the model on the input: the more concentrated the distribution, the more clearly the model knows which experts should handle it; the more dispersed the distribution, the more the model is "hesitating" among multiple experts. Therefore, routing statistics naturally carry information about the model's internal state, which provides the architectural basis for using them as a confidence signal.

Process Reward Models (PRMs) and process supervision: Traditional reward models (ORMs) assign a single score only to the final answer, whereas process reward models score each intermediate step in a reasoning chain separately, providing finer-grained supervision. PRMs can be used for step filtering and search guidance during inference, as well as for reranking or reinforcement learning rewards during training. Their core bottleneck is the acquisition of process labels: manually annotating whether each step is correct is extremely costly and hard to scale; rollout-based estimation requires extensive sampling computation; and auxiliary verifiers introduce additional models. The routing signal proposed in this paper is valuable precisely because it bypasses these costs by obtaining step-level confidence directly from a natural byproduct of MoE inference.

RoCo-Rethink: Selectively Regenerating Low-Confidence Steps

Based on routing confidence, the researchers proposed the RoCo-Rethink method. This method first identifies the lowest-confidence step in a reasoning trajectory, then selectively regenerates the subsequent trajectory rather than restarting from scratch. This "local retry" strategy targets weak links for correction while maintaining the overall coherence of reasoning.

Experiments across multiple MoE architectures and diverse reasoning tasks show that RoCo-Rethink consistently outperforms strong baselines. Its advantage lies in concentrating computational resources on the steps that truly need correction, avoiding the waste of global regeneration.

RoCo-PRM: Automatic Annotation and Distillation with Strong MoE

Another contribution is RoCo-PRM. This method uses a strong MoE model to automatically annotate reasoning trajectories with routing-based step scores, then distills these scores into an independent PRM. In this way, PRM training no longer depends on human process labels but automatically obtains supervision signals from the MoE's routing behavior.

Experiments show that RoCo-PRM not only surpasses baselines on multiple tasks but also transfers effectively across model architectures and tasks. This means that routing-derived process supervision has good generalization ability and is not limited to a specific model or task type.

Application of knowledge distillation in PRM training: Distillation generally refers to training a small model (student) using the output distribution or soft labels of a large model (teacher), enabling the student to approximate the teacher's capability at lower cost. In RoCo-PRM, the teacher is a strong MoE model, and its "soft labels" do not come from output logits but from step scores computed from routing statistics. The student is an independent PRM that learns to score reasoning steps. The advantage of this approach is that the teacher signal requires no human annotation, and routing scores contain richer step-level information than whether the final answer is correct; the student model, meanwhile, does not need to access the MoE's internal routing at inference time, making deployment more flexible—which also explains why cross-architecture transfer is possible.

Implications of Architecture-Native Supervision

The core insight of this work is that expert routing is not merely a computational scheduling mechanism; it provides an architecture-native source of process supervision. For MoE models, routing decisions are themselves an expression of the model's internal state and can reflect confidence changes during reasoning. This perspective opens a new path for process supervision—extracting supervision signals directly from architectural characteristics without external annotation or auxiliary verifiers.

In the future, this idea may inspire more research on self-supervision using internal model signals, advancing the reliability of reasoning models on complex tasks.

One-line Summary

Uses MoE expert routing statistics as a native process supervision signal for low-confidence step regeneration and automatic PRM annotation and distillation.

Paragraph Summary

This article explores the potential of expert routing in Mixture-of-Experts (MoE) models as a new source of process supervision. The research shows that routing distribution statistics can reflect the model's confidence in each reasoning step, with effectiveness competitive with output logit confidence and without requiring human annotation or additional verifiers. Based on this, the authors propose RoCo-Rethink, which identifies the lowest-confidence steps and selectively regenerates subsequent trajectories, outperforming strong baselines across multiple MoE architectures and reasoning tasks; they also propose RoCo-PRM, which uses a strong MoE to automatically annotate routing scores and distill them into an independent PRM, enabling transfer across architectures and tasks. The core insight is that routing is not only a computational scheduling mechanism but also an architecture-native source of process supervision, opening a new path for PRM training without human process labels.