DecisionRank: How On-Policy Distillation Reshapes Decision Model Reranking

Decision models handle structured tasks by directly predicting choices or scores, avoiding the latency of autoregressive decoding. But transferring ranking knowledge from generative LLMs to decision models is not easy: the teacher model expresses preferences through ranked lists of document IDs, while the student model determines ranking based on scores at the first-token position. This difference in expressive form makes knowledge distillation challenging. The recently proposed DecisionRank bridges this gap through item-space On-Policy Distillation (OPD) and achieves leading results on multiple retrieval benchmarks.

Core Problem: The Expressive Gap Between Teacher and Student

Generative LLMs typically output sequences of document IDs as lists in ranking tasks, implicitly expressing preference order. Decision models, in pursuit of low latency, directly score each document and use the score at the first-token position as the ranking basis. Although both accomplish ranking, their output spaces differ—one is a sequence, the other is a score. Traditional distillation methods struggle to directly align these two representations, preventing the student model from fully absorbing the teacher's ranking knowledge.

DecisionRank's starting point is precisely to solve this mismatch. It does not require the student model to imitate the teacher's sequence output; instead, through item-space on-policy distillation, it lets the student receive direct guidance from the teacher within its own score space.

Autoregressive decoding refers to the process in which generative models produce output token by token, with each new token depending on previously generated tokens. In ranking tasks, the teacher model must sequentially generate document ID sequences, causing inference latency to grow linearly with list length. Decision models (such as BERT-based cross-encoders), by contrast, score each document independently through a single forward pass with no sequential dependency, resulting in extremely low latency. This difference in decoding method is one of the fundamental reasons for the differing output spaces, and is also a key reason why decision models are favored in industrial retrieval systems.

Item-Space OPD: Using Ranked Lists as a Bridge

The core design of item-space OPD is: sort the document scores output by the student model into a ranked sequence (ranked list), then feed this list as input to the generative teacher model. The teacher model provides conditional feedback based on this list, directly supervising the student model's scores. In this way, the teacher's ranking knowledge is passed back through the ranked list produced by the student itself, forming a closed loop.

Specifically, the student model first scores and ranks a set of documents; after seeing this ranking, the teacher model determines which positions need adjustment and provides supervision signals targeting the scores. Because the teacher sees the student's current ranking state, the feedback is conditional and targeted, which is more effective than using fixed teacher rankings in offline distillation. Combined with offline ranking supervision, DecisionRank utilizes both static data and dynamic feedback during training, improving generalization ability.

On-Policy Distillation (OPD) originates from the on-policy learning idea in reinforcement learning: the student model generates samples based on its current policy during training (here, document ranked lists), and the teacher model provides feedback on these self-generated samples. Unlike offline distillation, which uses fixed teacher-generated data, in OPD the student explores its own policy space while the teacher corrects its errors, thereby alleviating the mismatch between training and inference distributions. This method was first applied in sequence generation tasks; DecisionRank adapts it to the item space, using ranked lists as an intermediate representation so that the teacher can conditionally evaluate student outputs.

Experimental Results: Small Models Surpass Large Teachers

According to the paper, DecisionRank achieves state-of-the-art performance on both general and reasoning-intensive retrieval benchmarks, including TREC DL, BEIR, and BRIGHT. On four benchmarks, item-space OPD consistently outperforms other training objectives.

Particularly striking is parameter efficiency: the 1.7B-parameter DecisionRank surpasses its 27B generative teacher on TREC DL19–23. At the 8B scale, it outperforms the generative listwise counterpart while running 24.5× faster on TREC DL and 16.7× faster on BRIGHT. These data indicate that decision models combined with on-policy distillation can achieve, and even exceed, the ranking capability of large models while maintaining low latency.

TREC DL (Text REtrieval Conference Deep Learning track) is a classic evaluation in the information retrieval field, providing large-scale document collections and human relevance annotations, with a focus on evaluating deep models' performance in passage-level reranking. BEIR (Benchmarking IR) is a heterogeneous retrieval benchmark collection covering multiple domains and tasks, used to test models' zero-shot generalization ability. BRIGHT focuses on reasoning-intensive retrieval, requiring models to understand complex logic in queries. Together, these benchmarks form the standard system for evaluating the generality and robustness of ranking models.

Why Is On-Policy Distillation More Effective?

The advantage of on-policy distillation is that the student model generates its own rankings during training, and the teacher provides feedback on these rankings. This avoids the mismatch between teacher and student distributions in offline distillation (exposure bias). The student learns from its own mistakes while the teacher provides corrective signals, making training closer to the real state at inference time.

Moreover, the item-space representation transforms the ranking problem into supervision over document scores, bypassing the complexity of sequence generation. The teacher model does not need to regenerate the entire ranked list; it only needs to conditionally evaluate the list given by the student, which is more computationally efficient. This also explains why the 8B model can achieve tens of times inference speedup—the decision model itself has no autoregressive decoding overhead, and the distillation process occurs only during training.

Exposure bias refers to the phenomenon where sequence generation models use ground-truth prefixes during training (teacher forcing) but rely on their own generated prefixes at inference, leading to error accumulation. In offline distillation, the student imitates a fixed teacher ranked list, but at inference faces rankings produced by its own distribution and is prone to performance degradation due to distribution shift. On-policy distillation lets the student learn from its own generated rankings, and the teacher provides corrective signals for these real-time states, making the training distribution closer to the inference distribution and effectively alleviating exposure bias.

Implications for Retrieval System Practice

DecisionRank's approach offers several insights for practical retrieval systems. First, the reranking stage need not rely on massive generative models; small decision models with proper distillation can achieve better results. Second, on-policy distillation can serve as a general training paradigm for other structured prediction tasks, such as recommendation and question-answer pair ranking. Finally, the item-space bridge design suggests that when teacher and student output forms are inconsistent, one can seek an intermediate representation (such as ranked lists) for alignment rather than forcibly unifying the output space.

The paper does not disclose further details on training data scale or specific hyperparameters, but the results already demonstrated are sufficient to show the method's potential. For teams focused on retrieval efficiency and effectiveness, DecisionRank provides a technical route worth tracking.

One-line Summary

DecisionRank bridges the expressive gap between generative teachers and decision-model students through item-space on-policy distillation, enabling small models to surpass large models with substantial speedup.

Paragraph Summary

DecisionRank proposes item-space On-Policy Distillation (OPD) to solve the output-form mismatch between generative LLM teachers and decision-model students. The teacher expresses ranking preferences through document ID sequences, while the student ranks based on first-token scores. The method has the student first score documents and generate a ranked list, on which the teacher provides conditional feedback, forming closed-loop training and combining offline supervision to improve generalization. It achieves SOTA on benchmarks including TREC DL, BEIR, and BRIGHT, with a 1.7B student surpassing a 27B teacher and an 8B model achieving 24.5× and 16.7× speedup on TREC DL and BRIGHT respectively. On-policy distillation alleviates exposure bias, and the item-space representation bypasses the complexity of sequence generation, providing an efficient distillation paradigm for retrieval systems.