[KongchangAI]
· 2 min read· 1,096 words

COPD: Distilling Black-Box Teacher Capabilities through Natural Language Communication to Tackle Long-Horizon Tasks

COPD: Distilling Black-Box Teacher Capabilities through Natural Language Communication to Tackle Long-Horizon Tasks

COPD distills black-box teacher capabilities via natural language communication, greatly surpassing black-box OPD baselines on long-horizon tasks.

To address traditional on-policy distillation (OPD)'s reliance on teacher logits, which makes it inapplicable to purely text-based black-box models, the paper proposes Communicative On-Policy Distillation (COPD). The student model autonomously decides when to consult the black-box teacher during long-horizon tasks and expresses needs via open-ended natural language requests, without predefined templates or intervention plans. Successful teacher-student interactions are retained as privileged references to supervise advisor-free student rollouts through on-policy self-distillation, jointly optimized with task-reward reinforcement learning. On CAMG-II, COPD achieves 2.46x the gain of the strongest black-box OPD baseline, and the learned capabilities generalize to Terminal-Bench 2.1 and SWE-Bench Pro V2, improving success rates even without advisor access—indicating the student truly internalizes problem-solving ability rather than memorizing interactions.

Improving the capabilities of language model agents on long-horizon tasks has long been an important direction in AI research. Traditional on-policy distillation (OPD) relies on the teacher model's logit outputs, but purely text-based black-box models do not expose this information. Recently, a paper titled "Communicative On-Policy Distillation: Distilling Black-Box Teacher Capabilities through Communication for Long-Horizon Tasks" proposed a new method—Communicative On-Policy Distillation (COPD)—which uses natural language communication as a medium to distill the capabilities of stronger models into smaller student models.

COPD paper title screenshot

Limitations of Traditional OPD and the Proposal of COPD

On-policy distillation (OPD) is a powerful paradigm for improving the performance of language model agents on long-horizon tasks. However, its conventional form relies on teacher logits, and black-box models that only provide text output do not expose logits. This limits the application of OPD in black-box teacher scenarios. The core innovation of COPD is that it uses natural language communication as the distillation medium, allowing the student model to autonomously decide when to consult the black-box teacher during task execution and to express its needs as open-ended natural language requests, without predefined question templates or intervention plans.

According to the paper, COPD works as follows: while executing a task, the student model can proactively initiate consultation with the black-box teacher and freely formulate natural language requests. Successful teacher-student interactions are retained as privileged references, which are used to supervise new advisor-free student rollouts through on-policy self-distillation, combined with task-reward reinforcement learning for joint optimization. This approach does not rely on the teacher model's internal state and transmits capabilities solely through the content of communication.

On-policy distillation (On-Policy Distillation)—here "on-policy" means that the student model samples trajectories under its own policy distribution, and the teacher model then provides supervision signals for these trajectories, rather than using a fixed offline dataset. The benefit of this is alleviating the mismatch between training and inference distributions (exposure bias). Traditional OPD typically uses the teacher model's logits as soft labels and minimizes KL divergence, which is essentially dense token-by-token supervision; however, this approach requires access to the teacher model's output distribution and is infeasible for black-box models that only provide a text API. COPD replaces the supervision signal from logits with natural language communication content, which is equivalent to transforming dense distribution matching into sparse high-level semantic guidance—this is also the precondition for its viability in black-box scenarios.

Experimental Results: Significantly Outperforming Black-Box OPD Baselines

The paper evaluated COPD's performance on the CAMG-II task. The results show that COPD's gain reaches 2.46 times that of the strongest black-box OPD baseline. This data indicates that distillation through communication can more effectively transmit teacher capabilities. In addition, the capabilities learned by COPD also generalized to Terminal-Bench 2.1 and SWE-Bench Pro V2 tasks, improving task success rates without advisor access. This validates the generalizability of communicative distillation—the student model is not simply memorizing interactions but truly internalizing the ability to solve problems.

COPD experimental results illustration

Analysis of COPD's Key Mechanisms

COPD's success stems from several key designs. First, the student model autonomously decides when to consult, which simulates the behavior of agents seeking help on demand in real scenarios and avoids the rigidity of fixed intervention plans. Second, open-ended natural language requests allow the student to express confusion or needs flexibly, and the teacher responds in natural language—this interaction is closer to the "question-answer" pattern in human learning. Third, successful interactions are used as privileged references to supervise new rollouts through on-policy self-distillation, meaning the student model can reproduce successful behavior even without the teacher. Finally, combining with task-reward reinforcement learning ensures that the distillation process remains aligned with task objectives.

This method avoids the bottleneck of relying on teacher logits and opens a new path for knowledge distillation of black-box models. Especially for long-horizon tasks, where agents need multi-step decision-making and complex reasoning, COPD obtains high-level guidance through communication, which is more efficient than pure logit matching.

Privileged references and on-policy self-distillation are key to understanding COPD's training pipeline. The term "privileged" means that these successful interaction trajectories contain information that the student could not have obtained on its own at execution time (i.e., the teacher's answers), so they can only serve as training-time references and cannot be used as inference inputs. On-policy self-distillation works by retaining these successful trajectories containing teacher guidance as supervision targets, then having the student model resample (rollout) without the advisor and minimizing the difference between the new rollout and the privileged references. In this way, the student learns "what I should have done if I had received this advice at the time," thereby internalizing external guidance into its own policy. Unlike behavior cloning, the supervision distribution of self-distillation comes from the student's own current policy, so it better fits its capability boundary and avoids learning behaviors it cannot reproduce.

Generalization Capability and Future Outlook

The model trained by COPD on CAMG-II can generalize to Terminal-Bench 2.1 and SWE-Bench Pro V2. These two tasks involve terminal operations and software engineering and differ considerably from the training task. This shows that the capabilities learned through communicative distillation have cross-task transferability. The paper notes that the student model can still improve task success rates without advisor access, indicating that it has learned to solve problems independently.

This research provides a new idea for black-box model distillation: communication can serve as a bridge for knowledge transfer, not just logits. In the future, COPD may be extended to more modalities and more complex tasks, advancing the autonomy of language model agents on long-horizon tasks.

The CAMG-II, Terminal-Bench 2.1, and SWE-Bench Pro V2 mentioned in the article are all evaluation benchmarks for long-horizon agent tasks. CAMG-II focuses on goal achievement under multi-turn interaction and long-horizon decision-making; Terminal-Bench 2.1 assesses the agent's multi-step ability to execute commands and troubleshoot errors in real terminal environments; SWE-Bench Pro V2 is based on real software engineering issues and requires the agent to complete code localization, modification, and verification. A common feature of all three is long task spans, many intermediate states, and cumulative amplification of single-step errors, so merely imitating local outputs is unlikely to work. This also explains why COPD, which obtains high-level guidance through communication, demonstrates transfer advantages on these benchmarks.

Conclusion

COPD distills black-box teacher capabilities through natural language communication, addressing the pain point of traditional OPD's reliance on logits. The 2.46x gain on CAMG-II and cross-task generalization demonstrate the potential of communicative distillation. For AI agent research, this suggests that teaching models to "ask questions" may be more effective than directly imitating outputs.

#COPD#On-Policy Distillation#Black-Box Teacher#Natural Language Communication#Long-Horizon Tasks#Language Model Agents
Share:

Related articles