Black-Box Distillation

Traditional distillation usually requires access to the teacher model's full output probabilities (logits), but in black-box API scenarios attackers can only obtain the final text, so it has evolved into "black-box distillation"—instruction fine-tuning on large numbers of input-output pairs.

A white-box teacher allows access to weights and logits, supporting soft-label distillation; a black-box teacher can only provide text outputs, allowing only hard-label offline distillation.

Black-box distillation typically requires millions of API calls to achieve ideal results

The effectiveness of black-box distillation methods depends on the diversity and coverage of query data, often requiring millions of API calls to achieve ideal results.

In traditional white-box distillation, the student model can directly access the complete probability distribution (logits) of the teacher model's output layer, whereas commercial large language model APIs typically return only the final generated text or top-k token probabilities.

Black-box distillation collects high-quality input-output pairs by making large numbers of calls to a target model's API and then trains its own model using supervised learning, often requiring millions of API calls.