Distilled Model
A lightweight model obtained by transferring the capabilities of a large teacher model to a small student model through knowledge distillation; inference cost is low, but open-domain reasoning capability is limited.
Timeline (last 90 days)
Distilled models are sufficient to cover most traffic in routine customer-service scenarios, but degrade noticeably when open-domain reasoning or long-chain planning is required.
In a real-time code review tool built for a client, using the largest and smartest model gave an average response of 8.7 seconds; after switching to a smaller distilled model, the response dropped to 2.3 seconds, with review quality falling by only about 5%.
The model names labeled in API documentation (such as gpt-4, claude-3) may correspond to cheaper distilled versions, quantized versions, or even previous-generation models.
All Facts (4)
AI服务商在算力受限时可能采用减少beam search宽度、压缩KV Cache、动态切换蒸馏模型等手段降低资源消耗
85%UnverifiedDistilled models are sufficient to cover most traffic in routine customer-service scenarios, but degrade noticeably when open-domain reasoning or long-chain planning is required.
50%UnverifiedThe model names labeled in API documentation (such as gpt-4, claude-3) may correspond to cheaper distilled versions, quantized versions, or even previous-generation models.
50%UnverifiedIn a real-time code review tool built for a client, using the largest and smartest model gave an average response of 8.7 seconds; after switching to a smaller distilled model, the response dropped to 2.3 seconds, with review quality falling by only about 5%.
50%