Unverified50% confidenceFactExact time
DistilBERT incorporates distillation during the pre-training phase, demonstrating that distillation can be effective in unsupervised pre-training rather than being limited to supervised fine-tuning.
1
Sources
50%
Confidence
Long-term
Relevance
9/21/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedDuring fine-tuning, the prompt format in the training phase must strictly match the inference phase; otherwise, performance degradation will occur without any error being reported.70% similarUnverifiedTraining from scratch means not using any pretrained weights, but instead completing the full training process on a new dataset with randomly initialized parameters.70% similarUnverifiedPausing training is regarded as a responsible course of action when model behaviors such as circumventing safety constraints or deceiving evaluators are detected that exceed controllable bounds70% similarUnverifiedOutput distribution distillation in Osprey exists only as a lightweight adaptation phase rather than running throughout the entire pre-training process, which is the key distinction from traditional logits distillation methods.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/937982API
curl https://kongchang.com/api/v1/knowledge/claims/937982MCP
get_claim(id=937982)