DistilBERT
A lightweight, distilled version of BERT that reduces the original model's parameters by approximately 60% while achieving roughly 60% faster inference. It is commonly deployed for NLP tasks such as intent classification in resource-constrained environments.
Core Facts
Timeline (last 90 days)
Compression techniques such as model distillation and quantization (INT8/FP16) can reduce the parameter count of lightweight neural networks (such as DistilBERT) to below tens of millions, enabling them to achieve acceptable inference speeds on CPUs.
A study published on arXiv (arXiv:2609.28703v1) proposes a hate speech detection framework that balances multi-level classification with interpretability, with its core consisting of three components: DistilBERT, bidirectional LSTM, and an attention mechanism.
Fine-tuned DistilBERT achieves F1=0.9993 on in-distribution testing, but loses 12.9 mean average precision points under topic shift, a greater degradation than the linear model's 5.2 points.
DistilBERT was proposed by Hugging Face in 2019, training a student model of only 66 million parameters using the original BERT-base (110 million parameters) as the teacher.
DistilBERT incorporates distillation during the pre-training phase, demonstrating that distillation can be effective in unsupervised pre-training rather than being limited to supervised fine-tuning.
DistilBERT retains approximately 97% of the original model's performance on the GLUE benchmark, with inference speed improved by approximately 60% and model size reduced by 40%.
DistilBERT is a lightweight model obtained by the Hugging Face team through knowledge distillation from BERT, retaining only 6 Transformer layers, reducing parameter count by 40%, and improving inference speed by 60%
DistilBERT compressed BERT-base from 110M parameters to 66M, improving inference speed by 60% while retaining 97% of performance.
DistilBERT retains 97% of BERT's language understanding capability while increasing inference speed by 60% and reducing model size by 40%.
DistilBERT's distillation process uses a triple loss function: distillation loss, masked language model loss, and cosine embedding loss.
3 more timeline events
All Facts (15)
知识蒸馏由Hinton等人在2015年提出，核心思想是让小型学生模型模仿大型教师模型的输出分布
80%VerifiedDistilBERT reduces BERT's parameter count by 40% through distillation, increases inference speed by 60%, and incurs only approximately 3% performance loss
75%UnverifiedDistilBERT compresses BERT's size by 40% while retaining 97% of its performance.
60%UnverifiedCompression techniques such as model distillation and quantization (INT8/FP16) can reduce the parameter count of lightweight neural networks (such as DistilBERT) to below tens of millions, enabling them to achieve acceptable inference speeds on CPUs.
50%UnverifiedA study published on arXiv (arXiv:2609.28703v1) proposes a hate speech detection framework that balances multi-level classification with interpretability, with its core consisting of three components: DistilBERT, bidirectional LSTM, and an attention mechanism.
50%UnverifiedFine-tuned DistilBERT achieves F1=0.9993 on in-distribution testing, but loses 12.9 mean average precision points under topic shift, a greater degradation than the linear model's 5.2 points.
50%UnverifiedDistilBERT retains approximately 97% of the original model's performance on the GLUE benchmark, with inference speed improved by approximately 60% and model size reduced by 40%.
50%UnverifiedDistilBERT incorporates distillation during the pre-training phase, demonstrating that distillation can be effective in unsupervised pre-training rather than being limited to supervised fine-tuning.
50%UnverifiedDistilBERT was proposed by Hugging Face in 2019, training a student model of only 66 million parameters using the original BERT-base (110 million parameters) as the teacher.
50%UnverifiedDistilBERT is a lightweight model obtained by the Hugging Face team through knowledge distillation from BERT, retaining only 6 Transformer layers, reducing parameter count by 40%, and improving inference speed by 60%
50%UnverifiedDistilBERT compressed BERT-base from 110M parameters to 66M, improving inference speed by 60% while retaining 97% of performance.
50%UnverifiedDistilBERT was proposed by Sanh et al. at Hugging Face in 2019, and is a lightweight model compressed from BERT-base to 6 layers and 66 million parameters through knowledge distillation.
50%UnverifiedDistilBERT's distillation process uses a triple loss function: distillation loss, masked language model loss, and cosine embedding loss.
50%UnverifiedDistilBERT retains 97% of BERT's language understanding capability while increasing inference speed by 60% and reducing model size by 40%.
50%UnverifiedDRET-enhanced DistilBERT, with only 66 million parameters, achieves performance on core metrics such as balanced accuracy, recall, and ROC-AUC that is comparable to biomedical-specific models an order of magnitude larger
50%