GoiBERT: Building a Compact Vietnamese Model with Two-Stage Knowledge Distillation

Vietnamese pretrained encoders have made significant progress in recent years, but the strongest-performing models tend to be large in parameter count, use multilingual vocabularies, or depend on external tokenization tools, keeping deployment costs high. A new paper proposes GoiBERT to address this tension: it comes in two compact versions, 26M and 49M parameters, processes raw text directly without tokenization preprocessing, yet approaches or even surpasses larger baseline models on multiple benchmarks.

Two-Stage Distillation: From General Representations to Task-Level Guidance

The core design of GoiBERT is "dual-teacher" distillation. In the first stage, a Vietnamese encoder with a shared vocabulary acts as the teacher, imparting general language representations; in the second stage, a much larger teacher model provides task-level guidance through soft labels—crucially, these soft labels are generated without the need for manual annotation of the teacher's own training data.

This two-stage arrangement allows the small student model to both absorb language-level knowledge and align with downstream task behavior. Compared with single-stage distillation, it decouples "learning the language" from "learning the task," preventing the small model from being squeezed by two types of signals simultaneously under limited capacity.

Knowledge Distillation is a model compression technique: a small "student" model imitates the output distribution of a large "teacher" model rather than learning only from ground-truth labels. The probability distribution output by the teacher model is called "soft labels," which encode information about the relative similarity between classes and provide richer supervision signals than one-hot labels. In NLP, distillation can be divided into representation-level (aligning hidden states) and task-level (aligning output logits). GoiBERT's two-stage design corresponds to these two types respectively: the first stage aligns language representations, and the second stage aligns task behavior, thereby transferring knowledge more efficiently under capacity constraints.

Performance: Half the Parameters, Twice the Throughput

The paper evaluates on eight benchmarks, covering text classification, aspect-based sentiment analysis, natural language inference, and named entity recognition. GoiBERT-large achieves the highest average score among non-teacher models, with fewer than half the parameters of the strongest compact baseline while delivering twice its throughput. GoiBERT-base is even more aggressive: it retains 99% of that baseline's performance at one-quarter the size.

The significance of these numbers is that they show the deployment barrier for Vietnamese NLP can be significantly lowered. For scenarios requiring localized inference, edge deployment, or cost sensitivity, the benefits of halving parameters and doubling speed are quite direct.

Throughput refers to the number of samples or tokens a model can process per unit time. In inference scenarios, it is jointly affected by parameter count, sequence length, batch size, and hardware parallelism. Compact models, due to their smaller parameter count and lower memory footprint, can typically support larger batch sizes on the same hardware and reduce memory bandwidth bottlenecks, thereby significantly improving throughput. In addition, not relying on external tokenization tools means eliminating the computation and pipeline latency of tokenization preprocessing, which also directly contributes to end-to-end inference throughput. Therefore, the "twice the throughput" reported in the paper is the result of both model size and the tokenization-free design.

Compression Is Not Lossless: Sentiment and Classification Come at Almost No Cost, but Reasoning and Entity Recognition Still Suffer

The authors' analysis does not shy away from the shortcomings. Compression is almost lossless on sentiment analysis and text classification tasks, but still incurs a clear cost on natural language inference and named entity recognition. This divergence points to a clear direction for follow-up work: reasoning-type tasks involve more complex logical composition, and entity recognition depends on fine-grained boundary judgments—both place higher demands on model capacity, and distillation alone may not be enough.

For engineering practice, this means GoiBERT is better suited as the preferred compact solution for classification and sentiment tasks; if the target task is NER or NLI, the trade-off between accuracy and cost needs to be re-evaluated during model selection.

Natural Language Inference (NLI) requires the model to determine the logical relationship between two sentences (entailment, contradiction, or neutrality), involving deep semantic composition and reasoning abilities, and places higher demands on model capacity than shallow classification. Named Entity Recognition (NER) is a sequence labeling task that requires the model to make fine-grained boundary judgments for each token, relying on rich contextual representations. These two types of tasks are often harder for small models to fully inherit in knowledge distillation, because although soft labels can convey relative information between classes, they struggle to fully encode the internal representations needed for long-range dependencies and fine-grained positional discrimination, which is consistent with the accuracy losses observed in the paper.

Open Source and Reproducibility

The code and models are open-sourced on GitHub (Nhantran65/GoiBERT). For the Vietnamese community and small teams, a compact encoder that does not depend on external tokenization and directly consumes raw text can save the maintenance burden of preprocessing pipelines and make it easier to embed the model into existing services.

From a broader perspective, GoiBERT represents a pragmatic path: rather than chasing parameter-scale competition, it uses distillation to compress the capabilities of large models into deployable sizes, and honestly reports which tasks incur high compression costs. This "lossy compression map" is more valuable as a reference for practical model selection than a single average score.

One-line Summary

GoiBERT achieves tokenization-free deployment of Vietnamese encoders with 26M/49M compact parameters, approaching large models on multiple benchmarks.

Paragraph Summary

GoiBERT is a family of compact Vietnamese pretrained encoders, available in 26M and 49M versions, that process raw text directly without external tokenization. Its core is two-stage knowledge distillation: first, a shared-vocabulary teacher imparts general language representations; then a large model provides task-level guidance through unlabeled soft labels, decoupling "learning the language" from "learning the task." On eight benchmarks, GoiBERT-large achieves the highest average score among non-teacher models, with fewer than half the parameters of the strongest compact baseline and double the throughput; GoiBERT-base retains 99% of performance at one-quarter the size. Compression is nearly lossless on sentiment analysis and classification tasks, but NLI and NER still show notable accuracy losses. The models are open-sourced, lowering the deployment barrier for Vietnamese NLP.