mmBERT
A multilingual BERT model used for text classification and natural language understanding tasks.
Timeline (last 90 days)
Laya-multilingual is a 322M-parameter, Apache-2.0-licensed model built on mmBERT that outputs probability judgments for chat-card questions.
Using a quantization combination of INT8 weights with INT4 embedding layers in ONNX, the F1 score difference is only approximately 0.005 compared to more conservatively quantized versions.
mmBERT supports an input window of up to 8192 tokens.
Cascade/early-exit architectures use cheap lightweight classifiers that operate in the same latent space as mmBERT to make easy decisions first, routing only uncertain cases to the full model.
The goal of the cheap classifier is not to replace mmBERT, but to identify cases where running the full model would not change the answer.
When performing text classification on CPU, a chunk size of 256 or 512 tokens generally outperforms an 8192 large window.
Both Wolf Defender and Wolf Defender Small were retrained from fresh mmBERT checkpoints with a focus on hard negative samples.
All Facts (7)
Laya-multilingual is a 322M-parameter, Apache-2.0-licensed model built on mmBERT that outputs probability judgments for chat-card questions.
50%UnverifiedUsing a quantization combination of INT8 weights with INT4 embedding layers in ONNX, the F1 score difference is only approximately 0.005 compared to more conservatively quantized versions.
50%UnverifiedmmBERT supports an input window of up to 8192 tokens.
50%UnverifiedCascade/early-exit architectures use cheap lightweight classifiers that operate in the same latent space as mmBERT to make easy decisions first, routing only uncertain cases to the full model.
50%UnverifiedThe goal of the cheap classifier is not to replace mmBERT, but to identify cases where running the full model would not change the answer.
50%UnverifiedWhen performing text classification on CPU, a chunk size of 256 or 512 tokens generally outperforms an 8192 large window.
50%UnverifiedBoth Wolf Defender and Wolf Defender Small were retrained from fresh mmBERT checkpoints with a focus on hard negative samples.
50%