BertNormalizer
A BERT text normalization component in the Hugging Face tokenizers library. It performs preprocessing steps such as text cleanup, Unicode decomposition, accent stripping, and lowercasing.
Timeline (last 90 days)
In a warm English encoding with bert-base-uncased, Normalization consumed 88% of the encoding time, while WordPiece accounted for only 8%.
The preprocessing that BertNormalizer is responsible for includes: cleaning the text, inserting spaces around Chinese characters, performing NFD decomposition on Unicode characters, removing non-spacing marks (removing accents), and lowercasing.
The ordered table in BertNormalizer used to determine whether a code point is a non-spacing mark (Mn) contains 1567 characters.
In the original implementation, even a pure ASCII letter (such as s) goes through the complete Unicode property lookup path: first decode, then consult the 1567-character table, then pass into the normalization pipeline.
Any change to the normalization implementation must not be judged on throughput alone; the token output must also be verified to be exactly identical.
All Facts (5)
In a warm English encoding with bert-base-uncased, Normalization consumed 88% of the encoding time, while WordPiece accounted for only 8%.
50%UnverifiedThe preprocessing that BertNormalizer is responsible for includes: cleaning the text, inserting spaces around Chinese characters, performing NFD decomposition on Unicode characters, removing non-spacing marks (removing accents), and lowercasing.
50%UnverifiedThe ordered table in BertNormalizer used to determine whether a code point is a non-spacing mark (Mn) contains 1567 characters.
50%UnverifiedIn the original implementation, even a pure ASCII letter (such as s) goes through the complete Unicode property lookup path: first decode, then consult the 1567-character table, then pass into the normalization pipeline.
50%UnverifiedAny change to the normalization implementation must not be judged on throughput alone; the token output must also be verified to be exactly identical.
50%