#tokenizer
2 related articles
·1 min
India's Language Token Costs Are Staggering: Hindi at 4.4x English, Malayalam at 7.3x
Testing shows Hindi consumes 4.44x more tokens per sentence than English, Malayalam 7.27x. Tokenizer choice (Gemma-3 just 1.31x) matters more than serving optimization for GPU concurrency costs; romanization isn't a universal fix.
Read more →
·2 min
BERT Tokenization Bottleneck Moves Upstream: Normalization Swallows 88% of Encoding Time
BERT tokenization profiling reveals normalization consumes 88% of English encoding time, WordPiece only 8%. Inside Simon Brandeis's optimization: SIMD skips ASCII, page cache speeds non-ASCII scripts, and why to measure tokenization separately in RAG and embedding services.
Read more →