Tokenizer Fertility

A metric of tokenizer efficiency expressing the average number of tokens produced per byte. Lower values indicate shorter sequences and higher inference throughput, but a large vocabulary inflates the embedding matrix and consumes more FLOPs.