Meta Tests Small Models: Skipping Tokenization to Read Raw Bytes Pays Off Later

Large models follow a standard pipeline for processing text: first split the text into tokens, then feed them to the model. A new paper from Meta tries a different path: let the model skip the tokenizer and read raw bytes directly—in other words, read it letter by letter. The idea isn't new, but the paper uses distillation experiments to offer an interesting observation: byte models start off worse, but their training curves don't flatten out early the way token models do.

How the experiments were done

According to discussion of the paper (arXiv:2609.12303) on Reddit, the research team trained a batch of small models with about 1B parameters, some trained on tokens and some on bytes. To help the byte models avoid detours, they used distillation: a token model "teaches" the byte model, with Llama 3 8B as the teacher model.

The tricky part is that the teacher thinks in tokens while the student reads in bytes, so their granularities don't line up. The team needed to design a clean correspondence that translates the teacher's outputs into supervision signals the byte model can understand. This step is the key prerequisite that made the work feasible.

The difference between tokenizers and byte-level modeling: mainstream large models rely on tokenizers to split text into subword units (such as tokens produced by the BPE algorithm), with each token corresponding to an ID in the vocabulary. The vocabulary usually contains tens of thousands to hundreds of thousands of entries, and the model makes predictions over discrete token sequences. Byte-level models skip the tokenizer entirely and directly use the raw UTF-8 encoded bytes (0-255) as the input sequence; a single letter may correspond to one or more bytes. The advantages are a tiny vocabulary (only 256), no out-of-vocabulary problem, and more unified handling across languages and symbols; the cost is a significant increase in sequence length, because one token corresponds on average to 4-5 bytes, so the model must process longer sequences, and computation and inference costs rise accordingly.

Two conclusions from the training curves

In the early stage, token models win hands down. Under the same amount of training, byte models lag clearly behind.

But as training continues, things change: the token model's performance begins to flatten, while the byte model keeps improving. Extrapolating the two curves, the authors predict that the byte model will eventually surpass Llama 3.2 1B and Gemma 3 1B by a few percentage points. Another data point is that the byte model needs only about one-sixth of the text volume required by the token model to reach the same level.

Here it's necessary to distinguish fact from speculation. Surpassing Llama and Gemma is the result of curve extrapolation; in actual training, the token model is still ahead. The paper's value lies in showing a trend, not in delivering a victory that has already been won.

Knowledge distillation: a training technique in which a small "student" model imitates the outputs of a large "teacher" model. The teacher model provides soft labels (probability distributions) for inputs, and the student model learns by minimizing the difference from the teacher's distribution rather than relying only on hard labels (ground-truth answers). Distillation can transfer dark knowledge such as the similarities between classes that the teacher has learned, and it is often used to improve small models or accelerate training. In this paper, the teacher is Llama 3 8B (token-based) and the student is a byte model; because their input granularities differ, an alignment mapping between token and byte sequences must be designed to convert the teacher's token-level outputs into supervision signals usable by the byte model.

Costs and limitations

Byte models are not without cost. One token roughly corresponds to 4 to 5 letters, so processing byte by byte means running about 5 times as many steps for the same text, making inference slower. Data usage goes down, but computation goes up.

Another point that is easy to misread: tokens have not been eliminated. The teacher model is still token-based, and the distillation pipeline itself depends on the token model to provide high-quality signals. The byte route currently looks more like a supplement than a replacement.

In terms of scope, the experiments cover only small models on the order of 1B. Whether byte models retain the same advantage at larger scales is a question the paper does not answer, and that is precisely the key question for judging their long-term value.

The limits of curve extrapolation: the paper extends training curves into future steps and predicts that byte models will eventually overtake. But extrapolation assumes the trend stays unchanged, while in actual training factors such as learning-rate scheduling, data distribution, and model capacity can all bend or saturate the curve. Extrapolated results should therefore be treated as hypotheses, not verified conclusions. Readers should distinguish between "observed trends" and "actually delivered performance": within the trained steps, token models are still ahead.

Why this matters

For small models running on phones and laptops, squeezing out a bit more quality at the same parameter count has real significance. If the byte route holds up on edge devices, models could achieve better performance without increasing size.

There is also a side benefit: once the huge vocabulary is gone, some "low-level errors" caused by tokenization may disappear, such as being unable to count how many letters are in a word. Such problems have long existed in token models, and their root cause is vocabulary splitting.

Someone in the Reddit discussion asked: is there any work making byte models run as fast as token models? This points to the core obstacle to deploying the route—speed. Until inference efficiency catches up, byte models are better suited as a research direction and a fallback for specific scenarios rather than a general-purpose solution.

Summary

What this paper offers is not the conclusion that "tokens are dead," but a curve worth continuing to watch. Slow start, strong late momentum, high data efficiency, and slower inference—these four characteristics hold at the same time. Whether it can scale from 1B small models to larger scales, and whether the speed problem can be solved, will determine whether the byte route becomes a supplement beyond the mainstream or remains merely an interesting footnote in a paper.

One-line Summary

A Meta paper uses distillation to compare byte and token models, finding that byte models start slower but gain momentum later, with higher data efficiency but slower inference.

Paragraph Summary

A new Meta paper tries letting large models skip the tokenizer and read raw bytes directly, using Llama 3 8B as a teacher model to distill and compare small models with about 1B parameters. Experiments show that token models lead clearly early on, but after the token model curve flattens, the byte model keeps improving; extrapolation predicts it will eventually surpass Llama 3.2 1B and Gemma 3 1B by a few percentage points, and it needs only one-sixth of the text volume to reach the same level. The cost is that byte-by-byte processing increases inference steps by about 5 times. The authors stress that the overtaking is based on extrapolation rather than actual results, the experiments cover only 1B small models, tokens have not been eliminated, and the byte route currently looks more like a supplementary direction.