Speed Post Overload: What Kind of Inference Performance Sharing Does the Community Need?

Every day when I open the community, I see someone showing off a screenshot of 300 t/s inference speed. These posts often come with a long list of parameters: Q1_XXXS REAP quantization, greedy decoding, TAIL_SKIP tricks. For players with only 5GB of memory, this looks like a victory—until you realize there might be a lot of water in these numbers.

The Prosperity and Concerns of Speed Posts

The progress of open-source inference tools is evident to all. From quantization schemes to expert offloading, community developers are constantly pushing hardware limits to get large models running on consumer-grade devices. But the problem is that when "speed" becomes the only thing to show off, discussions can easily go astray.

The original poster raised a sharp question: if a post only shows tokens/s but does not provide KLD (KL divergence) benchmark data, how much reference value does it really have? KLD measures the difference between the output distribution of the quantized model and the original model, and is a key metric for judging whether quantization "hurts quality." Without this data, a speed record is like a 0-100 acceleration time without braking performance—fast, yes, but whether it can safely hit the road is another matter.

KLD (KL divergence), short for Kullback-Leibler Divergence, is a metric in information theory used to measure the difference between two probability distributions. In the context of model quantization, it is usually calculated as follows: have the original full-precision model and the quantized model each output the probability distribution of the next token for the same batch of text, then compute the KL divergence between these two distributions, and finally average over the entire dataset. The lower the value, the closer the behavior of the quantized model is to the original model; it is generally believed that when the average KLD is below 0.1, the quality loss is small, and when it exceeds 0.5, there may be significant degradation. Compared with perplexity, KLD is more sensitive to changes in the shape of the distribution, so it has become a common benchmark in the quantization community (such as llama.cpp quantization discussions) for judging whether quantization hurts quality. However, KLD also has limitations: it depends on the choice of reference model and calibration dataset, and values from different experiments cannot be directly compared across studies.

Those "Clever" Acceleration Tricks

If you look closely at the configurations in these speed posts, you will find some commonalities:

• Greedy decoding: No sampling, directly selecting the token with the highest probability. This does speed things up, but output diversity is greatly reduced, and the actual experience and evaluation scenarios may be worlds apart.
• TAIL_SKIP: Skipping some computation steps. The original poster joked that they only have 5GB of memory, but if the API ignores all experts not in memory, then it can also "generate something" and claim to have broken a record.
• Extreme quantization: Quantization schemes like Q1_XXXS REAP compress the model to extremely low bits. Speed goes up, but the quality loss is often not quantitatively evaluated.

These tricks are not wrong in themselves; each has its uses in engineering. The problem is that when they are combined to produce a beautiful number without explaining the cost, they can easily mislead inexperienced readers in the community.

Greedy decoding is the simplest decoding strategy: at each step, it directly selects the token with the highest probability without random sampling, so the result is completely deterministic and reproducible, and it is slightly faster than sampling. But it has two obvious costs: first, the output lacks diversity—the same prompt always gets the same answer, and it easily falls into repetitive loops; second, it is not equivalent to "more accurate." In open-ended generation tasks, greedy decoding often performs worse than temperature sampling or beam search. Using greedy decoding in evaluation is equivalent to compressing the model's capabilities onto a single deterministic path, and the resulting scores may deviate from the actual interactive experience. In addition, greedy decoding masks the quality fluctuations caused by sampling parameters (temperature, top-p), making speed comparisons lose some of their comparability.

TAIL_SKIP is not a standardized academic term, but a colloquial name in the community for a class of "skip part of the computation" optimization tricks, commonly seen in inference implementations of mixture-of-experts (MoE) models. The idea is: when some expert weights are not in VRAM or memory, instead of waiting for them to load, directly skip the computation of these experts and complete the forward pass with the remaining experts. On devices with extremely limited memory, this can indeed make inference "run," but the cost is that the output deviates from the full model, and the deviation increases sharply as the skip ratio rises. Similar tricks also include computing only part of the layers, early exiting, and so on. They are engineering trade-offs and have their own applicable scenarios, but if performance reports do not specify the skip ratio and the corresponding quality loss, readers cannot judge the true meaning of the speed number.

What Kind of Performance Sharing Does the Community Need?

The discussion in the original post points to a core demand: transparency. A valuable performance report should at least include:

1. Complete test conditions: hardware configuration, quantization scheme, decoding parameters, prompts or datasets used.
2. Quality benchmarks: KLD data, perplexity, or downstream task accuracy, proving that the speed is not achieved at the expense of quality.
3. Reproducibility: whether others can run similar results with the same configuration.

This is not to deny the value of speed optimization. On the contrary, only by looking at "how much faster" and "how much quality was lost" together can the community judge whether a technology is truly worth adopting. Too many posts that simply chase numbers will dilute truly in-depth technical discussions.

Conclusion

The progress of open-source inference tools deserves applause, but the community's attention is a scarce resource. Rather than posting a 300 t/s screenshot every day, it would be better to spend a few extra minutes attaching KLD curves and reproduction scripts. Speed records can attract attention, but only transparent data can win trust.

What do you think? Feel free to share your views in the comments.

Background Supplement

To understand this kind of discussion, it is also necessary to distinguish several metrics that are often confused. tokens/s measures throughput, that is, the number of tokens generated per unit time. It is affected by multiple factors such as hardware, batch size, quantization precision, and context length, and talking about speed without configuration is meaningless. Perplexity is the exponent of the negative log-likelihood of a language model on test text; the lower the value, the less "surprised" the model is, but it is less sensitive to distribution shifts caused by quantization than KLD. Downstream task accuracy directly reflects the model's performance on specific tasks and is closest to the actual experience, but evaluation is costly and limited in coverage. The relationship among the three is: speed determines usability, KLD and perplexity reflect quality risk, and downstream accuracy verifies real performance. A credible performance report should provide both speed and at least one quality metric, and explain the context length and batch size used in the test; otherwise, the numbers cannot be compared.