Beyond tok/s: A Community Initiative for Private Quality Benchmarks

In the local large model community, tokens per second (tok/s) has become almost the sole metric for judging a model's worth. But a post by a Reddit user in r/LocalLLaMA offers sharp criticism: the endless pursuit of tok/s is "rotting our brains," while what truly matters—model quality, especially changes after quantization—is severely neglected. The post calls for a community revolution: everyone should build their own private quality benchmarks and share only the chart results, not the test sets.

The tok/s cult and the spread of Benchmaxxing

The post points out that every few hours someone in the community publishes a tok/s-related post, as if inference speed were the entirety of a model's capability. Meanwhile, "Benchmaxxing"—optimizing to game leaderboards—is becoming increasingly common in the HuggingFace open-weight model ecosystem. The author argues that this trend deviates from the original purpose of the local LLM community: exploring how models perform on real tasks, rather than competing on a single metric.

This observation has a real basis. tok/s is easy to measure, easy to compare, and naturally suited to spreading; but quality assessment is complex, subjective, and time-consuming, leading to a severe imbalance in discussion volume. When speed becomes the sole focus, key issues—quality loss from quantization, stability under different prompts, consistency over long contexts—are obscured.

Benchmaxxing is slang that has emerged in the model community in recent years, combining benchmark with -maxxing (the pursuit of extremes). It refers to targeted optimization by model developers or fine-tuners to achieve high scores on public leaderboards (such as MMLU, HumanEval, GSM8K, etc.). Common tactics include mixing samples highly similar to the test set into training data, aligning formats for specific question types, or even direct data contamination. The result is a disconnect between leaderboard scores and actual user experience: a model at the top of the leaderboard may perform mediocrely in open-ended dialogue, long-form writing, or understanding real codebases. For local deployment users, the harm of Benchmaxxing is especially insidious, because users often choose models based on public scores yet find it hard to detect the overfitting behind those scores. Private benchmarks are proposed precisely to bypass this incentive distortion—when the test set is not public, targeted optimization loses its target.

Private quality benchmarks: an actionable alternative

The author's proposal is not complicated: community members each design private benchmarks, focusing on evaluating model quality, especially performance across quantization levels. Test sets remain private, and only result charts are shared for mutual reference.

The cleverness of this design lies in preventing benchmark contamination. Once a public benchmark becomes popular, model developers may optimize specifically for it, inflating scores. Private benchmarks let each user design tests based on their actual usage scenarios (code generation, writing, translation, reasoning, etc.), making results closer to real needs. Sharing charts rather than test sets preserves the value of community collaboration—you can see the relative performance of different quantization versions on the same test, but you cannot directly game the score.

The post emphasizes that this is exactly what r/LocalLLaMA "should be like." The core value of the community lies in collective intelligence, not chasing a single number.

Why quantization quality assessment is undervalued

In local deployment scenarios, quantization is an unavoidable topic. Users need to trade off between VRAM limits and model quality, but the impact of quantization on quality is hard to measure with tok/s. A Q4 quantized model may be faster, but its error rate on complex reasoning tasks rises significantly—and this loss does not show up in speed metrics.

Private benchmarks can fill exactly this gap. Users can design a set of tests covering their own tasks, compare output quality across quantization levels, and form their own judgment. Although this assessment method is less "authoritative" than public leaderboards, it is more valuable for personal decisions.

Quantization is the technique of compressing model weights from high precision (such as FP16, BF16) to lower precision (such as 8-bit, 4-bit, or even lower), aiming to reduce VRAM usage and improve inference speed so that large models can run on consumer hardware. Common approaches include GPTQ, AWQ, and the GGUF series (such as Q4_K_M). The core cost of quantization is numerical precision loss: weights are rounded onto a coarser grid, which may cause degradation in fine semantic distinctions, multi-step reasoning, and long-tail knowledge. This degradation is task-dependent—some models are almost lossless at Q4, while others clearly drop points on complex reasoning. More troublesome is that degradation often does not manifest as reduced fluency, but as "plausible but actually wrong" outputs, which users can hardly detect without targeted testing. Therefore, quantization quality assessment requires systematic comparison covering one's own task distribution, rather than merely looking at perplexity or speed metrics.

A new paradigm for community collaboration

This initiative is essentially restructuring how the community collaborates: from a "unified leaderboard" to "distributed private testing + result sharing." It acknowledges the subjectivity and scenario dependence of model evaluation—there is no universal benchmark, but transparent result comparisons are possible.

The challenges are also obvious. Private benchmarks lack standardization, making results hard to compare across users; participants need to invest time in designing tests; sharing charts may spark new "chart comparisons." But the post author clearly believes these costs are far smaller than the cost of continuing to obsess over tok/s.

For local LLM enthusiasts, this proposal deserves serious consideration. Next time you see a tok/s post, ask yourself: how good is this model's quality on the tasks I care about?