Stop Obsessing Over tok/s! A Deep Dive into LLST, the Open-Source Local LLM Benchmarking Tool

Why "How Many tok/s" Tells You Almost Nothing

When evaluating local LLMs, the most common question is: "How many tokens per second can this model do on my GPU?" But what does that number actually tell you? It only reflects single-batch, zero-context generation speed under ideal conditions for a specific piece of hardware. It can't answer the questions that truly matter in production.

According to the analysis by this Bilibili creator, a single throughput number hides three critical blind spots: the model's real-world capabilities (Chinese language performance, runnable code generation, math and multi-step reasoning); the cost of long contexts (how much does speed degrade when context grows to 4K, 16K, or 28K? how high is the time-to-first-token latency?); and cross-run comparability (can two benchmark runs on different machines or drivers be fairly compared?).

The core argument is straightforward: fast generation doesn't mean a useful model. One model might generate tokens blazingly fast but fail miserably at instruction-following and produce buggy code; another might be a few tok/s slower but handle complex agent tasks with high quality. To break through the illusion of single-metric benchmarking, the author distilled years of accumulated testing methodology into a standardized tool: LLST (Local LLM Standard Test).

A standard toolset

LLST's Two-Layer Architecture: Capability Benchmarks + Gradient Stress Tests

LLST consists of two components that together answer a model's capability, performance degradation, time-to-first-token latency, and long-term usability — all in one run.

Layer One: 102 Fixed Capability Benchmark Questions

The capability test uses 102 fixed-prompt samples across five dimensions:

Knowledge & Reasoning: 42 questions covering broad professional knowledge and multi-step reasoning
Instruction Following: 20 questions testing strict format and constraint adherence
Advanced Math: 10 high-difficulty math and logical deduction problems
Chinese Language: 20 questions on general and domain-specific Chinese knowledge
Code Generation: 10 real-world coding tasks testing generation quality and logical correctness

Because the questions and dataset are fixed, results from different machines and drivers using the same tool are directly comparable — something ordinary benchmarks rarely achieve.

Layer Two: Deterministic Long-Context Stress Tests

The stress test uses four real-world gradient input lengths: 512, 4K, 16K, and 28K tokens, with output fixed at 512 tokens. It precisely captures time-to-first-token (TTFT), per-token decode latency, generation throughput, and quasi-throughput, producing a true long-context performance degradation curve. The entire pipeline interfaces via an OpenAI-compatible API in token mode, supporting multiple inference frameworks.

Why ordinary benchmarks are unreliable

It's worth noting that the author clarified the tool's origins: the Mac-side application is not entirely self-developed — it's a secondary build on top of the free, open-source UnaScope, with its built-in test tasks standardized into evaluation reports. Since the dataset is fixed and reports are unified, users can absolutely build their own specialized tests on top of similar tooling.

Prefill vs. Decode is the key concept for understanding long-context performance. LLM inference splits into two phases: the prefill phase processes all input tokens at once, computing and caching their attention key-value pairs (KV Cache); the decode phase then autoregressively generates output one token at a time. Time-to-first-token (TTFT) is primarily determined by prefill speed — the longer the input, the greater the prefill computation, and the longer the wait. Generation throughput (tok/s), on the other hand, reflects decode phase efficiency, which is affected by KV Cache size and memory bandwidth but is far less sensitive to input length than prefill. This is exactly why a 28K context results in a 38-second TTFT, yet subsequent generation remains steady at 25–28 tok/s: the two phases have entirely different bottlenecks and must be evaluated separately to be meaningful.

Dual 7900 XTX Running Qwen3 Flash Next: Real Test Results

The author ran Qwen3 Flash Next (the model from the previous day's deployment tutorial) on dual 7900 XTX GPUs, using Qwen3 Q3_K_XL quantization with a 32K context window, producing a quite informative scorecard.

Capability Results: Lopsided, But Competitive

Knowledge & Reasoning: 37/42 questions passed, score 88
Instruction Following: 100% composite accuracy across 20 questions
Advanced Math: Only 3 questions passed — 30% accuracy — a clear weakness for Flash-class models
Chinese Language: 90 points — strong performance
Code Generation: 80 points

Overall, this model is quite solid for instruction following, Chinese, and code — with the notable exception of advanced math — making it well-suited for everyday local use.

What many people will notice

Long Context: That 38-Second First-Token Delay Isn't as Scary as It Sounds

The most eye-catching number from the long-context tests is the 38-second TTFT at 28K input. But the author provides important context: because the stress test forces fixed large inputs and outputs, this extreme latency only appears during the initial load when the context is completely packed with ultra-long input. Everyday tasks will almost never trigger this.

The underlying reason is straightforward: prefill speed is fixed at roughly 500–600 tokens/s, so TTFT increases linearly with input length — entirely predictable. Generation speed, by contrast, remains remarkably stable: even at 28K context, decode throughput holds steady at 28, 27, 26, and 25 tok/s, barely affected by input length.

The author also compared results with the previous day's test: that run produced faster output speeds because the output was only ~100 tokens; this time, output was fixed at 512 tokens with higher load, so a slight speed reduction is expected. Even so, nearly 30 tok/s is more than adequate for real-world workflows.

Q3_K_XL quantization is a mixed-precision quantization scheme within the GGUF format. GGUF is defined by the llama.cpp community; in its naming convention, Q3 means the main model weights use 3-bit quantization, K means the k-quant algorithm is applied (which differentially allocates bit depth across layers based on weight sensitivity to minimize precision loss), and XL is a newer specification indicating that critical layers (such as attention and output layers) retain higher bit-depth for greater precision. Compared to plain Q3, Q3_K_XL slightly increases VRAM usage while significantly improving inference quality — especially on precision-sensitive tasks like instruction following and math reasoning. Choosing a quantization spec is fundamentally a three-way tradeoff between VRAM capacity, generation speed, and model capability, which is exactly why test reports must clearly state the quantization version used — the same model at different quantizations can produce significantly different capability scores.

The Value of Standardized Testing and What's Coming Next

Common questions

LLST's core value isn't about "whose peak numbers look best" — it's about using a deterministic, reproducible pipeline to lay out model capability, performance degradation curves, TTFT, and long-term usability all at once. For local users torn between "is it worth switching to a different model," this kind of comparability is far more actionable than an isolated tok/s figure.

The author revealed that the next step is to benchmark the highly anticipated Qwen3 27B using the same toolset, for a direct comparison against Flash Next. While the tests aren't lab-grade precision, the fact that tools and datasets are identical makes the output reliably informative. The tool itself is open-source; interested users can leave a comment under the video to get access.

For those who have been tinkering with local LLMs for a long time, the emergence of standardized "health check" tools like this signals that benchmarking is evolving — from showing off peak numbers to actually measuring real-world effectiveness.