How Fast Is 10 Tokens Per Second from an LLM? An Open-Source Tool for Visualizing Token Generation Speed

When we see a large language model claiming "30 tokens per second" or "100 tokens per second," what do those numbers actually mean? Developer Mike Veerman built a simple yet practical tool that lets you intuitively experience what different token generation speeds actually feel like.

The Real Experience Behind TPS Numbers

In the LLM space, token generation speed (tokens per second, TPS) is one of the core metrics for measuring model inference performance. Major model providers often highlight TPS as a key selling point when launching products. But for most users, the difference between "30 tokens per second" and "100 tokens per second" is hard to grasp from numbers alone.

To truly understand TPS, you first need to understand what tokens actually are. A token is the basic unit that large language models use to process text. Unlike our intuitive notion of "characters" or "words," tokens are subword units split by the model's tokenizer. In English, one token corresponds to roughly 0.75 words; in Chinese, due to higher character density, one Chinese character typically maps to 1-2 tokens. This means when we say "30 tokens per second," the actual amount of text varies by language—approximately 22 English words/second, or about 15-30 Chinese characters/second. It's worth noting that TPS measurement typically involves two phases: Time to First Token (TTFT) and the subsequent token generation throughput. The former determines the perceived "response speed," while the latter determines the "generation speed"—and they affect user experience in fundamentally different ways.

The HTML application Mike Veerman developed is designed precisely to bridge this cognitive gap. It simulates text output from 5 tokens per second up to 800 tokens per second, letting you watch text gradually appear—just like in a real AI conversation—at different speeds.

How Different Token Speed Tiers Actually Feel

The best frame of reference for understanding TPS is human natural reading speed. Research shows that the average adult silent reading speed is about 200-400 English words per minute, which translates to roughly 4-9 tokens per second; fast readers can exceed 700 words per minute, corresponding to about 15 tokens per second. Neuroscience research also shows that humans have a perceptual window of approximately 250 milliseconds for visual information processing—below 4 TPS, the brain clearly perceives "pauses" and generates a sense of anxiety; above approximately 15 TPS, the brain begins processing information in "paragraphs" rather than "words," and the experiential improvement from speed increases gradually plateaus. Based on this cognitive foundation, we can divide token generation speeds into several experience tiers:

5-10 TPS: Noticeable Waiting

At 5 to 10 tokens per second, text appears word by word in a "popping" fashion, and users can clearly perceive the generation of each word. This speed is common in early locally deployed models or CPU inference with larger parameter models. It's usable, but the waiting experience during long text generation is far from ideal.

30-50 TPS: Smooth Reading Rhythm

This range represents the typical speed for many cloud API services today. Text generation roughly matches the speed of normal human reading, allowing users to read as they wait, creating a relatively comfortable experience. Most users won't feel noticeable anxiety at this speed.

100-200 TPS: Near-Instant Response

When speed reaches 100+ tokens per second, text practically "pours out," and users' reading speed can no longer keep up with generation speed. For most interactive scenarios, this speed is more than sufficient.

400-800 TPS: Speed Surplus

Beyond 400 TPS, the human eye can barely distinguish the difference from lower speeds—text appears to materialize almost instantly. While this extreme speed does create "perceptual surplus" in real-time conversations, it delivers significant value in AI Agent workflows and batch processing scenarios. Modern Agent systems often require models to complete dozens or even hundreds of LLM calls within a single task. Consider an Agent task requiring 20 LLM calls, each generating 200 tokens: at 30 TPS it takes approximately 133 seconds to complete, while at 300 TPS it takes only 13 seconds. This 10x efficiency difference carries significant commercial value in enterprise applications like automated code review and batch document processing.

The Hardware Foundation of Inference Speed

LLM inference speed is fundamentally constrained by hardware architecture. Traditional GPUs (such as NVIDIA A100/H100) face a "memory bandwidth bottleneck" during inference—model weights need to be repeatedly read from VRAM with each token generation, making bandwidth the key factor limiting TPS. Groq takes a completely different technical approach: its proprietary LPU (Language Processing Unit) statically compiles model weights into the chip's SRAM, eliminating dynamic memory access latency and achieving extremely high throughput of hundreds of TPS. Additionally, quantization is another common technique for boosting TPS—compressing model weights from FP16 to INT8 or INT4 can significantly reduce memory usage at the cost of minor precision loss, enabling consumer GPUs (like the RTX 4090) to achieve practical speeds of 30-80 TPS for local deployment.

Why This Token Speed Testing Tool Deserves Attention

The value of this tool goes beyond satisfying curiosity—it helps developers and users make more rational technical decisions:

A reference point when choosing models. When you're torn between Groq's ultra-fast inference and other providers, try using this tool to feel the speed difference first. You might discover that for your use case, the experiential gap between 30 TPS and 300 TPS isn't as large as you imagined.

Prioritizing performance optimization. If your application already achieves 50 TPS, investing further resources to push speed to 200 TPS might yield far less marginal user experience improvement than optimizing Time to First Token (TTFT).

Managing expectations for local deployment. When running open-source models on consumer GPUs locally, understanding what different speeds actually feel like helps you make informed choices about model size and quantization schemes.

The Elegant Simplicity of the Implementation

It's worth mentioning that this tool is a single pure HTML file, with its source code fully open source. No complex framework dependencies, no backend services—just open it in a browser and use it. This philosophy of "solving a specific problem in the simplest possible way" is itself an engineering approach worth learning from.

In today's increasingly fierce competition over LLM inference performance, this little tool reminds us: numbers are just numbers—what truly matters is the user's actual experience. Rather than blindly chasing higher TPS, it's better to first figure out—for your specific scenario, how fast is "fast enough"?