What Does 318 tps Inference Speed Mean? A Deep Dive into the LLM Performance Race

A Single Tweet That Shook the Industry

Recently, an AI practitioner posted on Twitter: "318 tps.....crazy..... how can you make this happen... we have to work harder even more..." This seemingly simple tweet reflects the fact that the LLM inference performance race has entered a white-hot phase.

tps (tokens per second) is the core metric for measuring the inference speed of large language models. First, let's understand what a token is — a token is the basic unit that LLMs use to process text, and it's not simply equivalent to a single character or word. In English, one token corresponds to roughly 4 characters or 0.75 words; in Chinese, a single character is typically encoded as 1–2 tokens. So 318 tps translates to approximately 240 words per second in English, or about 160–300 Chinese characters per second. For reference, normal human reading speed is around 200–300 English words per minute, or 3–5 words per second — meaning 318 tps generates text nearly 50 times faster than a human can read.

318 tps means the model can generate over 300 tokens per second. For everyday conversational use cases, this far exceeds human reading speed, delivering a virtually instantaneous experience. A number like this would have been unimaginably high-end just a year ago, yet it has now become the new benchmark the entire industry is racing to achieve.

Why 318 tps Inference Speed Is So Remarkable

Inference Speed Differences from a User Experience Perspective

In real-world applications, inference speed directly determines product usability. Taking common conversational scenarios as an example:

• Below 50 tps: Noticeable "typing" lag; long text generation requires waiting several seconds;
• 100–150 tps: Near-fluent, basically meeting real-time conversation requirements;
• Above 300 tps: Almost instantaneous — users barely need to wait at all.

Once speed breaks through the 300 tps barrier, the AI response experience undergoes a qualitative leap. This isn't just about being "fast" — it means the system can support far more complex application scenarios, such as real-time code completion, multi-turn Agent reasoning, and batch processing of long documents — tasks that are extremely latency-sensitive.

High-Throughput Inference from a Technical Perspective

Achieving such high inference throughput isn't the result of any single factor — it's the product of coordinated hardware-software optimization. The industry typically attacks this challenge from several dimensions:

1. Hardware: The Rise of Dedicated Inference Chips

Dedicated inference chips (such as Groq's LPU and Cerebras' wafer-scale engine) or latest-generation GPUs boost speed through higher memory bandwidth and compute density.

Groq's LPU (Language Processing Unit) employs an architectural philosophy fundamentally different from traditional GPUs — deterministic computing. Traditional GPUs improve throughput through massive parallelism, but they hit a memory bandwidth bottleneck during autoregressive decoding (i.e., token-by-token generation) because every token generated requires reading the entire model weights from VRAM. The LPU eliminates dependency on external HBM memory by loading model weights entirely into on-chip SRAM, reducing inference latency to extremely low levels. Cerebras took an even more radical approach: fabricating an entire wafer into a single massive chip (WSE, Wafer Scale Engine), integrating hundreds of thousands of compute cores and tens of gigabytes of on-chip memory on a single die, fundamentally eliminating the communication overhead of multi-chip interconnects. These two companies represent two extreme design philosophies for "inference-specific hardware" — and are likely the source of the tweet author's astonishment.

2. Model-Level: Doing More with Less Computation

Techniques like quantization, model distillation, and sparsification reduce the computational cost per token.

Quantization refers to compressing model parameters from high-precision numerical formats (such as FP32 floating point, which occupies 32 bits) to lower-precision formats. INT8 uses 8-bit integers to represent weights, while FP8 uses 8-bit floating point. The direct benefit of reduced precision is: model size shrinks to one-quarter (from FP32 to INT8), memory usage and bandwidth requirements drop proportionally, and computation speeds up dramatically due to native hardware acceleration for low-precision operations — for example, NVIDIA's H100 GPU has 16× the theoretical throughput for FP8 compared to FP32. Through techniques like GPTQ, AWQ, and SmoothQuant, the industry has achieved aggressive quantization compression while keeping model output quality virtually intact.

Knowledge Distillation was first proposed by Hinton et al. in 2015. The core idea is to have a small model (student model) learn from the output distribution of a large model (teacher model), rather than learning directly from raw data. The teacher model's softmax outputs contain rich "dark knowledge" — for example, its probability assignments to incorrect answers also encode similarity information between categories. Through distillation, a student model with only one-tenth the teacher's parameter count can often retain 80%–95% of the teacher's performance. In the era of large models, distillation is widely used to compress 70B or even larger models into efficient 7B or smaller models — this is one of the key reasons why many open-source small models deliver surprisingly strong performance.

Sparsification is based on the core insight that not all parameters need to be activated during inference. Structured sparsity removes weight connections that contribute least to the output through pruning, while unstructured sparsity leverages the large number of near-zero values in weight matrices to skip unnecessary computations. More representative is the Mixture of Experts (MoE) architecture, as seen in Mixtral 8x7B and the architecture GPT-4 is speculated to use: the model contains multiple "expert" sub-networks, and each input token is routed to only 2–4 experts for processing while the rest remain dormant. This means a MoE model with hundreds of billions of total parameters may only require the computational load of a dense model with tens of billions of parameters per inference, dramatically reducing inference costs while maintaining the capabilities of a large model.

3. Inference Framework: Squeezing Every Last Drop of Compute

Techniques like speculative decoding, KV Cache optimization, and continuous batching extract every bit of available compute power.

Speculative Decoding cleverly breaks the constraint of sequential token generation in traditional autoregressive decoding. In the conventional approach, generating each token requires a complete forward pass, with strict dependencies between tokens. Speculative decoding introduces a lightweight "draft model" that quickly predicts the next several tokens, then uses the full large model to verify all these predictions at once. Since verifying multiple tokens can be done in parallel (essentially checking multiple positions in a single forward pass), when the draft model's prediction hit rate is high, multiple tokens are effectively generated with a single large-model inference call. This technique typically delivers 2–3× speedup without sacrificing any output quality.

KV Cache Optimization is key to inference acceleration in the Transformer architecture. During autoregressive generation, the attention computation for each new token must reference the Key and Value vectors of all preceding tokens. Without caching, generating each token would require recomputing KV values for the entire sequence, with computational cost growing quadratically with sequence length. KV Cache stores previously computed KV vectors in GPU memory, so each step only needs to compute the new token's KV and append it to the cache. However, this creates enormous memory pressure — for long-context scenarios, the KV Cache can consume tens of gigabytes of VRAM. The industry has developed a series of optimizations around this problem, including PagedAttention (the core technology of the vLLM project, which borrows virtual memory paging concepts from operating systems to manage KV Cache memory), GQA (Grouped Query Attention, which reduces the number of KV heads), and MQA (Multi-Query Attention).

Continuous Batching addresses GPU utilization. Traditional static batching requires all requests in a batch to start and end simultaneously. Since generation lengths can vary dramatically between requests, shorter requests must wait for the longest one to finish before resources can be freed, leaving the GPU idle much of the time. Continuous batching changes this paradigm: it dynamically schedules requests at every decoding iteration — completed requests immediately release resources, and new requests can be inserted at any time, keeping the GPU running at high utilization. The Orca paper first introduced this concept, and it has since been widely adopted by mainstream inference frameworks including vLLM, TensorRT-LLM, and TGI.

You may have noticed that the tweet author's phrase "how can you make this happen" expresses surprise at a competitor's achievement, suggesting that 318 tps likely comes from a team that deployed a breakthrough inference architecture — only through deep integration of the technologies described above could inference speed be pushed to such levels.

The Far-Reaching Implications of the Inference Speed Race for the AI Industry

Inference Efficiency Is Becoming the New Competitive Moat

Over the past two years, the LLM race has focused on "parameter scale" and "benchmark scores." But as model capabilities increasingly converge, inference efficiency is becoming the key differentiator for product competitiveness. For the same quality of output, whoever is faster and cheaper wins developers and enterprise customers.

Take inference-focused startups as an example: companies like Groq and Cerebras have circumvented the architectural bottlenecks of traditional GPUs through custom hardware, achieving throughput several times higher than mainstream solutions on specific models. Traditional GPU architecture faces a fundamental contradiction in inference scenarios: GPUs were originally designed for graphics rendering and massively parallel computation, excelling at matrix operations where larger batches mean greater efficiency. But autoregressive LLM inference is inherently a sequential process — generating only one token at a time with minimal computation but high memory access demands — leaving GPU compute units largely idle. This is known as the "memory-bound" bottleneck. Dedicated chips address this pain point through architecture-level redesign. This "speed-first" approach is reshaping the entire inference services market.

The Industry Anxiety Behind "We Have to Work Harder"

The tweet's closing line — "we have to work harder even more" — captures the sense of urgency felt across the industry. When competitors keep raising the performance ceiling, no team can afford to rest on its laurels. This healthy competitive pressure is objectively accelerating technology iteration across the entire industry — from the rapid evolution of open-source inference frameworks (projects like vLLM, SGLang, and llama.cpp) to innovations in chip design (NVIDIA's generational leaps from A100 to H100 to B200, plus the emergence of competitors like AMD MI300X and Intel Gaudi). Notably, this race involves not only chip manufacturers and cloud providers, but an increasing number of open-source community contributors who continuously lower the barrier to high-performance inference through algorithmic innovation, enabling more teams to approach top-tier performance on commodity hardware.

Real-World Application Scenarios Unlocked by High-Speed Inference

When inference speed is no longer the bottleneck, many previously "impractical" application scenarios become viable:

• Real-time Agent Systems: When multiple models chain together for inference, speed determines the usability of the entire pipeline. In a typical Agent workflow, a single user request may require 5–10 or more inference calls (including task planning, tool selection, result verification, etc.). If each inference takes 3–5 seconds, the entire pipeline balloons to over 30 seconds of latency — completely unsuitable for interactive use. 300+ tps enables each inference call to complete in milliseconds, making complex Agent workflows responsive and truly delivering the experience of "thinking and acting like a human assistant";
• Intelligent Code Completion: Developers are extremely sensitive to latency — research shows that delays exceeding 200 milliseconds break coding flow. High-speed inference delivers an experience nearly as smooth as local IDE autocompletion, which is the core reason products like GitHub Copilot and Cursor continuously optimize inference speed;
• Large-Scale Content Production: Costs for batch generation, translation, summarization, and similar tasks drop significantly as speed increases. Doubling inference speed means the same GPU resources can handle twice the requests per unit of time, effectively cutting the inference cost per million tokens in half;
• Edge and On-Device Deployment: As efficient inference technologies trickle down, running LLMs locally on phones and PCs becomes possible. Combined with model compression techniques like quantization and distillation, plus the maturation of on-device inference frameworks like Apple MLX and Qualcomm AI Engine, 7B-parameter models can already run at usable speeds on flagship smartphones, providing entirely new solutions for privacy-sensitive scenarios.

Conclusion: Speed Is Just the Beginning of the LLM Race

The emergence of 318 tps marks the transition of large models from "can we use it?" to "is it a pleasure to use?" But we should be rational: raw speed numbers don't tell the whole story — inference quality, context length, and cost-effectiveness are equally important. In real-world deployments, tps figures are closely tied to specific conditions like model size, input length, and concurrent user count — discussing speed without these contexts can be misleading. The true winners will be teams that achieve the optimal balance among "fast, accurate, and affordable."

This performance race is far from over. As the tweet says, "we have to work harder." It's foreseeable that tps records will continue to be broken in the near future — from next-generation chip architectures at the hardware level to continuous innovation at the algorithmic level, each breakthrough paves the way for broader AI adoption. And when inference costs drop low enough and speeds become fast enough, we may well enter an era where AI capabilities are as ubiquitous as electricity and running water.