Nori LLM Breaks the Million Tokens-Per-Second Barrier: Pushing the Limits of Inference Engine Performance

What Does Nori LLM's Million-Token Throughput Actually Mean?

A technical post recently surfaced on Hacker News that caught attention: a project called Nori claims to have achieved over one million tokens per second (1M tok/s) in LLM inference throughput. While the discussion hasn't gone viral (8 points, 1 comment at the time of writing), the number itself deserves a closer technical look — because it cuts to the heart of the cost and efficiency challenges in large language model deployment.

For any team running LLM services, token throughput directly determines how many concurrent requests a given piece of hardware can serve, which maps directly to inference cost. Reaching the million tok/s range is rarely the speed of a single model generating a single response. It's an aggregate throughput figure achieved through the combined effect of batching, concurrent request handling, and low-level kernel optimizations.

The Key Technical Approaches Behind High Throughput

To understand numbers like this, you need to break down the optimization techniques common to modern inference engines. High-throughput solutions in the industry typically rely on a few pillars: continuous batching lets the GPU dynamically fill in new requests without waiting for an entire batch to complete; PagedAttention-style techniques manage KV caches using a paging approach, dramatically reducing memory fragmentation; and there's also operator fusion, quantization (e.g., INT8/FP8), and custom kernels tuned for specific hardware.

The Throughput vs. Latency Trade-off

It's worth emphasizing that high aggregate throughput often comes at the cost of larger batch sizes — and larger batches drive up the Time To First Token (TTFT) for individual requests. So a peak metric like "1M tok/s" needs to be interpreted against real service-level objectives (SLOs) in production. A system that posts extraordinary throughput numbers may not simultaneously meet the low-latency demands of interactive use cases. This is the key question to ask when evaluating any inference engine: what model size, hardware, and batch configuration was used to generate this number?

In LLM serving, latency is typically broken into two key metrics: TTFT (Time To First Token) measures how long a user waits from sending a request to receiving the first output token — directly affecting perceived responsiveness — while TPOT (Time Per Output Token) measures the rate at which subsequent tokens are generated, determining the smoothness of streamed output. The larger the batch, the longer each request waits in the queue, and TTFT rises accordingly. Engines optimized for maximum throughput therefore tend to sacrifice TTFT, making them better suited for offline batch processing (e.g., data annotation, document summarization) than for latency-sensitive real-time conversation. In production, the goal is typically to find the optimal throughput-latency balance under SLO constraints, not to maximize throughput in isolation.

The Implicit Assumptions About Hardware and Scale

Million-token-per-second throughput usually implies smaller model sizes, shorter sequence lengths, or aggregation across multi-GPU or multi-node clusters. The same engine running a 7B model versus a 70B model will show order-of-magnitude differences in throughput. Discussing throughput numbers without specifying model scale is therefore of limited informational value — which is why the community should remain cautious when evaluating such claims.

Continuous Batching is a core scheduling mechanism in modern inference engines. Traditional static batching requires an entire batch of requests to complete before new ones are accepted, leaving GPU compute idle for significant stretches. Continuous batching allows new requests to be dynamically added and completed requests to be removed at the granularity of each generation step (iteration), keeping GPU utilization near 100%. vLLM popularized this mechanism and it has since become the industry standard.

PagedAttention borrows the concept of paging from operating system virtual memory, dividing the KV cache (Key-Value Cache) into fixed-size "pages" and allocating physical GPU memory on demand. Traditional approaches require pre-allocating contiguous memory blocks for each request, leading to severe fragmentation and waste. PagedAttention allows different requests — and even different sequences within the same request — to share physical pages, improving memory utilization by several times and directly translating into higher concurrent batch limits, which in turn boosts aggregate throughput.

Quantization is another technique with a significant impact on throughput. The core idea is to represent model weights and/or activations using lower-precision numerical formats (such as INT8, FP8, or even INT4) to reduce memory footprint and computation. Compared to an FP16 baseline, INT8 quantization can halve a model's memory requirements while achieving higher arithmetic throughput on hardware that supports it (e.g., the FP8 Tensor Cores in NVIDIA H100s). Different quantization schemes involve clear trade-offs between accuracy loss and speed gains: weight-only quantization (W8A16) has a smaller impact on accuracy, while full quantization (W8A8) offers greater gains but requires more careful calibration. For this reason, high throughput figures reported by a given engine need to specify whether quantization was enabled and at what precision level — otherwise cross-system comparisons are of limited value.

Why Inference Performance Has Become a Competitive Battleground

As LLMs move from research labs into large-scale production deployment, inference costs beyond training are becoming the dominant portion of operational expenditure. For high-concurrency API services, agent workflows, and batch data processing pipelines, throughput improvements translate directly into meaningful cost savings.

This explains why so many open-source projects and commercial solutions focused on inference optimization have emerged in recent years — from vLLM and TensorRT-LLM to a growing ecosystem of newer engines, all competing on this front. If Nori can validate its performance claims in a reproducible way, it will be a meaningful data point in this ongoing trend.

How to Rationally Evaluate Performance Claims Like This

Faced with an eye-catching figure like "1M tok/s," a pragmatic evaluator should look at several dimensions:

• Test Transparency: Are the full benchmark parameters disclosed — model, hardware configuration, batch size, and sequence length?
• Reproducibility: Can the results be independently reproduced by third parties using standardized benchmarks (e.g., public datasets, fixed workloads)?
• End-to-End Metrics: Beyond peak throughput, does the report include P50/P99 latency, TTFT, and other metrics that are more relevant to real-world user experience?
• Cost Comparison: Does the hardware cost per million tokens offer an advantage over existing solutions?

Given the limited public information currently available, the specific implementation details of Nori, the range of models it supports, and its benchmarking methodology still need further disclosure. Until that information is available, it's more appropriate to treat this as a technical signal worth monitoring rather than drawing definitive conclusions.

Summary

Nori LLM's claim of breaking the one-million tokens-per-second barrier reflects the fact that inference performance optimization remains one of the most active fronts in LLM engineering. For teams focused on deployment costs, projects like this are worth tracking — but in the absence of complete benchmarks and reproducible verification, their performance claims should still be approached with measured skepticism. Meaningful judgment comes from transparent test data and independent third-party validation.