TokenSpeed: An LLM Inference Engine Built for Coding Agents

Teams self-hosting large language models usually face a dilemma: vLLM and SGLang are easy to deploy but don't squeeze the most out of the hardware, while TensorRT-LLM extracts maximum performance from NVIDIA hardware at the cost of a complex compilation and configuration process. TokenSpeed, an open-source inference engine from the LightSeek Foundation, aims to end that trade-off — delivering TensorRT-LLM-level performance with vLLM-level ease of use, and with deep tuning specifically for coding agent scenarios.

Why "Agents" Change the Inference Problem

Most inference benchmarks simulate a chat scenario: a short prompt, a short reply, done. But coding agents work in a completely different mode.

A typical agent session has three defining traits. First, extremely long context — combining code files, tool-call output, and conversation history, the prompt easily exceeds 50K tokens. Second, dense turns — the same conversation comes back repeatedly, each time longer than the last. Third, a hard speed floor — if token streaming is too slow, the agent feels broken. The TokenSpeed team sets a target lower bound of roughly 70 tokens/sec per user, chasing 200+ tokens/sec in some scenarios.

This means the optimization goal isn't simply "maximum throughput" but "maximum throughput per GPU while guaranteeing fast generation for every user." That framing directly determines most of TokenSpeed's design choices.

TTFT and TPOT: The Two Core Metrics in Agent Scenarios

The "tokens/sec per user" mentioned above is the reciprocal of TPOT (Time Per Output Token), which measures how fast the model produces each token. The other equally critical metric in agent scenarios is TTFT (Time To First Token), determined by the prefill phase — the time the model needs to process the entire input prompt and build the KV cache. When context exceeds 50K tokens, prefill computation grows roughly quadratically with sequence length, and TTFT often becomes the main cause of perceived lag. TokenSpeed's optimizations therefore must address both dimensions at once: lower prefill latency for long contexts through efficient attention kernels, and boost decode-phase throughput through KV cache reuse and scheduling strategies. This is also why a pure "maximum throughput" metric is misleading in agent scenarios — it can inflate overall QPS at the expense of single-user latency.

Architecture Breakdown: Four Key Design Decisions

Modeling Layer: Replacing Hand-Written Parallel Code with a Compiler

The traditional approach is to hand-write tensor-parallel communication code for each model — labor-intensive and error-prone. TokenSpeed's approach lets developers annotate where data should live at module boundaries, then a lightweight static compiler automatically generates collective communication operations. Less custom parallel code means fewer bugs introduced when adding new models.

Scheduler: C++ Control Plane + Python Execution Plane

The scheduler's control plane is written as a finite state machine in C++. Request lifecycle and KV cache ownership are explicitly modeled, and the type system can catch unsafe KV cache reuse at compile time rather than blowing up in production at 3 a.m. The execution side stays in Python for fast iteration and experimentation.

Kernels: Pluggable Registration Mechanism

Kernels live in separate pluggable packages managed through a registry, allowing NVIDIA and AMD backends to coexist. As one detail, their Blackwell-optimized MLA (Multi-head Latent Attention) kernel is fast enough that vLLM has already adopted the implementation — a form of technical endorsement from a competitor.

MLA and the Blackwell Architecture

MLA (Multi-head Latent Attention) is an attention variant introduced by the DeepSeek^Ad series of models. It projects Key/Value into a low-dimensional latent space before caching, which compresses KV cache usage substantially compared to traditional MHA/GQA — making it especially advantageous for long-context scenarios. However, its computation pattern differs considerably from standard attention, requiring purpose-built efficient kernels to run fast on GPUs. Blackwell is the codename for NVIDIA's architecture following Hopper (corresponding to B100/B200 chips), introducing a new generation of Tensor Cores, larger shared memory, and faster memory bandwidth, plus fifth-generation NVLink for higher multi-GPU interconnect bandwidth — features that are especially critical for communication-heavy tensor-parallel MoE inference. The fact that TokenSpeed's Blackwell-optimized MLA kernel was adopted back by vLLM indicates its low-level implementation has indeed reached an engineering quality recognized by the mainstream community.

Entry Point: OpenAI-Compatible API

Requests enter through an OpenAI-compatible server built on the SMG project, keeping CPU-side overhead low. This means existing client code using the OpenAI API only needs to change base_url to connect.

Quick Start: Get an Inference Service Running in Minutes

Requirements

You need an NVIDIA GPU host (nightly wheels require CUDA 13) or an AMD host with ROCm 7.2; Python 3.11 to 3.13; and if you choose the Docker route, GPU-capable Docker.

Option 1: Docker (Recommended)

The project provides runner images with all dependencies pre-installed:

docker pull lightseekorg/tokenspeed-runner:latest

docker run -itd \
  --shm-size 32g \
  --gpus all \
  --ipc=host \
  --network=host \
  -v ~/.cache:/home/runner/.cache \
  --name tokenspeed \
  lightseekorg/tokenspeed-runner:latest \
  /bin/bash

After mounting the cache directory, model weights persist across container restarts without needing to be re-downloaded.

Option 2: pip Install of the Nightly Build

On a CUDA 13 machine:

python -m pip install --upgrade tokenspeed \
  --extra-index-url https://lightseek.org/whl/nightly

Nightly builds are date-numbered, so you can pin versions for reproducibility. AMD users need to install the ROCm build of PyTorch first and use a separate wheel source.

Launching Your First Model

GPT-OSS 20B can run on a single GPU and is good for a smoke test:

tokenspeed serve openai/gpt-oss-20b \
  --served-model-name gpt-oss-20b \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1

Because the server uses the OpenAI API protocol, calling it is identical to OpenAI:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="gpt-oss-20b",
    messages=[{"role": "user", "content": "Explain KV caching in two sentences."}],
    max_tokens=200,
)
print(response.choices[0].message.content)

Any agent framework already integrated with OpenAI can point directly at this server.

Deploying Large Models

For large MoE models, launch parameters closely mirror vLLM. The official model recipes page provides tested launch commands for DeepSeek V4, the Qwen3 series, GLM 5.x, MiniMax M3, Kimi K3, Nemotron-3 Super, and more — it's best to start from a recipe rather than tuning from scratch.

Performance: Official Numbers and How to Verify Them

All performance figures published by the TokenSpeed team are self-reported and should be treated with caution:

• Running Kimi K2.5 on NVIDIA B200 with real coding agent trajectories (SWE-smith, EvalScope evaluation), TokenSpeed outperforms TensorRT-LLM across the entire latency/throughput curve. At batch size 1 it's about 9% faster, and at roughly 100 tokens/sec per user it delivers about 11% higher throughput.
• Their MLA decode kernel achieves roughly half the latency of TensorRT-LLM under typical speculative decoding workloads.
• On Qwen3.5-397B-A17B, agent workloads reported speeds up to 580 tokens/sec.

These are vendor benchmarks and should be viewed as "reasons worth testing" rather than "performance guarantees." The good news: the benchmarking tools are included in the repo's test/agentic_benchmark directory, so you can reproduce them on your own traffic.

Speculative Decoding and MoE Models

Speculative decoding is a general technique for accelerating decoding: a lightweight "draft model" or an internal draft head generates multiple candidate tokens at once, which the main model then verifies in parallel — compressing several serial decode steps into one forward pass. It speeds up single-user generation without changing the output distribution, but at the cost of greater pressure on the KV cache and attention kernels during verification, making it sensitive to kernel efficiency — exactly the context behind the claim that "latency is about half of TensorRT-LLM under typical speculative decoding workloads." MoE (Mixture of Experts) is the dominant architecture among today's frontier open-source large models, such as DeepSeek, Qwen, and Kimi. Its defining trait is a large parameter count with only a subset of experts activated per token, reducing actual compute at inference. MoE inference bottlenecks are often not in compute but in memory bandwidth and communication (experts scattered across GPUs require all-to-all communication) — which is precisely where TokenSpeed's emphasis on automatic generation of communication in the modeling layer, and its focus on multi-GPU tensor parallelism, adds value.

Which Teams Is It For

TokenSpeed is currently best suited for the following scenarios:

• Deploying large MoE models (DeepSeek, Kimi, Qwen, GLM, MiniMax, etc.) for coding agents or long-context, multi-turn conversation workloads.
• Running on newer NVIDIA hardware (Hopper, especially Blackwell) or AMD MI350-class GPUs, with a focus on cost per token.
• Having considered TensorRT-LLM's speed advantages but not wanting to take on its build and configuration complexity.
• Needing day-zero support for new frontier open-source models. TokenSpeed supported Kimi K3 and Thinking Machines' Inkling on release day.

In terms of positioning, TokenSpeed fills a real gap in the inference engine market: teams that don't want to sacrifice performance, nor bear TensorRT-LLM's engineering burden. Whether it delivers on its promises depends on real-world workload performance — and the open benchmarking tools at least make verification feasible.

One-line Summary

LightSeek open-sources TokenSpeed, matching TensorRT-LLM performance with vLLM-level ease of use, tuned for coding agents.

Paragraph Summary

TokenSpeed is an MIT-licensed open-source inference engine from the LightSeek Foundation aimed at ending the "ease of use or performance" binary: it offers a vLLM-level deployment experience while pursuing TensorRT-LLM-level hardware utilization, with deep tuning for coding agent scenarios. The article notes that agent workloads have three defining traits — long context (50K+ tokens), dense multi-turn conversations, and a hard speed floor for generation (70-200+ tokens/sec) — shifting the optimization goal from "maximum throughput" to "maximum throughput per GPU while guaranteeing single-user speed." Architecturally, it employs four key designs: compiler-generated collective communication, a C++ finite state machine scheduler, a pluggable kernel registry, and an OpenAI-compatible entry point. Deployment supports both Docker and pip, and existing OpenAI clients can connect simply by changing the base_url. Official performance numbers are self-reported, but the benchmarking tools are open and reproducible.