TokenSpeed
A PyTorch-based token generation acceleration tool for the inference stage of large models. It improves inference throughput through techniques such as speculative decoding, KV cache optimization, and quantization, and has been incorporated into the official PyTorch ecosystem.
Timeline (last 90 days)
TokenSpeed provides the Docker runner image lightseekorg/tokenspeed-runner:latest, and after mounting the cache directory, model weights persist across container restarts.
TokenSpeed's MLA kernel optimized for the Blackwell architecture has been adopted by vLLM.
TokenSpeed provides an OpenAI-compatible API server built on the SMG project, and existing OpenAI clients only need to change the base_url to connect.
TokenSpeed requires an NVIDIA GPU host (the nightly wheel requires CUDA 13) or an AMD host with ROCm 7.2, and Python 3.11 to 3.13.
Running Kimi K2.5 on an NVIDIA B200 with real coding agent trajectories (SWE-smith, EvalScope evaluation), TokenSpeed outperforms TensorRT-LLM across the entire latency/throughput curve; at batch size 1 it is about 9% faster, and at about 100 tokens/second per user its throughput is about 11% higher.
TokenSpeed's MLA decode kernel has about half the latency of TensorRT-LLM under typical speculative decoding workloads.
On Qwen3.5-397B-A17B, TokenSpeed reported speeds of up to 580 tokens/second on agent workloads.
TokenSpeed's benchmarking tool is included in the repository's test/agentic_benchmark directory and can be used to reproduce on your own traffic.
TokenSpeed is an open-source LLM inference engine launched by the LightSeek Foundation, positioned as combining vLLM-level ease of use with TensorRT-LLM-level performance, and deeply tuned specifically for coding agent scenarios.
TokenSpeed's scheduler control plane is written in C++ as a finite state machine, while the execution side remains in Python.
7 more timeline events
All Facts (17)
In this test, the model's output speed in short-context scenarios is approximately 20 tokens/second, dropping to approximately 13 tokens/second for contexts of 200K+.
60%UnverifiedTokenSpeed provides the Docker runner image lightseekorg/tokenspeed-runner:latest, and after mounting the cache directory, model weights persist across container restarts.
50%UnverifiedTokenSpeed's scheduler control plane is written in C++ as a finite state machine, while the execution side remains in Python.
50%UnverifiedTokenSpeed provides an OpenAI-compatible API server built on the SMG project, and existing OpenAI clients only need to change the base_url to connect.
50%UnverifiedTokenSpeed requires an NVIDIA GPU host (the nightly wheel requires CUDA 13) or an AMD host with ROCm 7.2, and Python 3.11 to 3.13.
50%UnverifiedRunning Kimi K2.5 on an NVIDIA B200 with real coding agent trajectories (SWE-smith, EvalScope evaluation), TokenSpeed outperforms TensorRT-LLM across the entire latency/throughput curve; at batch size 1 it is about 9% faster, and at about 100 tokens/second per user its throughput is about 11% higher.
50%UnverifiedTokenSpeed's MLA decode kernel has about half the latency of TensorRT-LLM under typical speculative decoding workloads.
50%UnverifiedOn Qwen3.5-397B-A17B, TokenSpeed reported speeds of up to 580 tokens/second on agent workloads.
50%UnverifiedTokenSpeed's benchmarking tool is included in the repository's test/agentic_benchmark directory and can be used to reproduce on your own traffic.
50%UnverifiedTokenSpeed is an open-source LLM inference engine launched by the LightSeek Foundation, positioned as combining vLLM-level ease of use with TensorRT-LLM-level performance, and deeply tuned specifically for coding agent scenarios.
50%UnverifiedTokenSpeed is an LLM inference engine open-sourced by the lightseekorg team, officially positioned as a 'speed-of-light LLM inference engine' (light-speed large language model inference engine), with Python as its primary language.
50%UnverifiedThe TokenSpeed project has currently received 2,281 stars and 331 forks, with 25 new stars in a single day.
50%UnverifiedThe real test for TokenSpeed is whether it can turn 'speed of light' from a marketing phrase into a reproducible benchmark number and remain stable under real loads; its benchmark data, community feedback and production cases will be key signals for judging its value.
50%UnverifiedTokenSpeed focuses on accelerating token generation during the inference phase.
50%UnverifiedTokenSpeed's MLA kernel optimized for the Blackwell architecture has been adopted by vLLM.
50%UnverifiedIf TokenSpeed is to build a barrier in speed, possible areas of effort include more aggressive quantization strategies, finer-grained operator fusion, or scheduling optimization for specific hardware topologies.
50%UnverifiedThe PyTorch ecosystem working group announced the addition of 10 new projects to the PyTorch Ecosystem Landscape: Perforated, AReaL, TorchJD, RLinf, Miles, SMG, FiftyOne, TokenSpeed, VisualTorch, and TorchSurv.
50%