dlmrt
An inference engine for diffusion language models that captures the denoising loop into a CUDA graph for execution on the GPU, reducing CPU synchronization overhead and improving end-to-end throughput.
Timeline (last 90 days)
On a single H100 and at similar measured accuracy, QwenDLM-4B + ORL reaches 1,298 tokens/s, DFlash 746 tokens/s, and EAGLE3 405 tokens/s (end-to-end system comparison: ORL runs on dlmrt, speculative decoding runs on vLLM).
The core idea of the dlmrt inference engine is to capture the entire denoising loop into a single CUDA graph and execute it directly on the GPU, avoiding launching and synchronizing from the CPU at every step.
The author reports that under the same model and the same decoding settings, dlmrt is 1.2 to 1.6 times faster than SGLang/vLLM on a single H100.
All Facts (3)
On a single H100 and at similar measured accuracy, QwenDLM-4B + ORL reaches 1,298 tokens/s, DFlash 746 tokens/s, and EAGLE3 405 tokens/s (end-to-end system comparison: ORL runs on dlmrt, speculative decoding runs on vLLM).
50%UnverifiedThe core idea of the dlmrt inference engine is to capture the entire denoising loop into a single CUDA graph and execute it directly on the GPU, avoiding launching and synchronizing from the CPU at every step.
50%UnverifiedThe author reports that under the same model and the same decoding settings, dlmrt is 1.2 to 1.6 times faster than SGLang/vLLM on a single H100.
50%