Single RTX 5090 Hits 2788 tokens/s: Diffusion LLM Acceleration Framework ORL Fully Open-Sourced

Diffusion language models (dLLMs) have long had an awkward problem: in theory they can refine multiple tokens in parallel, but in practice they often fail to translate that into a speed advantage due to slow convergence. The ORL (On-Policy Residual Learning) project recently released by the Reddit machine learning community attempts to push this into practical territory by working on both the training strategy and the backend engine.

What Problem Does ORL Solve?

Autoregressive LLMs generate one token at a time, while diffusion LLMs can theoretically determine multiple tokens in parallel in a single forward pass. But getting the model to converge in fewer forward passes has always been difficult.

ORL focuses on two areas. The first is residual learning: instead of predicting from scratch at each step, the model carries the previous step's prediction as residual context and refines it progressively. The problem is that these residuals are generated by the model itself, and their distribution shifts as training progresses. If you train on offline-cached old residuals, you get a distribution mismatch at inference time. ORL's approach is to train directly on fresh trajectories generated by the current model, replaying the residual states it actually encounters during inference.

The second is making uncertainty drop faster. Supervising only the final output doesn't necessarily teach the model to converge quickly. ORL introduces Adjacent Entropy Contraction, which requires each denoising step to reduce uncertainty relative to the previous step, and gates already high-confidence predictions to avoid ineffective pressure. The result is that the model learns to lock in more tokens per forward pass while maintaining or even improving accuracy.

Residual learning and distribution mismatch is the key background for understanding ORL. The denoising process of a diffusion language model can be seen as an iterative refinement loop: at each step, the model receives the current noisy state and outputs a prediction for the clean token. Residual learning has the model predict not the final result directly, but the 'increment relative to the previous step', carrying the previous prediction as context into the next step, thereby reducing the complexity of the mapping that must be learned at each step.

The difficulty is that at inference time, these residuals are generated by the current model itself, and their distribution drifts as the model weights update. If training uses offline-cached residuals produced by an old model, the mapping the model learns won't match the states it encounters at inference, and errors accumulate step by step. This is the origin of 'on-policy': training data must be generated in real time by the current policy (the current model), similar to on-policy sampling in reinforcement learning, to ensure the training distribution matches the inference distribution.

Measured Data Across Three Model Groups

QwenDLM-4B is a diffusion model converted from Qwen3-4B. According to the author's report in the Reddit post, ORL increases tokens per forward pass by up to 6.2x, and with the dlmrt engine, end-to-end throughput improves by about 6.9x, with HumanEval pass@1 rising from 82.9% to 89.6%.

On LLaDA-2.1-mini, ORL requires only 4.2M self-generated training tokens and 36 H100 GPU hours of post-training to achieve up to 3x throughput improvement at comparable quality. DiffusionGemma 26B MoE already has very high parallelism, yet ORL still delivers up to 1.2x improvement in tokens per forward pass.

In terms of hardware peaks, a single RTX 5090 with LLaDA-2.1-mini + ORL at NVFP4 precision reaches 2,788 tokens/s; on 8×B200, DiffusionGemma 26B MoE + ORL at NVFP4 reaches 7,708 tokens/s. The author explicitly states that these numbers are peaks under single-request decoding (batch size 1), not concurrent aggregate throughput, and not averages over arbitrary prompts.

The comparison with speculative decoding is also worth noting. On a single H100, at similar measured accuracy, QwenDLM-4B + ORL reaches 1,298 tokens/s, DFlash reaches 746 tokens/s, and EAGLE3 reaches 405 tokens/s. The author also notes this is an end-to-end system comparison: ORL runs on dlmrt, speculative decoding runs on vLLM.

NVFP4 is a 4-bit floating-point inference format introduced by NVIDIA on the Blackwell architecture, existing alongside earlier FP8 and INT4 quantization schemes. It uses an E2M1 (1 sign bit, 2 exponent bits, 1 mantissa bit) micro-scaling structure combined with block-level scaling factors, compressing weights and activations to 4 bits while preserving numerical range. Compared to INT4, the floating-point format tolerates outliers better and typically retains better accuracy on language models.

Blackwell's Tensor Cores natively support NVFP4 matrix multiplication, so the actual speedup comes from two places: first, memory footprint and bandwidth requirements drop by about 4x, allowing a single card to hold larger models; second, matrix multiply throughput increases significantly. The article's 2,788 tokens/s on a single RTX 5090 is a direct demonstration of this format on a consumer-grade Blackwell card. It's worth noting that quantization itself may introduce slight quality loss, and the author reports throughput numbers achieved at comparable accuracy.

What dlmrt Inference Engine Gets Right

A training strategy alone isn't enough—the scheduling approach on the inference side can eat away the low-latency advantage of diffusion decoding. dlmrt's core idea is to capture the entire denoising loop into a single CUDA graph and execute it directly on the GPU, rather than launching and synchronizing from the CPU at every step.

Under the same model and same decoding settings, the author reports that dlmrt is 1.2 to 1.6x faster than SGLang/vLLM on a single H100. dlmrt also supports NVFP4 inference on Blackwell architecture, including the consumer-grade RTX 5090, which is the prerequisite for hitting 2,788 tokens/s on a single card. On the training side, supporting work includes residual state capture, specialized attention kernels, and compute-balanced packing, speeding up training steps by up to 5.2x.

CUDA graph is a GPU execution optimization mechanism provided by NVIDIA. In conventional deep learning inference, each operator (kernel) is launched individually by the CPU, and the Python interpreter and framework scheduling introduce tens of microseconds of overhead. For autoregressive models generating tens of thousands of tokens, this overhead can be amortized; but a diffusion LLM may decode only a small number of steps per forward pass, and the denoising loop itself has a limited number of steps, so the CPU scheduling overhead becomes proportionally much larger, directly eating into the low-latency advantage.

CUDA graph works by recording the entire denoising loop once, capturing all kernel launches, memory operations, and dependencies, forming a replayable graph structure. After that, each inference requires only one launch to execute the entire loop continuously on the GPU, without step-by-step CPU intervention and synchronization. This requires the model structure, tensor shapes, and memory addresses to remain stable between recording and replay, which imposes some constraints on dynamic control flow and variable sequence lengths.

SGLang and vLLM are currently the mainstream open-source LLM inference serving frameworks, both deeply optimized for autoregressive models in terms of continuous batching, KV cache management, and PagedAttention. Their scheduling granularity is per-token or per-step, naturally suited to the serial generation pattern of autoregression.

When running a diffusion LLM on such frameworks, two mismatches arise: first, the steps within the diffusion denoising loop are tightly coupled, and step-by-step framework scheduling introduces extra synchronization; second, when decoding multiple tokens in parallel, the organization of KV cache and attention computation differs from autoregressive assumptions. The 1.2–1.6x advantage dlmrt reports essentially comes from treating the entire denoising loop as an atomic execution unit, bypassing the scheduling paths that general frameworks designed for autoregressive scenarios—not from differences in the model's own compute.

Open-Source Checklist and Getting Started

The project adopts a fully open-source strategy, covering model weights, training code, evaluation pipeline, and inference engine. The entry points released by the author include the arXiv paper (2610.20687), the project homepage and interactive demo, training and evaluation code on GitHub, the dlmrt inference engine repository, and checkpoints on Hugging Face.

If you have an RTX 5090 on hand, you can directly run the LLaDA-2.1-mini ORL NVFP4 checkpoint with dlmrt. The author is also soliciting reproductions, testing under different workloads, and inference engine contributions.

A Signal Worth Watching

Discussion of diffusion LLMs has long remained at the level of architectural possibility, but ORL brings the problem back to an engineering-verifiable level: how to resolve residual distribution mismatch, how to supervise convergence speed, and how to keep the denoising loop uninterrupted on the GPU. Together, these three things determine whether diffusion LLMs can compete head-on with autoregressive models in local inference scenarios.

It's worth noting that all the impressive numbers so far are limited to single-request decoding and concentrated on peak performance on specific benchmarks. Real-world concurrent throughput, long-context stability, and convergence behavior under different task distributions still need community reproductions to fill in.

One-line Summary

ORL uses online residual learning and GPU-side denoising loops to boost diffusion LLM throughput by up to ~6.9x.

Paragraph Summary

The ORL project released by the Reddit machine learning community targets the pain point of diffusion language models—'parallelizable in theory, slow to converge in practice'—by working on both the training strategy and the inference engine. On the training side, it uses On-Policy residual learning, training on fresh trajectories generated in real time by the current model to avoid the distribution mismatch caused by offline-cached residuals; it also introduces Adjacent Entropy Contraction, requiring each denoising step to reduce uncertainty and gating high-confidence predictions, so the model locks in more tokens in fewer forward passes. On the inference side, dlmrt captures the entire denoising loop into a CUDA graph and executes it directly on the GPU, running 1.2–1.6x faster than SGLang/vLLM under the same settings. In measured tests, QwenDLM-4B throughput improves by about 6.9x, with HumanEval pass@1 rising from 82.9% to 89.6%; LLaDA-2.1-mini needs only 36 H100 GPU hours to achieve 3x throughput. All numbers are single-request peaks, and the project is fully open-sourced.