Single RTX 6000 Pro Runs Qwen3.8-Flash-Next at NVFP4, Reaching 184 tok/s

Single-Card 184 tok/s: Qwen3.8-Flash-Next NVFP4 Benchmark Results

A technical post on Hacker News shows that someone successfully ran the Qwen3.8-Flash-Next model at NVFP4 precision on a single NVIDIA RTX 6000 Pro, achieving an inference speed of 184 tok/s. This figure is quite impressive for local deployment of large models, especially under single-card conditions. However, the post has only 4 points and 0 comments, meaning it has not yet been fully discussed and verified by the community, and readers need to judge its credibility for themselves.

Why This Speed Is Worth Noting

The RTX 6000 Pro is a professional-grade workstation GPU with large VRAM capacity, suitable for carrying the weights of medium-scale models. A generation speed of 184 tok/s is already close to the response experience of some cloud APIs, making it practically attractive for users who need local inference, are sensitive to data privacy, or operate in offline scenarios. As a low-precision format, NVFP4's core idea is to compress VRAM usage and computational overhead while preserving model quality as much as possible, which is also a hot topic in recent quantized deployment efforts.

The Positioning of the RTX 6000 Pro

The RTX 6000 Pro is a workstation-grade GPU launched by NVIDIA based on the Blackwell architecture (a member of the RTX PRO 6000 Blackwell series), equipped with 96GB of GDDR7 VRAM, targeting professional rendering, scientific computing, and local AI inference. Its VRAM capacity is close to the 80GB of the data-center-grade H100, but its power consumption and price are closer to workstation scenarios, so it is often regarded as a practical choice for "running large models on a single card." Compared with the 24GB of the consumer-grade RTX 4090, 96GB of VRAM can accommodate larger parameter counts or higher-precision weights, which is precisely the prerequisite for it to carry medium-scale models at NVFP4 while leaving room for KV Cache.

The Trade-offs of NVFP4 Quantization

NVFP4 is a 4-bit floating-point quantization scheme that has certain advantages in dynamic range compared with INT4, and is often used to improve throughput on VRAM-constrained hardware. However, low-bit quantization usually comes with precision loss. For models like Qwen3.8-Flash-Next, whether the actual output quality is acceptable requires task-level evaluation. The original post did not provide quality comparison data, so 184 tok/s can only serve as a speed reference, not an end-to-end deployment conclusion.

The Difference Between NVFP4, MXFP4, and INT4

NVFP4 is a 4-bit floating-point micro-scaling format introduced by NVIDIA in the Blackwell architecture, where each data block shares an FP8 (E4M3) scaling factor, and the block size is typically 16 elements. In contrast, the OCP-standard MXFP4 uses an E8M0 (pure exponent) scaling factor with a block size of 32; INT4 is fixed-point integer quantization with limited dynamic range. Thanks to finer block granularity and floating-point scaling, NVFP4 can better preserve small-value information in layers where activation distributions span widely, so at the same 4-bit width it often has a smaller perplexity loss than INT4.

In actual deployment, NVFP4 quantization and calibration are usually completed through native kernels in inference stacks such as TensorRT-LLM and vLLM, or through the NVIDIA ModelOpt toolchain. It requires the GPU to support FP4 via fifth-generation Tensor Cores to leverage hardware acceleration; older-generation GPUs can only simulate operation through dequantization, and the speed benefit will be greatly reduced.

The Significance and Limitations of Single-Card Deployment

The single-card approach lowers the hardware threshold and system complexity, eliminating the need for multi-card parallelism and complex communication scheduling, and is friendlier to individual developers and small to medium-sized teams. The fact that a single RTX 6000 Pro can run at 184 tok/s shows that, with reasonable quantization, the usability of local inference is improving.

But it should also be noted that this information lacks reproducible details: it does not give key conditions such as prompt length, batch size, context length, or whether the prefill stage is included. tok/s varies greatly under different test settings, and comparisons detached from these parameters are easily distorted. If readers want to use it as a reference, they should treat it as a directional signal rather than a strict benchmark.

The Issue of tok/s Metric Definitions

Throughput metrics for large model inference are usually divided into two categories: the speed of processing input prompts in the prefill stage, and the speed of token-by-token generation in the decode stage. The tok/s commonly mentioned in the community mostly refers to the single-request generation rate in the decode stage, but if the test includes long prompts or batching, the aggregate throughput will be significantly higher than the single-request figure. In addition, increasing context length slows down decoding due to KV Cache expansion, and whether quantization covers the KV Cache also affects the results. Therefore, when comparing tok/s from different sources, conditions such as batch size, input/output length, and whether speculative decoding is enabled must be aligned; otherwise, the numbers are not directly comparable.

Implications for Local Inference Practice

From an engineering perspective, this post suggests a viable path: choose a professional GPU with ample VRAM and pair it with a low-precision format like NVFP4 to obtain usable generation speed on a single card. For AI application developers and teams that need private deployment, this is worth adding to the evaluation checklist.

At the same time, it is recommended to record the complete test configuration when reproducing, including model version, quantization toolchain, input/output length, and hardware power consumption, so that the speed data can be comparable. The community currently has very little discussion of this result; only with more independent verification in the future can it be determined whether 184 tok/s is generally representative.