Local Deployment of NVIDIA's FP4 Qwen3.8: Blackwell-Exclusive and 55 Environment Variables

NVIDIA squeezed Qwen3.8-Flash-Next down to 96 GiB, compared to 355 GB for the original BF16, with GPQA Diamond accuracy dropping only 0.7 points. Sounds like a性价比 choice for local deployment, but the first line of the model card says: upstream vLLM can't read these weights, only Blackwell architecture can run it, and the RTX PRO 6000 startup script requires exporting 55 environment variables before vllm serve.

What Exactly Did NVIDIA Change

The underlying model is Alibaba's Qwen3.8-Flash-Next, a 125B total parameter, 6B activated parameter MoE, with an additional 51B n-gram embedding table (called PLE in the card) and a 4B MTP head. It includes a vision encoder and hybrid attention (Gated DeltaNet plus Qwen Sparse Attention), with native context of 262K tokens, expandable to 1M.

The most notable aspect of NVIDIA's FP4 version is Quantization-Aware Distillation (QAD), using Model Optimizer v0.47.0, rather than standard post-training quantization. The BF16 model acts as teacher, and low-precision weights are trained to match the teacher's token probabilities and hidden states, even setting separate losses for "end thinking" and "end reply," specifically targeting the issues of quantized models "thinking endlessly" or "not stopping." The training budget is 3 epochs totaling 500M tokens, first 2500 main updates, then 1500 joint fine-tuning updates.

The precision allocation isn't all FP4: decoder routing experts and MTP routing experts use NVFP4, decoder attention and vision attention linear layers use MXFP8, vision MLP and merger use NVFP4, and remaining vision tensors and norms stay BF16/FP32. Keeping attention at 8-bit and letting expert layers benefit from FP4 is a reasonable split, since the bulk of bytes is in routing experts.

Quantization-Aware Distillation (QAD) is a compression method that directly trains low-precision models under quantization constraints, distinct from common post-training quantization (PTQ). PTQ first trains BF16 weights then reduces precision, with calibration data usually only hundreds to thousands of samples, and quantization errors cannot be backpropagated for correction; QAD embeds quantization operators into the training graph, so low-precision weights are actually used in forward passes, and then uses the full-precision teacher's output as supervision signal, updating weights through backpropagation. Common losses include KL divergence of token probabilities, cosine similarity of intermediate hidden states, and task-specific losses for particular behaviors. NVIDIA additionally sets separate losses for "end thinking" and "end reply" in the card, precisely using distillation data to constrain the quantized model's termination behavior in reasoning mode—such fine-grained behavior alignment is beyond what ordinary PTQ calibration can cover. The cost is significantly increased training compute and the need for the teacher to participate throughout.

NVFP4 and MXFP8 are two different low-precision numerical formats. FP4 refers to 4-bit floating point; NVFP4 is NVIDIA's defined block-scaled 4-bit format, where every 16 numbers share an FP8 scale, balancing dynamic range and decimal precision, mainly suitable for layers with large weight variance. MXFP8 is the OCP-defined 8-bit block floating point, where every 32 numbers share an E8M0 scale, offering higher precision but double the footprint. The article mentions keeping attention at MXFP8 and routing experts at NVFP4, precisely based on the trade-off between precision and size: attention is sensitive to numerical perturbations, so 8-bit is safer; expert layers have the largest parameter share, so 4-bit is needed to capture the main compression gains.

How Much Memory Does FP4 Save Compared to BF16 and GGUF

Version	Size	Source
BF16 Original	355 GB	Unsloth docs
GGUF Q8_0	188.26 GB	bartowski
GGUF Q4_K_M	119.60 GB	bartowski
GGUF UD-Q4_K_XL	111.3 GB	Unsloth
GGUF IQ4_XS	97.68 GB	bartowski
NVIDIA FP4-Local	~96 GiB	NVIDIA card
GGUF UD-IQ1_S	72.5 GB	Unsloth

There's a unit trap: 96 GiB is approximately 103 GB, falling between IQ4_XS and UD-Q4_K_XL, and larger than bartowski's IQ4_NL (100.29 GB). So it's not as small as "96" sounds. The reason to choose it should be the training method, not the file size.

The arithmetic is also worth doing: 96 GiB of weights on a 96 GB RTX PRO 6000 leaves no room for KV cache. NVIDIA's validated configuration offloads the n-gram embedding table to CPU (VLLM_PLE_CPU_OFFLOAD=1), claiming this frees up 850K tokens of FP8 KV cache. On DGX Spark, it can run with or without PLE offload, and with the table on disk, there's 1M tokens of FP8 KV cache.

Accuracy Numbers Published by NVIDIA

Precision	GPQA Diamond	ToolEvalBench
BF16	92.2%	89.4%
FP4-Local	91.5%	88.7%

Both dropped 0.7 points. GPQA Diamond has only 198 questions, so 0.7 points is about 1.4 questions, falling within single-sample noise range—it neither proves small loss nor large loss.

There's also a comparison issue: NVIDIA's BF16 baseline is 92.2, while Qwen's own card reports 91.7, probably due to different harness or settings. Comparing FP4 to NVIDIA's BF16 is correct; comparing to Qwen's numbers would lead to the erroneous conclusion of "only dropping 0.2 points."

The vision part is a clear gap. The card lists image and video inputs, quantizes vision MLP to NVFP4, but publishes no vision benchmark, and the distillation data is all text. If you're choosing this version for image input, you are the eval.

For comparison, Unsloth disclosed their quantization's KL divergence: UD-Q4_K_XL average KLD 0.046893, top-1 agreement with BF16 92.255%. This is a different dimension of metrics, and the publicly available numbers from both sides are insufficient to determine which is better.

GPQA Diamond is the hardest subset of the GPQA benchmark, with graduate-level physics, chemistry, and biology multiple-choice questions written and validated by domain experts, designed to require expert-level reasoning rather than retrieval. The Diamond subset has 198 questions, and precisely because of the small size, evaluation results have large statistical fluctuations: at 95% confidence interval, the standard error for a single sample is about 2 percentage points, so a 0.7-point difference is well within noise. Therefore, such small-sample high-difficulty benchmarks are more suitable as evidence of "no obvious collapse" rather than fine-grained precision grading. To judge the true gap between FP4 and BF16, larger-scale benchmarks or multiple sampling averages are needed, along with公开 of sampling temperature, few-shot count, and answer extraction method.

KL Divergence (KLD) measures the difference between two probability distributions, often used in quantization evaluation to compare the output distribution of a quantized model versus a BF16 model on the same batch of inputs. Unsloth reports UD-Q4_K_XL average KLD 0.046893, meaning the quantized model's token distribution differs little from the full-precision model, but KLD is sensitive to distribution tails and long sequences, and highly dependent on the evaluation dataset and prompt length. When different vendors' calibration and evaluation sets are inconsistent, values cannot be directly compared horizontally. It's more complete to look at it together with top-1 agreement: low KLD and high top-1 agreement indicate quantization doesn't change the preferred token in most cases, but there's still偏移 on low-probability branches, which may be amplified in long-chain reasoning.

Which GPUs and Runtimes Support FP4

Blackwell plus vLLM, and only in community containers.

• Runtime: vLLM only. The card says NVFP4 scale factors use "a new lossless compression format," and "upstream vLLM support for this format has not yet landed," so you must use one of two containers built by the Local Inference Lab community organization.
• Verified hardware: RTX PRO 6000 Blackwell (single card) and DGX Spark.
• Not mentioned: RTX 5090, any Hopper or Ada card, SGLang, TensorRT-LLM, llama.cpp, Ollama. "Supported hardware microarchitecture: NVIDIA Blackwell" is the entire list.

The original Qwen weights support vLLM, SGLang, Transformers, and TokenSpeed, with SGLang having day-0 support on release day.

NVIDIA's version is not the only NVFP4 checkpoint. The LMSYS blog mentions RadixArk/Qwen3.8-Flash-Next-NVFP4, an SGLang checkpoint, which LMSYS says is available the same day as the model. It's a standard post-training quantization: only quantizes routing experts, uses 128 CNN/DailyMail articles truncated to 512 tokens for calibration, validated on GB300 and B300. LMSYS reports 540 tok/s at batch size 1 with B200 TP4 plus MTP. This is data center hardware, but it shows what NVIDIA's QAD is competing with: on one side, 128 short calibration samples; on the other, 500M tokens of distillation.

Blackwell is NVIDIA's GPU microarchitecture generation name, corresponding to data center B100, B200, GB200, GB300, and desktop RTX PRO 6000 Blackwell, RTX 50 series, etc. Its fifth-generation Tensor Cores natively support FP4 matrix operations and introduce a hardware acceleration path for block-scaled formats, where NVFP4 scale factor decompression and multiply-add can be done within tensor cores, which Hopper and Ada architectures lack. This is exactly the source of the article's "Blackwell-exclusive": even if you put the weight files on a Hopper card, it lacks the corresponding instructions and container runtime support, so FP4 computation cannot be executed. DGX Spark uses the GB10 superchip, also part of the Blackwell family, hence listed as verified hardware.

n-gram Embedding Table (PLE) is the offload object repeatedly mentioned in the article. PLE typically refers to Per-Layer Embedding or similar token-level embedding structures, where the model maintains an additional high-dimensional vector table for each token, with parameter counts reaching billions. This part is pure lookup operation, not participating in matrix multiplication, so it can be placed in CPU memory or even disk for on-demand reading without significantly slowing GPU computation. NVIDIA uses VLLM_PLE_CPU_OFFLOAD to move the 51B PLE table out of VRAM, freeing space for KV cache—this is how the card claims 850K tokens FP8 KV cache. The cost is that each forward pass may trigger CPU-to-GPU data transfer, with actual throughput affected by PCIe bandwidth and memory latency, requiring real-world testing in long-context scenarios.

Run Commands: RTX PRO 6000 Blackwell

Simplified one-line command from the card, copied verbatim:

docker run -d --name qwen38-tp1 --init --gpus '"device=0"' \
  --restart on-failure --network host --ipc host --shm-size 32g --stop-timeout 30 \
  --ulimit memlock=-1 --ulimit stack=67108864:67108864 \
  -v model-cache:/root/.cache/huggingface -v qwen38-runtime:/cache \
  -e PROFILE=qwen38-flash-next -e PORT=8000 \
  ghcr.io/local-inference-lab/vllm:release-10152026

Even with the one-line command, it's recommended to read the detailed version on the card, which shows what the profile enables: --quantization modelopt_mixed, --kv-cache-dtype fp8, --max-model-len 262144, --max-num-seqs 16, --gpu-memory-utilization 0.975, and MTP speculative decoding with "num_speculative_tokens":3. The 0.975 VRAM utilization leaves almost no desktop margin—either use a headless machine or dedicate another card for display.

Run Commands: DGX Spark

First set HF_TOKEN on the host, then start the container:

mkdir -p $HOME/.cache/huggingface $HOME/.cache/vllm $HOME/.cache/flashinfer $HOME/.triton $HOME/.cache/b12x

docker run --rm -it \
  --gpus all \
  --ipc host \
  --network host \
  --shm-size=16g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --security-opt seccomp=unconfined \
  --security-opt apparmor=unconfined \
  -e HF_TOKEN=$HF_TOKEN \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $HOME/.cache/vllm:/root/.cache/vllm \
  -v $HOME/.cache/flashinfer:/root/.cache/flashinfer \
  -v $HOME/.triton:/root/.triton \
  -v $HOME/.cache/b12x:/root/.cache/b12x \
  --entrypoint /bin/bash \
  eugr/spark-vllm-b12x:latest

Inside the container, the card exports 14 variables (including VLLM_PLE_TABLE_MEMORY="disk" and CUTE_DSL_ARCH="sm_121a"), then runs vllm serve nvidia/Qwen3.8-Flash-Next-FP4-Local with parameters --load-format b12x, --max-num-seqs 8, --gpu-memory-utilization 0.8, and 4 speculative tokens for MTP. It's recommended to copy the whole block directly from the card, don't hand-type it.

Both configurations use --reasoning-parser qwen3 and --tool-call-parser qwen3_xml, with tool calls available via the OpenAI-compatible API on port 8000. Qwen recommends thinking mode with temperature=1.0, top_p=0.95, top_k=20.

Cheaper Alternatives Not on Blackwell

GGUF plus llama.cpp. Unsloth's run command:

unsloth run --model unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL

People in Unsloth's speed thread aren't using Blackwell workstations. Someone reports RTX 3090 with 128 GB DDR4 running Q4_K_XL decoding at 19 t/s; someone using a Metal-optimized llama.cpp branch on M1 Ultra 128GB running IQ4_NL at 39.48 t/s at depth 0; another running Q4_K_XL on RTX 6000 PRO with TG 59.70 t/s. Unsloth docs cite a single RTX 6000 PRO with MTP reaching 170 t/s, baseline 100 t/s, but don't specify quantization level. NVIDIA published not a single throughput number for the FP4 version—a strange omission for a local inference release.

GGUF and llama.cpp are the ecosystem opposite to the vLLM route in local inference. GGUF is the single-file model format used by llama.cpp, packaging weights, tokenizer, quantization metadata, and meta-information together, supporting various quantization levels from Q8_0 to IQ1_S, and can run on CPU, Metal, CUDA, Vulkan, and other backends. Due to the open format and mature toolchain, community quantization publishers like bartowski and Unsloth can quickly produce multi-level quantization files, and users can choose based on VRAM and precision needs. llama.cpp merges into the mainline quickly, doesn't depend on vendor proprietary containers, and doesn't require exporting大量 environment variables—in sharp contrast to the deployment threshold of NVIDIA's FP4 version. The downside is that operator optimization for new architectures usually lags behind vLLM, and MoE and hybrid attention scheduling efficiency may be inferior to dedicated runtimes, so throughput data needs to be compared under the same hardware and quantization level for fairness.

My Take

If you have an RTX PRO 6000 or DGX Spark, you can try NVIDIA's version. QAD is a real method, and the 0.7-point drop is about what 4-bit level quantization can achieve. Everyone else should stick with GGUF—it runs on mainline llama.cpp, uses your existing hardware, and requires zero environment variables. Revisit after upstream vLLM merges that scale format, because those 55 variable settings are the part least wanting long-term maintenance.

So the question is yours: Do you trust vendor FP4 quantization, or the community GGUF from unsloth and bartowski? Has anyone run FP4-Local and Q4_K_XL side-by-side on the same prompt batch and seen quality differences? Feel free to post numbers, especially for image input, since NVIDIA didn't provide any.

Sources

• nvidia/Qwen3.8-Flash-Next-FP4-Local model card
• Qwen/Qwen3.8-Flash-Next model card
• Qwen blog: Qwen3.8-Flash-Next
• Unsloth: Qwen3.8-Flash-Next local run guide
• bartowski/Qwen3.8-Flash-Next-GGUF
• Unsloth GGUF speed discussion thread
• LMSYS: Day-0 SGLang support
• RadixArk/Qwen3.8-Flash-Next-NVFP4

One-line Summary

NVIDIA releases 96 GiB FP4 version of Qwen3.8, Blackwell-exclusive with 55 environment variables for deployment; regular users should stick with GGUF.

Paragraph Summary

NVIDIA compressed Qwen3.8-Flash-Next to about 96 GiB in an FP4 version, using Quantization-Aware Distillation (QAD) rather than standard post-training quantization, with 3 epochs and 500M tokens to make low-precision weights match a BF16 teacher's token probabilities and hidden states. Accuracy-wise, GPQA Diamond and ToolEvalBench each drop 0.7 points, but the former has only 198 questions, so the difference is within sampling noise, and the vision part has no benchmarks at all. The file size isn't smaller than some 4-bit GGUF. Choosing it is about the training method. Deployment restrictions are strict: only supports Blackwell architecture on vLLM community containers, RTX PRO 6000 requires exporting 55 environment variables and PLE CPU offload, DGX Spark has a 14-variable configuration. Upstream vLLM doesn't yet support its scale format, while original Qwen weights support vLLM, SGLang, etc. Non-Blackwell users are advised to continue using llama.cpp GGUF, with community-measured speeds on RTX 3090, M1 Ultra, and other hardware.