GLM-5.3-Flash Local Deployment: NVIDIA FP4 Requires Two Blackwell GPUs

GLM-5.3-Flash is a multimodal MoE model with 320B total parameters and 18B active per token, under the MIT license, supporting a 1M token context. NVIDIA released a checkpoint for it called FP4-Local, advertised as runnable "locally" — but flip open the model card and you'll find only two verified hardware configurations: two RTX PRO 6000 Blackwells, or two DGX Sparks.

"Flash" in this version means cheap per-token API pricing, not a smaller model; and the word "Local" carries more marketing weight than technical substance.

What exactly is GLM-5.3-Flash

The previous generation, GLM-4.7-Flash, was a 30B-A3B text MoE that could run on a single consumer GPU. GLM-5.3-Flash is in an entirely different league: 320B total parameters, 18B active, and Z.ai calls it the first natively multimodal model in the GLM-5 series, supporting image-text input.

Architecturally it uses hybrid sparse plus linear attention, along with Manifold-Constrained Hyper-Connections (mHC). The context window is 1M tokens, and reasoning_effort offers low, high, and max, defaulting to max.

Z.ai claims it approaches Claude Opus 4.8 on coding and agent benchmarks, at one-tenth the price of GLM-5.2. Those are vendor numbers; the model card reports Terminal-Bench 2.1 at 84.3 and DeepSWE at 63.4, so you can compare for yourself.

Two NVIDIA quantized versions — don't mix them up

NVIDIA released two sets of quantized weights with completely different purposes.

nvidia/GLM-5.3-Flash-NVFP4 is conventional post-training quantization (ModelOpt 0.47.0), tested on GB200, with examples using 4-way tensor parallelism — aimed at data centers.

nvidia/GLM-5.3-Flash-FP4-Local is the version this article focuses on. It uses quantization-aware distillation (QAD): the BF16 model serves as teacher, the student model's routed expert weights are quantized to NVFP4 on every forward pass, and gradients update the underlying BF16 weights. In other words, the quantized model is trained to recover what quantization destroys — something one-shot rounding cannot do.

On the precision mix: decoder routed experts use NVFP4 (E2M1 values, E4M3 block scaling), decoder attention and vision attention use MXFP8, vision MLP and merger linear layers use NVFP4, and the remaining vision tensors and norms stay in BF16.

NVFP4 is a Blackwell tensor core format, and the model card explicitly states Blackwell is the only supported microarchitecture, with no mention of Hopper or Ampere. The card also notes that scaling factors use a "new lossless compression format," which is precisely why upstream vLLM can't load it for now and a community container is required. If you have an H100 or a 4090, no matter how much VRAM, this checkpoint is not for you.

Quantization-aware distillation (QAD) versus common post-training quantization (PTQ) deserves elaboration. PTQ is one-shot rounding: take trained BF16 weights, compress them straight to low precision, then measure how many points you lost — whatever you lose is what you lose. QAD preserves gradient updates during quantization — on the forward pass a "fake quantization" operator rounds weights to FP4 and dequantizes them back, on the backward pass gradients flow through this operator into the underlying BF16 weights, and a BF16 teacher model provides soft-label constraints. The student model is thus explicitly trained to compensate for quantization error, especially at precision-sensitive spots like routing decisions and attention values. The cost is that distillation requires extra training compute and a data pipeline — which is exactly why NVIDIA can report a 0.8-point gap while most GGUF releases only report file size.

VRAM math: FP4-Local is not smaller than 4-bit GGUF

Put the sizes and memory requirements of all versions side by side and the conclusion is a bit counterintuitive (GiB and GB are mixed because the original sources mix them; 166 GiB is roughly 178 GB):

Version	Size/Memory	Platform
BF16 (Unsloth GGUF repo)	642 GB	Multi-GPU server
Q8_0 GGUF	341 GB	Server / large-memory machine
NVIDIA FP4-Local	~166 GiB VRAM	2× RTX PRO 6000 Blackwell, 2× DGX Spark
UD-Q4_K_XL GGUF	200 GB	llama.cpp, as long as RAM+VRAM suffice
UD-IQ4_XS GGUF	157 GB	llama.cpp
UD-Q2_K_XL GGUF	109 GB	llama.cpp
UD-IQ1_S GGUF	93.1 GB	llama.cpp
MLX 4-bit	~204 GB disk, 224 GB RAM	Apple Silicon

Two RTX PRO 6000s total 192 GB of VRAM; subtract about 166 GiB of weights and runtime overhead, and NVIDIA's validated configuration can still fit an FP8 KV cache of 1.8M tokens; the dual-Spark setup (128 GB each) reaches 2M tokens. The linear attention layers are why a million-token context isn't just talk on this hardware.

The key point: FP4-Local is not smaller than 4-bit GGUF — IQ4_XS is 157 GB. What FP4 actually buys you is native tensor core compute on Blackwell, plus a published accuracy number — the latter is the part worth paying attention to.

Accuracy loss: nearly zero on two benchmarks

The FP4-Local model card gives this comparison:

Precision	GPQA Diamond	ToolEvalBench
BF16	92.1%	91.9%
FP4-Local	91.3%	92.4%

GPQA drops 0.8 points, while tool calling actually gains 0.5 points. The PTQ version of NVFP4 tells a similar story across more benchmarks: GPQA Diamond 0.9211 vs 0.9217, MMMU Pro 0.763 vs 0.7688, Terminal Bench 2.1 0.8315 vs 0.8258.

But for a vision-language model, two benchmarks is a thin evaluation. The Local card has no MMMU and no long-context numbers — and those are precisely where aggressive quantization is most likely to show cracks. Meanwhile, Unsloth's GGUF publishes only file size and no accuracy at all, so on paper you're comparing a measured 0.8-point drop against an "unknown."

GPQA Diamond is a set of graduate-level physics, chemistry, and biology multiple-choice questions, each written by domain experts and subjected to multiple rounds of review, designed so a model can't answer by retrieval or memorization — which is why it's often used as a proxy for "real reasoning ability doesn't degrade." ToolEvalBench evaluates function calling and tool selection: given tool descriptions and user intent, the model must output structurally correct call arguments. The two cover reasoning and agent behavior respectively, but neither tests visual understanding or long-context retrieval — which is exactly why the article points out that "two benchmarks is thin for a VLM."

How to serve it on RTX PRO 6000

The Local card provides no TensorRT-LLM or SGLang instructions; the vLLM in the container is the only path.

For dual RTX PRO 6000 Blackwell, use the image ghcr.io/local-inference-lab/vllm. The model card's simplified run method uses tag release-10152026 with PRESET=glm53-tp2, but another line says release-10142026 — confirm which tag actually exists before pulling.

The detailed path is a vllm serve with about 40 arguments; key items excerpted from the model card are below (copy the full command from the model card — don't copy from here):

vllm serve nvidia/GLM-5.3-Flash-FP4-Local \
  --tensor-parallel-size 2 \
  --decode-context-parallel-size 2 \
  --quantization modelopt_mixed \
  --kv-cache-dtype fp8 \
  --max-model-len 1048576 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.985 \
  --attention-backend B12X \
  --moe-backend b12x \
  --linear-backend b12x \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"reasoning_effort":"max","clear_thinking":false}' \
  --speculative-config '{"method":"mtp","draft_sample_method":"probabilistic","rejection_sample_method":"standard","moe_backend":"b12x","attention_backend":"B12X","num_speculative_tokens":3}'

--max-num-seqs 8 reveals the target user: a single developer or a small team.

The dual DGX Spark setup points to eugr/spark-vllm-b12x:latest, with cross-node tensor parallelism 2, --max-model-len set to 500,000, --max-num-seqs 4, launched via run-recipe.sh --setup from the spark-vllm-docker repo. The model card reminds you to start the worker before the head node.

For chat use, two things are worth adjusting: the model card itself recommends clear_thinking: true for chat, while this configuration gives false; also, reasoning_effort defaults to max, which you can drop to high or low if latency matters more than that last benchmark point.

The --speculative-config in the command enables multi-token prediction (MTP) speculative decoding: the model ships with a lightweight draft head that guesses several subsequent tokens in one forward pass, then the main model verifies them in parallel, skipping token-by-token decoding for accepted portions. For a large MoE with 18B active, every decode step must move a lot of expert weights; speculative decoding amortizes "one weight read per token" into "one weight read per few steps" — one of the most significant speedups in low-concurrency scenarios. num_speculative_tokens: 3 means guessing 3 at a time; too high a value lowers the acceptance rate and actually slows things down. Meanwhile, --decode-context-parallel-size 2 shards the KV cache across two GPUs along the sequence dimension, working with tensor parallelism to support million-level context.

No Blackwell: llama.cpp or MLX

The llama.cpp side is messier than it should be. The Unsloth repo still points to its PR #27754 or Unsloth Desktop, but that PR was closed unmerged on October 1, 2026. A maintainer commented: "Support is in, but incompatible until Unsloth's GGUF is updated." So before downloading 200 GB, confirm whether Unsloth's files have been re-uploaded against upstream llama.cpp, or build the PR branch yourself.

Either way, read the caveats in that PR before you start, or your output may be a pile of gibberish: set NVIDIA_TF32_OVERRIDE=0 and use -fa off. The author says the TF32 switch raised top-1 agreement on the test fixture from 0.896 to 0.9995, and that flash attention converts F32 latents to F16, which MLA cannot tolerate. The PR also adds the vision tower, with image input covered.

NVIDIA_TF32_OVERRIDE=0 llama-server -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL -fa off

Given this is an MoE with 18B active, offloading experts to system RAM while keeping attention on a 24-32 GB GPU is the realistic low-cost route. It'll be slow, but it'll run.

On Mac there are two options. For the MLX route, note this is a vision model and requires mlx-vlm 0.6.17 or newer, not mlx-lm. The 4-bit MLX build is about 204 GB on disk and needs 224 GB of RAM, corresponding to a 256 GB Mac Studio. Orcarouter's guide steers 128 GB machines toward a roughly 102 GB "2bit-lite" build, provided you first raise the wired memory limit with sudo sysctl iogpu.wired_limit_mb=114688 — and the model card itself warns that long code generation is unreliable at that precision.

llama.cpp on Metal is also an option now. Issue #29867 once reported 1-to-9-second decoding stalls per token on M5, attributed to the fused Lightning Indexer falling back to CPU. ggerganov couldn't reproduce it and pointed to a fix already on master; the reporter subsequently measured about 31 t/s at low context and about 25 t/s at high context, with no more stalls. The issue remains open, but if you're on an old build, update before blaming the model.

Is Blackwell worth buying for FP4?

The QAD work is the most honest part of the whole release: NVIDIA trained the quantized model back toward BF16 behavior and published the gap — something most GGUF releases don't do. But the "Local" label presupposes you already own two workstation-grade Blackwell cards or two Sparks, and the checkpoint depends on a community vLLM container with two image tags appearing on the same model card. Calling it a preview aimed at the very few who already own that hardware is closer to reality.

If you have a 4090 plus 192 GB of DDR5, what you can realistically run this month is UD-Q4_K_XL (200 GB out of 216 GB total), leaving very little room for context; on that machine, the 157 GB IQ4_XS is the safer choice.

So we're back to the old question for local models: is FP4 with a published accuracy number worth upgrading to Blackwell for? Or is the Q4 GGUF you already have good enough? If you've run either version, feel free to share your tokens/sec and hardware configuration.

One-line Summary

GLM-5.3-Flash's FP4-Local checkpoint is Blackwell-only, no smaller than 4-bit GGUF, and its value lies in the accuracy data.

Paragraph Summary

The article dissects NVIDIA's FP4-Local quantized checkpoint for GLM-5.3-Flash. The model is a multimodal MoE with 320B total parameters and 18B active, supporting 1M context under the MIT license; "Flash" means cheap API pricing rather than a small model, and "Local" is more marketing than anything — the only verified hardware is two RTX PRO 6000 Blackwells or two DGX Sparks. Its technical highlight is quantization-aware distillation (QAD), which trains the quantized model to recover accuracy loss: GPQA drops only 0.8 points while ToolEvalBench gains 0.5. But at about 166 GiB it isn't smaller than the 157 GB IQ4_XS GGUF, and it only supports Blackwell with a community vLLM container. Users without Blackwell should take the llama.cpp UD-Q4_K_XL/IQ4_XS route or the mlx-vlm 4-bit route on Mac.