NVIDIA Releases Nemotron-3.5-Super-VL-120B-A12B Model Family

NVIDIA is once again stepping up its game in the open-source model space. According to Reddit community reports, NVIDIA has officially released the Nemotron-3.5-Super-VL-120B-A12B model family, covering the base version along with multiple quantization formats including NVFP4 and BF16, and introducing the LiLiCorr speculative decoding scheme.

Breaking Down the Model Family

The Nemotron-3.5-Super-VL-120B-A12B released here is not a single model but a complete technology stack. Judging from the naming convention, "120B-A12B" likely indicates an MoE (Mixture of Experts) architecture with 120B total parameters and 12B activated parameters, while "VL" points to vision-language multimodal capabilities.

NVIDIA has provided three versions at once — base, NVFP4, and BF16 — covering different scenarios from training research to inference deployment. NVFP4 is NVIDIA's flagship FP4 floating-point quantization format, specifically optimized for the Blackwell architecture, capable of significantly compressing memory usage while maintaining precision. The BF16 version targets scenarios requiring higher numerical stability.

MoE (Mixture of Experts) is a sparse architecture that activates subsets of parameters on demand within a Transformer. Specifically, it splits the feed-forward layers into multiple parallel "experts," then uses a learnable router to select Top-K experts for each token to participate in computation, while the remaining experts stay inactive. Taking "120B-A12B" as an example, the model has 120 billion total parameters, but only about 12 billion parameters actually participate in computation per token during the forward pass. This way, the model can match hundred-billion-scale dense models in parameter capacity, while computation and memory bandwidth consumption are closer to ten-billion-scale models. The trade-off is routing load balancing, expert parallel communication, and the need to still accommodate all weights during deployment, which places higher demands on total memory and multi-GPU interconnect — hence such architectures are typically deployed alongside quantization and tensor/expert parallelism.

What Is LiLiCorr Speculative Decoding

One technical highlight of this release is the addition of the LiLiCorr speculative decoding scheme. The core idea of speculative decoding is to use a lightweight draft model to quickly generate candidate tokens, which are then batch-verified by the main model, improving inference throughput without sacrificing output quality.

The name "LiLiCorr" suggests it may employ a linear correction or hierarchical verification mechanism. Speculative decoding delivers significant gains in long-text generation and high-concurrency serving scenarios. NVIDIA incorporating it into the model family means inference efficiency is treated as a first-class citizen — not just releasing weights, but bundling an end-to-end acceleration solution.

NVFP4 is a 4-bit floating-point data format introduced by NVIDIA on the Blackwell architecture. Unlike the common INT4 quantization, it preserves floating-point representation, making it more friendly to outliers in activation values. Its structure typically consists of a block-level scaling factor in E4M3 format plus a set of E2M1 4-bit mantissas, with every 16 or 32 elements sharing one scaling factor (microscaling), thereby maintaining precision at extremely low bit widths. Compared to FP8, NVFP4 can further compress the memory footprint of weights and activations by about half, and enables Blackwell's fifth-generation Tensor Cores to execute matrix multiplications at native FP4 throughput. Since this format depends on specific hardware instructions and kernels, equivalent support is often lacking on non-NVIDIA platforms — which is also why NVFP4 is seen as an ecosystem lock-in mechanism.

What Multimodal Plus MoE Means

With 120B total parameters paired with 12B activated parameters in an MoE design, plus vision-language capabilities, this combination targets the enterprise-grade multimodal inference market. The MoE architecture allows the model to activate only a portion of parameters during inference, offering better cost-effectiveness compared to dense models of the same scale.

Vision-language models have higher memory and latency requirements in actual deployment — the image encoder alone consumes substantial resources. NVIDIA providing an FP4-quantized version simultaneously indicates that the team, after weighing the precision sensitivity of multimodal scenarios, believes FP4 can already meet production needs. This "large model + aggressive quantization + speculative decoding" trifecta strategy is becoming NVIDIA's standard playbook for open-source models.

A vision-language model (VLM) typically consists of three parts: a vision encoder (such as ViT) that slices images into patches and encodes them into visual tokens; a projector that maps visual tokens into the language model's embedding space; and the language model backbone that concatenates these visual tokens with text tokens for autoregressive generation. Image resolution and patch count directly determine the scale of visual tokens — high-resolution inputs may produce thousands of tokens, so in actual inference, memory usage and first-token latency are significantly higher than pure text models. This is also why VLM deployment often adopts techniques such as dynamic resolution, visual token compression, or cache reuse, and it explains why low-bit quantization delivers particularly notable gains in multimodal scenarios.

Impact on the Developer Ecosystem

Judging from the iteration pace of the Nemotron series, NVIDIA is building a moat for its hardware ecosystem through open-source models. NVFP4 format has limited support on non-NVIDIA hardware, which objectively enhances the appeal of the Blackwell platform. For developers, choosing Nemotron means access to officially optimized inference paths, but also requires evaluating the risk of hardware lock-in.

The complete release of the model family (including multiple precisions and acceleration schemes) lowers the migration cost from experimentation to deployment. If the LiLiCorr scheme comes with open-source companion code, it will help drive the adoption of speculative decoding in real-world business applications.