Nemotron-3.5-Super
A mixture-of-experts (MoE) large language model introduced by NVIDIA. It uses a design with many small experts, with experts operating in a 1024-dimensional latent space, and supports the NVFP4 low-precision format.
Timeline (last 90 days)
A single expert of Nemotron-3.5-Super (up 2688×1024 + down 1024×2688) occupies only 3.10MB in NVFP4 format.
Nemotron-3.5-Super needs to read about 2.7GB of expert weights per token (22 experts × 40 layers × 3.10MB).
The developer built an inference engine based on PyTorch and Triton and achieved Nemotron-3.5-Super running on an RTX 5060 (8GB VRAM).
The skip-zero optimization (exploiting that 92.5% of neurons are zero after ReLU², reading down weights in groups of 16 neurons) raised speed from 6.6 tok/s to 7.3-7.6 tok/s, an improvement of about 12%.
After increasing the RAM cache to 20GB, disk reads dropped from 136.9ms to 121.5ms, and speed increased from 7.3 to 8.2 tok/s.
Numerical correctness validation of the engine shows that, compared with the CPU fp32 transformers reference implementation, the English paragraph PPL is 5.233 vs 5.280, average KL ≤ 0.003, and greedy answer matches are 63/64 and 64/64.
All Facts (6)
A single expert of Nemotron-3.5-Super (up 2688×1024 + down 1024×2688) occupies only 3.10MB in NVFP4 format.
50%UnverifiedNemotron-3.5-Super needs to read about 2.7GB of expert weights per token (22 experts × 40 layers × 3.10MB).
50%UnverifiedThe skip-zero optimization (exploiting that 92.5% of neurons are zero after ReLU², reading down weights in groups of 16 neurons) raised speed from 6.6 tok/s to 7.3-7.6 tok/s, an improvement of about 12%.
50%UnverifiedAfter increasing the RAM cache to 20GB, disk reads dropped from 136.9ms to 121.5ms, and speed increased from 7.3 to 8.2 tok/s.
50%UnverifiedNumerical correctness validation of the engine shows that, compared with the CPU fp32 transformers reference implementation, the English paragraph PPL is 5.233 vs 5.280, average KL ≤ 0.003, and greedy answer matches are 63/64 and 64/64.
50%UnverifiedThe developer built an inference engine based on PyTorch and Triton and achieved Nemotron-3.5-Super running on an RTX 5060 (8GB VRAM).
50%