Nvidia Nemotron 3.5 Lightning Launches Free on AI Gateway

Nvidia has dropped another heavyweight model on the open-source community. Nemotron 3.5 Lightning is now available on Vercel's AI Gateway, offered free of charge under the model name nvidia/nemotron-3.5-lightning-free, with inference served directly by Nvidia.

A Model Built for "Long-Running" Workloads

Nemotron 3.5 Lightning has a clear positioning: it's not a toy for one-shot Q&A, but rather built for long-running agents, sub-agent collaboration, chatbots, and RAG systems.

This means its design goals lean toward scenarios involving continuous uptime, multi-turn calls, and frequent tool interaction. For developers building agent workflows, a stable, cost-controllable inference backend is often more important than raw benchmark scores.

Agent here refers to a program driven by a large model that can autonomously plan multi-step operations and call external tools to complete tasks. Unlike traditional Q&A chat, an agent often needs to think repeatedly, call search or code execution tools, observe results, and then decide the next step — a process that can span dozens of rounds or more. Sub-agent collaboration means splitting a complex task among multiple specialized small agents processed in parallel or sequentially — for example, one handling retrieval, one writing code, and one verifying. RAG (Retrieval-Augmented Generation) retrieves relevant snippets from an external knowledge base before generating an answer, then has the model respond based on those snippets, commonly used to give models access to private or up-to-date information. What these three scenarios share is: long sessions, frequent calls, and high sensitivity to the stability and per-unit cost of the inference backend — which explains why the model emphasizes long context and tool-calling capabilities.

30B Parameters, Only 3B Activated: Hybrid MoE + Mamba-2 Architecture

The most interesting part of this model is its architecture. It has 30 billion total parameters, but only activates about 3 billion per token — a classic Mixture of Experts (MoE) approach that trades sparse activation for inference efficiency.

Going further, it uses a hybrid structure where Mamba-2 and MoE layers are interleaved, paired with a small number of attention layers. Mamba-2 belongs to the State Space Model (SSM) family, which is typically more compute-efficient than pure Transformers when handling long sequences. This "SSM + limited attention" combination is currently an important technical direction for long-context models.

The accompanying capability list is also quite complete:

• 1 million token context window, enough to fit an entire code repository or a large document collection
• Tool calling support, a must-have for agent scenarios
• Structured outputs support, making it easy to feed downstream systems for parsing

MoE (Mixture of Experts) is a sparse-activation model design approach: the network contains multiple parallel "expert" subnetworks plus a router responsible for dispatching. For each input token, the router selects only a few experts to participate in computation while the rest stay silent. This way, the model's total parameter count can be very large (high knowledge capacity), but the actual compute consumed per token is proportional only to the activated portion (low inference cost). Nemotron 3.5 Lightning's 30B total parameters with 3B activated is exactly this "large capacity, low activation" strategy — giving the model stronger expression and memory capabilities while reducing the compute overhead of a single forward pass to roughly that of a 3B dense model.

Mamba-2 belongs to the State Space Model (SSM) family. Unlike Transformers, which let each token interact with all other tokens in the sequence through self-attention, SSMs use a recursively updatable internal state to compress historical information, so compute and memory overhead when processing long sequences typically grow approximately linearly with length rather than quadratically as with attention. Mamba-2 improves the state update mechanism and parallel training efficiency over its predecessor, making it easier to integrate with existing architectures. Interleaving Mamba-2 layers with MoE layers and sprinkling in a few attention layers is essentially about getting both the long-sequence efficiency of SSMs and the precise retrieval modeling capability of attention.

How to Integrate: One Model Name Across Mainstream Tools

Integration follows AI Gateway's consistent "unified API" approach. Developers just need to set the model name to nvidia/nemotron-3.5-lightning-free to call it through multiple channels:

• AI SDK: A few lines of TypeScript for streaming calls, e.g., using streamText with the model name and prompt
• Chat Completions / Responses / Messages API: Compatible with mainstream API formats
• Coding agents: Usable in tools like Claude Code, Codex, and Cursor

To use it in coding agents, first install the latest Vercel CLI, run npx vercel ai-gateway setup, then select the corresponding model name in your agent.

import { streamText } from 'ai';

const result = streamText({
  model: 'nvidia/nemotron-3.5-lightning-free',
  prompt: 'Write a TypeScript function that retries a failed fetch with exponential backoff.',
});

AI Gateway is a unified proxy service sitting between developers and model providers. Instead of separately integrating Nvidia, OpenAI, or Anthropic's respective APIs and billing systems, developers send requests to the gateway, which routes them to the corresponding provider by model name. Benefits include: one interface compatible with multiple models, centralized usage and cost tracking, automatic retry or failover to backup models when a provider goes down, and budget caps per API key. BYOK (Bring Your Own Key) means developers use keys they've obtained from providers, forwarding requests through the gateway — enjoying the gateway's governance capabilities while continuing to use existing provider accounts and quotas. AI SDK is Vercel's open-source TypeScript toolkit, and streamText is its core function for streaming text generation, ideal for returning model output token by token in web applications.

AI Gateway's Value: Unified Entry Point with Zero Markup

Putting the model on AI Gateway, rather than having developers integrate with Nvidia's API individually, means the gateway layer provides a full set of engineering capabilities: a unified calling interface, usage and cost tracking, retry and failover configuration, and performance optimizations aimed at "higher availability than any single provider."

There are also governance features like custom reports, per-API-key budgets, and routing rules, which are practical for team collaboration and cost control.

On pricing, AI Gateway explicitly states it passes through provider pricing as-is, with no markup and no platform fee on inference, and the same applies to BYOK (bring your own key) requests. "Free" here refers to the model currently being offered free of charge, not the gateway itself.

A Detail Worth Noting

There's one note in the official documentation worth attention: providers may retain prompts for model improvement. For teams handling sensitive data, private code, or with strict compliance requirements, this needs to be factored into the selection process — and where necessary, evaluate data masking or alternative deployment methods.

Prompt retention is a common point of divergence in model service terms. For models offered free or at low cost, providers often state in their agreements that user-input prompts may be used for model training or improvement, in exchange for zero- or low-cost usage rights. This contrasts with enterprise-grade paid APIs, which typically promise "we don't train on customer data." For teams handling private code, customer personal information, or regulated data, even if the model itself is free, compliance requirements may prevent direct use — requiring data masking, switching to paid endpoints that explicitly promise no data retention, or choosing local/private deployment of open-weight versions. This is precisely one advantage of open-weight models: weights can be downloaded and self-hosted, fundamentally preventing data leakage.

Why This Matters

An open-weight model with 30B total parameters, only 3B activated, 1 million token context, and tool-calling capabilities offered for free is a low-barrier experimentation opportunity for agent developers.

It also reflects two trends: first, the fusion of Mamba-family state space models with Transformers is moving from papers to usable open weights; second, "model aggregation + governance" layers like AI Gateway are becoming an intermediate zone for developers to avoid single-provider lock-in.

If your project is looking for a long-context, low-cost, tool-calling-capable inference backend, this model is worth running through your actual workflow before making a judgment.

One-line Summary

Nvidia Nemotron 3.5 Lightning is now free on Vercel AI Gateway, targeting long-running agent scenarios.

Paragraph Summary

Nvidia has brought Nemotron 3.5 Lightning to Vercel's AI Gateway, offering free inference under the name nvidia/nemotron-3.5-lightning-free. The model targets long-running agents, sub-agent collaboration, chatbots, and RAG systems. Architecturally, it uses a hybrid MoE design with 30B total parameters activating only about 3B per token, interleaving Mamba-2 layers with a few attention layers to balance long-sequence efficiency. It features 1 million token context, tool calling, and structured outputs, and can be accessed via AI SDK, Chat Completions, and other mainstream APIs, as well as coding agents like Claude Code and Cursor. AI Gateway provides a unified entry point, usage tracking, and failover, passing through provider pricing without markup. Note that providers may retain prompts for model improvement.