RL Fine-Tuning on Expert Tasks: 12 Benchmark Gains for Nemotron 3.5 Super

NVIDIA Nemotron 3.5 Super achieves broad benchmark gains through independent RL training on five expert tasks.
NVIDIA's Nemotron 3.5 Super (120B total, 12B active parameters) was post-trained via RL on five expert-built task datasets, each with its own rank-32 LoRA adapter and no supervised stage. Twelve benchmarks show broad gains, with GDP_pdf improving the most (+13.8 points) and notable progress on SWE-bench Pro V2 and Terminal-Bench 2.1. Behavior analysis shows adapters adopt correct strategies more often rather than learning new knowledge. Data cost and adapter fusion remain barriers to practical use.
Introduction: When Reinforcement Learning Meets Expert Data
NVIDIA's Nemotron 3.5 Super, with 120B total parameters and 12B active parameters, was recently post-trained via reinforcement learning (RL) on five expert-built task datasets, with a separate LoRA adapter trained for each dataset. The core question behind this experiment: what can a large model actually learn from RL on high-quality expert data? The answer is reflected in significant gains across twelve benchmarks.

Methodology: Five Tasks, Five Independent RL Training Runs
Task Datasets and Training Configuration
The research team selected five expert-built task domains: real-world software engineering, complex constraints, long-horizon tool use, specialized chart understanding, and enterprise document reasoning. Each domain corresponds to an independent RL training run, using the CISPO algorithm on the Tinker platform to train a brand-new rank-32 LoRA adapter, with no supervised stage at any point. Each adapter was compared against the base model on the same tasks, the same test harness, and the same settings.
The CISPO algorithm is a policy optimization method for large-model reinforcement learning, with the full name Clip-Inspired Sequential Policy Optimization (the name may vary by paper). Its core idea borrows the clipping mechanism from PPO (Proximal Policy Optimization) but is adapted for sequence generation tasks, aiming to stabilize training and prevent overly large policy updates. CISPO typically combines rule-based rewards or verifiable reward signals, directly optimizing model outputs without supervised fine-tuning. Unlike traditional RLHF, which relies on human preference models, CISPO is better suited to tasks with clear correctness criteria (e.g., code passing tests, constraint satisfaction), giving it a natural advantage in this experiment's expert-task setting.
LoRA (Low-Rank Adaptation) adapters are a parameter-efficient fine-tuning technique that freezes the base model parameters and trains only low-rank decomposition matrices to adapt to downstream tasks. Rank-32 means the rank of the low-rank matrices is 32, which determines the parameter count and expressiveness of the adapter. This experiment trains a separate rank-32 LoRA adapter for each task, meaning each adapter uses only a small number of parameters (typically 0.1%–1% of the original model's parameters) yet can significantly change model behavior on a specific task. The benefits of this approach: the base model remains unchanged, and multiple adapters can be plugged in as needed; the drawback is that each task requires training and storing its own adapter, making it hard to directly merge multi-task capabilities.
Composition of the Evaluation Benchmarks
Of the twelve main benchmarks, eight were built by other teams and four were built by Surge AI. This mixed evaluation design ensures both external validity and test dimensions that are closer to real-world application scenarios.
Surge AI is a company focused on data annotation and expert task construction, and it provided four of the benchmarks for this experiment. Unlike commonly used academic benchmarks (such as SWE-bench and Terminal-Bench), the benchmarks built by Surge AI are closer to real enterprise application scenarios, typically involving complex constraint following, multi-turn dialogue, and document reasoning. The mixed evaluation combining third-party benchmarks (eight) and Surge AI benchmarks (four) aims to balance external validity and practical application relevance, avoiding overfitting to public benchmarks alone.
Results: Broad Progress from Code to Documents
Software Engineering and Terminal Tasks
SWE-bench Pro V2 rose from 86.9% to 92.1%, and Terminal-Bench 2.1 rose from 63.9% to 70.3%. These two benchmarks directly reflect the model's ability to solve real software problems and operate a terminal, and the magnitude of the gains shows that RL training genuinely improved the model's engineering practice level.
Constraint Following and Multi-Turn Dialogue
IFBench gained +4.5 points, and MultiChallenge rose from 58.3% to 66.6%. The latter specifically examines the model's ability to maintain constraints and context across multi-turn interactions, and the improvement indicates that the constraint adapter learned to carry information from earlier turns into later responses.
Automation and Chart Understanding
AutomationBench rose from 14.8% to 27.5%, and Chartography rose from 39.4% to 50.6%. You may not have noticed that the document adapter also gained +8.0 points on AutomationBench without using API tools, indicating that the model learned to read spreadsheet data directly.
A Major Breakthrough in Document Reasoning
GDP_pdf jumped from 10.4% to 24.2%, a gain of +13.8 points, the largest increase of all benchmarks. This shows that RL training has a particularly pronounced effect on complex tasks such as enterprise document reasoning.
GDP_pdf is a benchmark that evaluates a model's ability to extract and reason over information from enterprise documents (especially in PDF format). It requires the model to understand complex document structures such as tables, charts, and multi-column layouts, and to answer questions requiring multi-step reasoning based on document content. The difficulty of this benchmark lies in the diverse document layouts, scattered information, and the need for the model to have spatial reasoning and cross-paragraph integration capabilities. Although the absolute gain of GDP_pdf from 10.4% to 24.2% is not high, the relative improvement exceeds 130%, indicating that RL training significantly improved the model's strategies for handling complex enterprise documents, such as more systematically locating key information and more effectively using document structure cues.
Behavioral Analysis: How the Adapters Compensate for the Base Model's Shortcomings
The research team read side by side the attempt logs of the base model and the new adapters on the tasks, and found that the base model's failures clustered into a few patterns, while each adapter more frequently did the things the base model missed.
- Tool-use adapter: finding the data and rules before acting;
- Coding adapter: writing its own tests or temporary files on SWE-bench Pro V2;
- Constraint adapter: carrying content from earlier turns into answers;
- Chart adapter: measuring chart markers in code;
- Document adapter: reading AutomationBench spreadsheets.
These behavioral changes reveal the essence of RL training: it is not about teaching the model new knowledge, but about guiding it to adopt the correct problem-solving strategies more frequently.
Significance and Implications
This experiment demonstrates the post-training potential of RL on expert tasks: without supervised fine-tuning, reinforcement learning alone can deliver substantive improvements across multiple professional domains. For enterprises and developers hoping to customize large-model capabilities, this means they can build expert datasets for specific tasks and train dedicated LoRA adapters via RL, achieving domain-expert-level performance without changing the base model.
However, each task requires training its own adapter and depends on high-quality expert data, which remains a barrier in practical applications. How to reduce data construction costs and achieve fusion of multi-task adapters will be directions worth watching going forward.
One-line Summary
NVIDIA Nemotron 3.5 Super achieved comprehensive performance gains across twelve benchmarks through independent RL training on five expert tasks.
Paragraph Summary
NVIDIA's Nemotron 3.5 Super (120B total parameters, 12B active parameters) underwent reinforcement learning post-training separately on five expert-built task datasets, training an independent rank-32 LoRA adapter for each task with no supervised stage at any point. Twelve benchmarks showed comprehensive gains, with GDP_pdf seeing the largest improvement (+13.8 points), and SWE-bench Pro V2 and Terminal-Bench 2.1 also making significant progress. Behavioral analysis shows that the adapters did not learn new knowledge but rather more frequently adopted correct strategies the base model missed, such as finding data before acting and writing test files on their own. The experiment proves the potential of RL in post-training on expert tasks, but each task requires training its own adapter and depends on high-quality expert data; data cost and adapter fusion remain barriers to practical application.
Related articles

Gemini 4 Neon Leak: A Potential Strong Contender Against Opus 5.5
A Hacker News thread suggests Gemini 4 Neon looks like an Opus 5.5 competitor. This article unpacks the product positioning behind this community signal, shifts in the top-model competitive landscape, and what it means for developers' model selection strategy.

Claude Fable 5.2's Self-Authored Short Film 'Fable' Sparks Discussion
A short film 'Fable,' credited to Claude Fable 5.2 for script, visuals, and directing, appears on Hacker News. This article analyzes the significance of full-pipeline AI creation, community reactions, and what it means for content creators.

Anthropic IPO Pricing: A Signal of AI Giants Going Public
Anthropic's IPO pricing draws attention. This article analyzes its potential impact on AI industry valuation, commercialization paths, and the competitive landscape, and highlights public-market expectations and challenges for AI companies.