[KongchangAI]
· 2 min read· 1,112 words

RSIBench-Infra: Can Agents Autonomously Optimize Production-Grade Training and Inference Infrastructure?

RSIBench-Infra: Can Agents Autonomously Optimize Production-Grade Training and Inference Infrastructure?

RSIBench-Infra evaluates agents' ability to autonomously optimize training and inference infrastructure with 20 production-grade tasks.

RSIBench-Infra is an infrastructure-optimization benchmark for agent recursive self-improvement, featuring 20 training and inference tasks, deployments of up to 32 accelerators, and support for both NVIDIA GPUs and Ascend NPUs. Agents start from official recipe baselines and can modify the software stack at engine, kernel, and instruction levels, with an OCI image-layer submission protocol ensuring reproducibility and independent evaluation measuring end-to-end performance under quality constraints. Results show 66.2% of tasks exceed 1.10x verified speedup, while about one-third fall short—indicating agents can already deliver verifiable gains at production scale, but reliably translating local optimizations into end-to-end benefits remains the main challenge.

RSIBench-Infra Benchmark Overview

Can agents autonomously optimize their own training and inference infrastructure the same way they optimize ordinary software? This is a very practical aspect of recursive self-improvement. Most existing agent-optimization benchmarks focus only on single software layers like kernels, and it remains unclear whether agents can deliver end-to-end performance gains under production-grade workloads. RSIBench-Infra is designed precisely to answer this question.

From Single Software Layers to Production-Grade End-to-End

RSIBench-Infra is a benchmark comprising 20 training and inference tasks, covering multiple model families and modalities, with deployment scales reaching up to 32 accelerators and support for both NVIDIA GPUs and Ascend NPUs. This scale distinguishes it from evaluations that merely need to pass in single-GPU or small-scale environments—production-grade means that communication, scheduling, memory, compilation, and other components all come into play.

Agents start from runnable baselines derived from official recipes rather than writing code from scratch. They can modify the software stack at three levels: engine, kernel, and instruction. In other words, agents aren't just tuning a single CUDA kernel; they can touch multiple abstraction layers across the entire technology stack, which is much closer to how real engineering teams optimize.

Recursive Self-Improvement refers to an agent's closed-loop ability to improve its own capabilities (including the software and hardware infrastructure it depends on for operation), a key topic in AI safety and capability evaluation. In the infrastructure context, it means agents can autonomously analyze bottlenecks in training/inference systems, propose modifications, verify results, and iterate. RSIBench-Infra grounds this grand concept in measurable engineering tasks: agents optimize the software stack on a given compute cluster, and the benchmark measures their end-to-end performance gains. This is stricter than "agents write code" because it requires modifications to remain effective under real workloads and multi-accelerator communication, not merely passing unit tests in isolated environments.

How the Submission Protocol and Quality Constraints Ensure Comparability

To ensure reproducibility, the benchmark uses an OCI image-layer submission protocol to capture agent modifications, while independent evaluation measures end-to-end performance under task-specific quality constraints. These two design choices are critical: image-layer submission allows any change to the software stack to be fully recorded and reconstructed, avoiding the reproducibility trap of "it runs fast on my machine but won't even start on yours"; quality constraints prevent agents from trading accuracy for speed.

Independent evaluation means performance numbers are not self-reported by agents but measured by the benchmark under controlled conditions—a fundamental difference from evaluations that rely on self-reported metrics, and one that increases the credibility of conclusions.

OCI Image Layer is part of the container image specification; each layer represents a set of incremental filesystem changes that, when stacked, can fully reconstruct a container environment. In RSIBench-Infra, any modification an agent makes to the software stack (such as swapping operator implementations, adjusting compilation flags, or modifying scheduling logic) is packaged as a new image layer that, combined with the baseline image, precisely reproduces the runtime environment. Compared to submitting only a code diff, image layers also capture binary dependencies, environment variables, and build artifacts, avoiding reproduction failures caused by "same code but different environment." Independent evaluators simply pull the image layer and run standardized tests to obtain comparable performance numbers—this is the technical foundation of the benchmark's credibility.

What Does 66.2% of Tasks Exceeding 1.10x Speedup Mean?

Among all task results, 66.2% exceeded the verified 1.10x speedup. This number should be viewed from two sides.

On the positive side, current agents can already deliver verifiable performance gains on production-scale training and inference workloads, not just toy-level kernel micro-optimizations. Two-thirds of tasks crossing the 1.10x line shows that the capability is widespread, not a fluke of individual tasks.

But the other side is equally clear: roughly one-third of tasks still fall short of this threshold. The paper's authors candidly note that identifying effective optimization strategies and reliably translating local improvements into end-to-end gains remains difficult. Local optimization does not equal global gains—a faster kernel may not offset added compilation overhead or communication costs, and such problems are amplified at production scale.

The 1.10x Speedup Threshold is the benchmark's "verified speedup" bar, designed to distinguish statistical noise from genuine performance improvements. In distributed training and inference, end-to-end latency is affected by communication, scheduling, kernel launch overhead, and many other factors, and measurements themselves fluctuate. Setting a 1.10x line and requiring independent re-testing filters out measurement error or serendipitous gains from one-off tuning. The threshold also echoes engineering practice: production systems typically require optimizations to deliver at least 10% net benefit to justify added complexity. Therefore, 66.2% of tasks crossing this line means most optimizations hold up both statistically and in engineering terms.

Implications for the Agent Self-Improvement Roadmap

RSIBench-Infra's conclusion is measured: current agents can improve production-scale training and inference performance, but they are still far from "reliably self-optimizing infrastructure." This calibrates the recursive self-improvement narrative—infrastructure optimization is not a single skill but a systems engineering challenge requiring trade-offs across engine, kernel, and instruction layers. Agents' capabilities at individual points have emerged, but system-level strategic choices remain a weakness.

The practical engineering takeaway is that placing agents in real multi-accelerator environments and measuring with reproducible submission protocols and independent evaluation reflects real-world usability far better than chasing metrics in controlled small environments. The coexistence of Ascend NPUs and NVIDIA GPUs also makes the conclusions more relevant to heterogeneous compute scenarios.

Heterogeneous Coexistence of Ascend NPU and NVIDIA GPU is a setting that distinguishes this benchmark from most agent-optimization evaluations. The two accelerator types differ significantly in programming models (such as Ascend's CANN and NVIDIA's CUDA), operator libraries, and communication primitives. Agents must adapt optimization strategies to different backends rather than relying on experience from a single ecosystem. This makes the conclusions more relevant to real heterogeneous compute scenarios: production often mixes multiple accelerators, and optimization solutions need cross-backend portability or separate adaptation. The heterogeneous setting also increases task difficulty—some tasks falling short of the 1.10x threshold may be related to this, suggesting agents still have room to improve in cross-stack knowledge transfer.

FAQ

How does RSIBench-Infra differ from previous agent-optimization benchmarks?

Previous benchmarks mostly focus on single software layers like kernels, while RSIBench-Infra emphasizes end-to-end performance at production scale, with tasks covering both training and inference, deployments of up to 32 accelerators, and support for both NVIDIA GPUs and Ascend NPUs.

At which levels can agents modify the software stack?

They can modify at all three levels—engine, kernel, and instruction—starting from runnable baselines derived from official recipes.

What does the 66.2% speedup ratio indicate?

It shows that most tasks can achieve verified speedups exceeding 1.10x, but some tasks still fall short, and translating local optimizations into end-to-end gains remains difficult.

#RSIBench-Infra#agent optimization benchmark#recursive self-improvement#training and inference infrastructure#Ascend NPU#NVIDIA GPU#end-to-end performance
Share:

Related articles