RSIBench-Infra
A benchmark for agents to autonomously optimize production-grade training and inference infrastructure, comprising 20 tasks spanning multiple model families and modalities, supporting multiple accelerator scales, and employing an OCI image layer submission protocol with independent evaluation.
Timeline (last 90 days)
RSIBench-Infra is a benchmark containing 20 training and inference tasks, covering multiple model families and modalities, with deployment scale reaching up to 32 accelerators.
RSIBench-Infra supports two types of accelerators: NVIDIA GPUs and Ascend NPUs.
Among all task results in RSIBench-Infra, 66.2% of tasks exceeded the validated 1.10x speedup.
RSIBench-Infra uses an OCI image layer submission protocol to capture the modifications made by agents and conducts independent evaluation.
In RSIBench-Infra, the agent's starting point is a runnable baseline from the official recipe, rather than writing code from scratch.
In RSIBench-Infra, agents can modify the software stack at three levels: engine, kernel, and instruction.
Independent evaluation in RSIBench-Infra measures end-to-end performance under task-specific quality constraints to prevent agents from trading accuracy for speed.
The authors point out that identifying effective optimization strategies and stably converting local improvements into end-to-end gains remains difficult.
The conclusion of RSIBench-Infra is that current agents can improve production-scale training and inference performance, but they are still far from stably self-optimizing infrastructure.
The 1.10x speedup threshold is the validated speedup threshold set by the benchmark, used to distinguish statistical noise from real performance improvements.
All Facts (10)
RSIBench-Infra is a benchmark containing 20 training and inference tasks, covering multiple model families and modalities, with deployment scale reaching up to 32 accelerators.
50%UnverifiedRSIBench-Infra supports two types of accelerators: NVIDIA GPUs and Ascend NPUs.
50%UnverifiedAmong all task results in RSIBench-Infra, 66.2% of tasks exceeded the validated 1.10x speedup.
50%UnverifiedRSIBench-Infra uses an OCI image layer submission protocol to capture the modifications made by agents and conducts independent evaluation.
50%UnverifiedIn RSIBench-Infra, the agent's starting point is a runnable baseline from the official recipe, rather than writing code from scratch.
50%UnverifiedIn RSIBench-Infra, agents can modify the software stack at three levels: engine, kernel, and instruction.
50%UnverifiedIndependent evaluation in RSIBench-Infra measures end-to-end performance under task-specific quality constraints to prevent agents from trading accuracy for speed.
50%UnverifiedThe authors point out that identifying effective optimization strategies and stably converting local improvements into end-to-end gains remains difficult.
50%UnverifiedThe conclusion of RSIBench-Infra is that current agents can improve production-scale training and inference performance, but they are still far from stably self-optimizing infrastructure.
50%UnverifiedThe 1.10x speedup threshold is the validated speedup threshold set by the benchmark, used to distinguish statistical noise from real performance improvements.
50%