Infrathon Benchmark: AI Agents Optimize Real Infrastructure Over Long Horizons

Infrathon proves with a 24-hour long-horizon benchmark that short-horizon evaluations misjudge model rankings and underestimate optimization gains.
Infrathon builds a long-horizon benchmark for real AI infrastructure optimization, running six frontier models across 150 runs with a 24-hour budget each and over 10,000 GPU hours total. It finds model rankings change with the evaluation horizon, with 135 of 150 runs still improving late in the budget—so short-horizon evaluations underestimate both gains and model differences.
Why Do We Need Long-Horizon Evaluation?
Current evaluations of language model agents mostly focus on short-horizon tasks, such as code generation, question answering, or simple tool calls. However, real-world AI infrastructure optimization often requires hours or even days of continuous iteration—tuning codec parameters, searching cache strategies, and balancing compute versus storage overhead. Can short-horizon benchmarks reflect an agent's true capabilities on such tasks? A new study called Infrathon answers with a resounding no.
The study builds a long-horizon benchmark for real AI infrastructure optimization, covering diverse tasks at different levels, and evaluates six frontier models across 150 runs, each given a 24-hour budget and consuming over 10,000 GPU hours in total. The core finding is: model rankings change with the evaluation horizon, and short-horizon evaluations simultaneously underestimate optimization gains and the relative performance differences between models.

Key Finding: The Longer the Horizon, the More the Landscape Shifts
Rankings Are Not Set in Stone
Of the 150 runs, 135 still achieved further improvements in the latter half of the budget. This means that observing only the first few hours not only misses substantial optimization headroom but can also lead to incorrect judgments about which model is stronger. The study explicitly points out that short-horizon evaluations simultaneously overlook optimization gains and changes in relative model performance. For engineering teams that need to select models, this suggests a risk: using short-task scores to predict long-horizon performance is risky.
Short-horizon evaluations typically refer to tasks that can be completed in minutes to hours and that rely on a single or a small number of interactions to produce a score, such as HumanEval-style code generation or single-turn question answering. Long-horizon evaluations require agents to continuously interact with the environment, run repeated experiments, and accumulate improvements over a time span of hours to days. The key difference is not just the length of time, but that long-horizon tasks naturally involve a closed loop of exploration, failure, correction, and re-evaluation; agents must manage experiment records, allocate budgets, and decide when to converge on their own. Therefore, short-horizon scores measure the upper bound of single-step capability, while long-horizon scores measure sustained optimization capability. The two are not necessarily correlated, which is the fundamental reason the study emphasizes that model rankings change with the evaluation window.
Agents Can Surpass Human-Designed Solutions
Even more striking, agents can discover solutions that outperform human-designed baselines. Take the KV-cache compression task: Claude Opus 5.5 raised the average compression rate from 6.82% with the human-designed Quant-VideoGen codec to 19.86%. This result shows that over a sufficiently long optimization horizon, frontier agents have the potential to explore non-intuitive optimization paths.
KV-cache (key-value cache) is the mechanism in Transformer inference that stores the Key and Value vectors of historical tokens, used to avoid recomputing attention during autoregressive generation. As context grows, KV-cache memory usage rises linearly, becoming one of the main bottlenecks for long-context inference. Common methods for compressing KV-cache include quantization (reducing the bit width of each element), low-rank decomposition, sparsification, and selective dropping by head or layer. Quant-VideoGen is the human-designed codec used as the control in the study, and its 6.82% average compression rate represents the baseline level of human experts on this task. A higher compression rate means supporting longer contexts or larger batches under the same memory, so the improvement from 6.82% to 19.86% is practically meaningful in engineering terms.
Three Major Obstacles to Sustained Improvement
By analyzing agent trajectories, the study identifies three types of problems that hinder sustained improvement: task violations, unreliable tests, and premature completion. Task violations mean the agent deviates from task constraints; unreliable tests mean self-built verification methods may give incorrect signals; and premature completion means the agent stops iterating when there is still optimization headroom. These problems are often masked in short-horizon evaluations but become performance bottlenecks in long-horizon tasks.
Task violations, unreliable tests, and premature completion essentially reflect long-horizon agents' lack of reliable self-verification mechanisms. In short tasks, agents can rely on external unit tests or standard answers to immediately judge correctness; in open-ended tasks such as infrastructure optimization, the objective function often must be measured by actually running benchmarks, so agents have to build verification pipelines themselves. Once the verification script itself has bugs or measurement noise is too high, the agent will continue optimizing along the wrong signal and may even misjudge violating actions as valid improvements. Premature completion relates to the agent's ability to estimate remaining optimization headroom: it tends to stop when metrics no longer rise noticeably, whereas gains in long-horizon tasks often come from strategy switches after a plateau. Together, these three problems point to a research direction—designing robust self-verification and budget management mechanisms for long-horizon agents.
Implications for AI Infrastructure Practice
The value of Infrathon is not just a leaderboard. It provides a framework resource for building long-horizon tasks, evaluating frontier agents, and discovering high-quality infrastructure solutions. For teams working on inference optimization, storage compression, or scheduling system development, this means agents can be incorporated into real optimization loops, rather than only used to generate code snippets.
At the same time, the study reminds us to pay attention to evaluation methodology itself: as the capability boundaries of agents continue to expand, evaluation design must keep pace with the time scale of tasks. Short and fast benchmarks remain useful, but they cannot replace the examination of sustained optimization capability.
Limitations and Outlook
The current work covers six models and specific types of infrastructure tasks, and whether the conclusions generalize to broader system optimization scenarios still needs verification. In addition, how to automatically detect task violations and unreliable tests remains an open problem for building reliable long-horizon agent systems. But one thing is certain: long-horizon, real-environment evaluation will become an important direction in research on agent capabilities.
One-line Summary
Infrathon uses a 24-hour long-horizon benchmark to show that short-horizon evaluations misjudge model rankings and underestimate optimization gains.
Paragraph Summary
Infrathon builds a long-horizon benchmark for real AI infrastructure optimization, evaluating six frontier models across 150 runs with a 24-hour budget each and over 10,000 GPU hours in total. The core finding is that model rankings change with the evaluation horizon, and 135 of the 150 runs still achieved further improvements in the latter half of the budget, indicating that short-horizon evaluations simultaneously underestimate optimization gains and relative performance differences between models. Agents can also surpass human-designed baselines; for example, Claude Opus 5.5 raised the average KV-cache compression rate from 6.82% to 19.86%. The study also identifies three major obstacles to sustained improvement—task violations, unreliable tests, and premature completion—and points out that long-horizon, real-environment evaluation will become an important direction in agent capability research.
Related articles

RePlan-VLA: Progress-Aware Replanning Breakthrough for Long-Horizon Robotic Manipulation
The RePlan-VLA framework converts VLA models' short-horizon capabilities into reliable long-horizon execution through progress-aware adaptive replanning, improving success rates by 21.9% and 10.4% on RoboCasa365 and CALVIN, and validating real-world deployment on a Franka Research 3 robot.

Geometric Bridge Flow Matching: A New Breakthrough in Blind Rigid Motion Correction
Geometric bridge flow matching enables blind rigid motion correction in head CBCT, achieving 37.1 dB PSNR and 0.27 mm RPE on simulated data in ~9.1 minutes, outperforming existing methods.

Google Releases Gemini 4 Hydrogen: 2.5x Faster Without Sacrificing Intelligence
Google releases the lightweight Gemini 4 Hydrogen: 2.5x faster task completion, AA Intelligence Index of 54 beating its flagship sibling, 7.3s single-step on Agent Arena faster than Claude and GPT rivals, four thinking levels with pay-as-you-need pricing, now available across AI Studio and more.