ServerBench
A benchmark for evaluating the ability of coding agents to design LLM inference serving systems. Agents must complete system design, implementation, and operational decisions under constraints such as latency and cost, and are evaluated using stateful replay and independent verification.
Timeline (last 90 days)
ServerBench is a benchmark that requires coding agents to design LLM inference serving systems under latency constraints, session dependencies and correctness requirements.
ServerBench uses 'dollars per dollar' as the core objective function, i.e., the value of latency-compliant tokens produced per dollar of GPU cost.
ServerBench uses stateful replay to preserve real session dependencies, ensuring that the request stream is contextually coherent.
ServerBench requires submitted systems to undergo independent verification to prevent agents from passing tests through hard-coding, cache cheating or bypassing key logic.
In ServerBench tests, Claude Sonnet 5.5 achieved 2.97 dollars per dollar on Gemma E4B.
In ServerBench tests, 22 of Claude Sonnet 5.5's 33 runs produced verified systems, 13 of which beat the strongest baseline.
ServerBench tests covered 11 coding agent models and three deployment configurations.
ServerBench uses vLLM as the baseline reference implementation.
High-performing agents tend to jointly optimize capacity planning (such as batch size and memory allocation) and load selection (such as request priority and routing strategy).
Agents need to interpret performance feedback by designing workload-related experiments in order to understand the causal relationships behind latency changes.
2 more timeline events
All Facts (12)
ServerBench is a benchmark that requires coding agents to design LLM inference serving systems under latency constraints, session dependencies and correctness requirements.
50%UnverifiedServerBench uses 'dollars per dollar' as the core objective function, i.e., the value of latency-compliant tokens produced per dollar of GPU cost.
50%UnverifiedServerBench uses stateful replay to preserve real session dependencies, ensuring that the request stream is contextually coherent.
50%UnverifiedServerBench requires submitted systems to undergo independent verification to prevent agents from passing tests through hard-coding, cache cheating or bypassing key logic.
50%UnverifiedIn ServerBench tests, Claude Sonnet 5.5 achieved 2.97 dollars per dollar on Gemma E4B.
50%UnverifiedIn ServerBench tests, 22 of Claude Sonnet 5.5's 33 runs produced verified systems, 13 of which beat the strongest baseline.
50%UnverifiedServerBench tests covered 11 coding agent models and three deployment configurations.
50%UnverifiedServerBench uses vLLM as the baseline reference implementation.
50%UnverifiedHigh-performing agents tend to jointly optimize capacity planning (such as batch size and memory allocation) and load selection (such as request priority and routing strategy).
50%UnverifiedAgents need to interpret performance feedback by designing workload-related experiments in order to understand the causal relationships behind latency changes.
50%UnverifiedReliable delivery requires verification of the execution paths introduced by optimization, otherwise a system may pass the benchmark by luck yet fail under real workloads.
50%UnverifiedServerBench is described as the first systematic experimental platform for studying the agent self-optimization loop.
50%