[KongchangAI]
Benchmark

ServerBench

A benchmark for evaluating the ability of coding agents to design LLM inference serving systems. Agents must complete system design, implementation, and operational decisions under constraints such as latency and cost, and are evaluated using stateful replay and independent verification.

Timeline (last 90 days)

Nov 5

ServerBench is a benchmark that requires coding agents to design LLM inference serving systems under latency constraints, session dependencies and correctness requirements.

Unverified50%
Nov 5

ServerBench uses 'dollars per dollar' as the core objective function, i.e., the value of latency-compliant tokens produced per dollar of GPU cost.

Unverified50%
Nov 5

ServerBench uses stateful replay to preserve real session dependencies, ensuring that the request stream is contextually coherent.

Unverified50%
Nov 5

ServerBench requires submitted systems to undergo independent verification to prevent agents from passing tests through hard-coding, cache cheating or bypassing key logic.

Unverified50%
Nov 5

In ServerBench tests, Claude Sonnet 5.5 achieved 2.97 dollars per dollar on Gemma E4B.

Unverified50%
Nov 5

In ServerBench tests, 22 of Claude Sonnet 5.5's 33 runs produced verified systems, 13 of which beat the strongest baseline.

Unverified50%
Nov 5

ServerBench tests covered 11 coding agent models and three deployment configurations.

Unverified50%
Nov 5

ServerBench uses vLLM as the baseline reference implementation.

Unverified50%
Nov 5

High-performing agents tend to jointly optimize capacity planning (such as batch size and memory allocation) and load selection (such as request priority and routing strategy).

Unverified50%
Nov 5

Agents need to interpret performance feedback by designing workload-related experiments in order to understand the causal relationships behind latency changes.

Unverified50%

2 more timeline events

All Facts (12)

Unverified

ServerBench is a benchmark that requires coding agents to design LLM inference serving systems under latency constraints, session dependencies and correctness requirements.

50%
Unverified

ServerBench uses 'dollars per dollar' as the core objective function, i.e., the value of latency-compliant tokens produced per dollar of GPU cost.

50%
Unverified

ServerBench uses stateful replay to preserve real session dependencies, ensuring that the request stream is contextually coherent.

50%
Unverified

ServerBench requires submitted systems to undergo independent verification to prevent agents from passing tests through hard-coding, cache cheating or bypassing key logic.

50%
Unverified

In ServerBench tests, Claude Sonnet 5.5 achieved 2.97 dollars per dollar on Gemma E4B.

50%
Unverified

In ServerBench tests, 22 of Claude Sonnet 5.5's 33 runs produced verified systems, 13 of which beat the strongest baseline.

50%
Unverified

ServerBench tests covered 11 coding agent models and three deployment configurations.

50%
Unverified

ServerBench uses vLLM as the baseline reference implementation.

50%
Unverified

High-performing agents tend to jointly optimize capacity planning (such as batch size and memory allocation) and load selection (such as request priority and routing strategy).

50%
Unverified

Agents need to interpret performance feedback by designing workload-related experiments in order to understand the causal relationships behind latency changes.

50%
Unverified

Reliable delivery requires verification of the execution paths introduced by optimization, otherwise a system may pass the benchmark by luck yet fail under real workloads.

50%
Unverified

ServerBench is described as the first systematic experimental platform for studying the agent self-optimization loop.

50%

Source Articles