SWE-Serve
An AI coding agent evaluation benchmark proposed by NVIDIA, focused on inference-serving software scenarios. It assesses the functional correctness and runtime performance of code patches by simulating real serving environments—including model loading, concurrent request handling, and KV cache management—addressing the limitation of traditional benchmarks that only verify test-pass rates.
Timeline (last 90 days)
NVIDIA has proposed SWE-Serve to reveal the gap where AI coding agents pass local tests but fail in live serving environments.
SWE-Serve validates agent patches in real serving environments by simulating the complete workflow of a server loading real models, receiving, and processing requests.
For inference serving, a patch that is functionally correct but causes a 30% drop in throughput is actually unacceptable, and performance should be incorporated as an evaluation dimension.
Understanding system runtime behavior and anticipating how changes perform under load are weak points of current large model coding capabilities.
All Facts (4)
SWE-Serve validates agent patches in real serving environments by simulating the complete workflow of a server loading real models, receiving, and processing requests.
50%UnverifiedFor inference serving, a patch that is functionally correct but causes a 30% drop in throughput is actually unacceptable, and performance should be incorporated as an evaluation dimension.
50%UnverifiedUnderstanding system runtime behavior and anticipating how changes perform under load are weak points of current large model coding capabilities.
50%UnverifiedNVIDIA has proposed SWE-Serve to reveal the gap where AI coding agents pass local tests but fail in live serving environments.
50%