AgentSeism
An open-source continuous integration evaluation framework for AI agents. It applies statistical experimental design methods to address the non-deterministic nature of agent outputs, supporting multi-dimensional capability testing and multiple testing correction to produce reliable and reproducible assessments of agent performance over time.
Timeline (last 90 days)
In a real-world case, a coding agent's success rate dropped from 92% to 88% without any changes, which was a false positive rather than a true capability regression.
The author deliberately introduced a genuine severe regression that reduced an agent's success rate from 92% to 52%, yet the statistical rule originally designed for this purpose passed the PR because the regression was concentrated in a small number of tasks rather than being uniformly distributed.
Agent evaluation should be decomposed into two independent questions: whether the reliability of the entire suite has generally degraded, and whether a previously stable capability has suddenly collapsed — a single average-based rule cannot answer both questions simultaneously.
AgentSeism introduces a third explicit state, 'insufficient evidence,' which honestly acknowledges uncertainty when the data cannot support a reliable judgment, rather than issuing a binary pass/fail.
All Facts (4)
In a real-world case, a coding agent's success rate dropped from 92% to 88% without any changes, which was a false positive rather than a true capability regression.
50%UnverifiedThe author deliberately introduced a genuine severe regression that reduced an agent's success rate from 92% to 52%, yet the statistical rule originally designed for this purpose passed the PR because the regression was concentrated in a small number of tasks rather than being uniformly distributed.
50%UnverifiedAgent evaluation should be decomposed into two independent questions: whether the reliability of the entire suite has generally degraded, and whether a previously stable capability has suddenly collapsed — a single average-based rule cannot answer both questions simultaneously.
50%UnverifiedAgentSeism introduces a third explicit state, 'insufficient evidence,' which honestly acknowledges uncertainty when the data cannot support a reliable judgment, rather than issuing a binary pass/fail.
50%