Agent Trials
An open-source tool for testing the real capabilities of coding agents. It points at an arbitrary repository, defines tasks in JSON, runs the agent CLI on a disposable copy, restores the verification script, and executes it. It provides reproducible evidence rather than leaderboard rankings.
Timeline (last 90 days)
Agent Trials is an open-source tool created by a Reddit developer (GitHub user princejain756) for running Coding Agents on developers' own repositories for evaluation.
Agent Trials runs a one-time copy on each Agent CLI in the project, rather than operating directly on the original repository.
The core design of Agent Trials is that after the Agent finishes running, the validation script is restored from the source repository before execution, to prevent the Agent from modifying the validation script to cheat.
The initial version of Agent Trials did not have a validation script restoration step, and the author fixed it after an audit.
Agent Trials is explicitly not a leaderboard, but a smoke test, and its scores and medians fluctuate between runs.
Agent Trials is mainly used to solve two problems: catching false successes where the CLI exit code is 0 but nothing was actually done, and observing cost drift.
In a test containing three tasks (fix-clamp, divide-guard, slug-test), the three models Opus, Sonnet, and Haiku all scored 3/3.
On the divide-guard task, Haiku took 90.5 seconds and Opus took 22.8 seconds, making Haiku nearly four times slower than Opus.
Agent Trials has a zero-dependency feature and can be run immediately after cloning, without installing additional packages.
Agent Trials' task definitions use pure JSON format, which is human-readable and machine-parseable.
2 more timeline events
All Facts (12)
Agent Trials is an open-source tool created by a Reddit developer (GitHub user princejain756) for running Coding Agents on developers' own repositories for evaluation.
50%UnverifiedAgent Trials runs a one-time copy on each Agent CLI in the project, rather than operating directly on the original repository.
50%UnverifiedThe core design of Agent Trials is that after the Agent finishes running, the validation script is restored from the source repository before execution, to prevent the Agent from modifying the validation script to cheat.
50%UnverifiedThe initial version of Agent Trials did not have a validation script restoration step, and the author fixed it after an audit.
50%UnverifiedAgent Trials is explicitly not a leaderboard, but a smoke test, and its scores and medians fluctuate between runs.
50%UnverifiedAgent Trials is mainly used to solve two problems: catching false successes where the CLI exit code is 0 but nothing was actually done, and observing cost drift.
50%UnverifiedIn a test containing three tasks (fix-clamp, divide-guard, slug-test), the three models Opus, Sonnet, and Haiku all scored 3/3.
50%UnverifiedOn the divide-guard task, Haiku took 90.5 seconds and Opus took 22.8 seconds, making Haiku nearly four times slower than Opus.
50%UnverifiedAgent Trials has a zero-dependency feature and can be run immediately after cloning, without installing additional packages.
50%UnverifiedAgent Trials' task definitions use pure JSON format, which is human-readable and machine-parseable.
50%UnverifiedAgent Trials supports any Coding Agent that provides a CLI interface.
50%UnverifiedAgent Trials and SWE-bench are complementary: the former answers whether an Agent can do work on a specific project, while the latter answers how models rank in capability on general programming tasks.
50%