Zoroval
A team specializing in LLMOps and AI coding quality assurance. The team pioneered the practice of replacing manual code review with evaluation-gate-based methods for verifying changes that affect LLM behavior.
Timeline (last 90 days)
Data from the Zoroval team shows that the median agent-generated PR changes only 133 lines of code.
Approximately one in every eight agent changes modifies more than 1,000 lines of code.
On July 13, Zoroval's judge model missed all failures that had been manually confirmed, running 1,138 judge calls and returning no for all four manually confirmed failures in the golden dataset without reporting any errors.
The root cause of the incident was a modification to the context constructor, while the team's evaluations were only triggered by changes to the model or prompt templates, and modifications to the context constructor never triggered an evaluation.
Zoroval uses an evaluation gate mechanism called the Intelligence Change Gate, which operates in five steps: declare a protected scope, fingerprint exact changes, require evaluation reports matching the fingerprint, automatically expire evidence, and retain visible broken-window-style owner overrides.
Zoroval's staging branch has 43 evaluation reports, of which 9 PRs used owner overrides.
A judge model tested by Zoroval that was approximately six times cheaper entered production after passing an evaluation containing both positive and negative examples, reducing monitoring costs by approximately 83%.
Zoroval's gate is distributed across four layers—local, PR, staging, and production—and a change must pass all of them.
All Facts (8)
Zoroval's gate is distributed across four layers—local, PR, staging, and production—and a change must pass all of them.
50%UnverifiedApproximately one in every eight agent changes modifies more than 1,000 lines of code.
50%UnverifiedData from the Zoroval team shows that the median agent-generated PR changes only 133 lines of code.
50%UnverifiedThe root cause of the incident was a modification to the context constructor, while the team's evaluations were only triggered by changes to the model or prompt templates, and modifications to the context constructor never triggered an evaluation.
50%UnverifiedZoroval uses an evaluation gate mechanism called the Intelligence Change Gate, which operates in five steps: declare a protected scope, fingerprint exact changes, require evaluation reports matching the fingerprint, automatically expire evidence, and retain visible broken-window-style owner overrides.
50%UnverifiedZoroval's staging branch has 43 evaluation reports, of which 9 PRs used owner overrides.
50%UnverifiedA judge model tested by Zoroval that was approximately six times cheaper entered production after passing an evaluation containing both positive and negative examples, reducing monitoring costs by approximately 83%.
50%UnverifiedOn July 13, Zoroval's judge model missed all failures that had been manually confirmed, running 1,138 judge calls and returning no for all four manually confirmed failures in the golden dataset without reporting any errors.
50%