ARCI
An open-source regression gate tool for AI agents that supports fault injection and pre-registered experiments and issues one of four verdicts: PASS, BLOCK, INCONCLUSIVE or ERROR. It is used for model selection and validation of configuration changes.
Timeline (last 90 days)
The core addition in ARCI v2.0 is the LangChain adapter, which can run create_agent configurations inside the fault-injection regression gate.
ARCI must fix the task set and decision rules before running, and after running outputs only one of four verdicts: PASS, BLOCK, INCONCLUSIVE, or ERROR.
The ARCI LangChain adapter example file is about 50 lines, with low integration cost.
In the pre-registered experiment, the 12B model passed all 96 trials on the 24-task order-desk test suite (96/96).
In the pre-registered experiment, the 4B model passed 70 of the same 96 trials.
Under fault-injection conditions, the 12B model's injected faults actually triggered 24/24 times, while the 4B model triggered 18 to 20/24 times.
The authors' pre-registered prediction was that replacing the 12B model with the 4B model would PASS, but the actual result was BLOCK.
ARCI currently explicitly reports errors rather than silently passing for these scenarios: asynchronous agents, interrupts, tools with injection parameters, and tool artifacts.
The experiment was completed on a single laptop using Ollama, took 18 minutes, and required no API key.
The authors state that the conclusions apply only to the registered test suite, and that this is a single run of 96 paired trials and cannot be used to infer statistical significance.
2 more timeline events
All Facts (12)
ARCI's install command is pip install "agent-reliability-ci[langchain]", the initialization command is arci init --langchain, and the run command is arci run arci.yaml.
50%UnverifiedARCI is an open-source regression gate for agents.
50%UnverifiedThe ARCI LangChain adapter example file is about 50 lines, with low integration cost.
50%UnverifiedIn the pre-registered experiment, the 12B model passed all 96 trials on the 24-task order-desk test suite (96/96).
50%UnverifiedIn the pre-registered experiment, the 4B model passed 70 of the same 96 trials.
50%UnverifiedUnder fault-injection conditions, the 12B model's injected faults actually triggered 24/24 times, while the 4B model triggered 18 to 20/24 times.
50%UnverifiedARCI must fix the task set and decision rules before running, and after running outputs only one of four verdicts: PASS, BLOCK, INCONCLUSIVE, or ERROR.
50%UnverifiedARCI currently explicitly reports errors rather than silently passing for these scenarios: asynchronous agents, interrupts, tools with injection parameters, and tool artifacts.
50%UnverifiedThe experiment was completed on a single laptop using Ollama, took 18 minutes, and required no API key.
50%UnverifiedThe authors state that the conclusions apply only to the registered test suite, and that this is a single run of 96 paired trials and cannot be used to infer statistical significance.
50%UnverifiedThe authors' pre-registered prediction was that replacing the 12B model with the 4B model would PASS, but the actual result was BLOCK.
50%UnverifiedThe core addition in ARCI v2.0 is the LangChain adapter, which can run create_agent configurations inside the fault-injection regression gate.
50%