[KongchangAI]
ProductARCI v2.0

ARCI

An open-source regression gate tool for AI agents that supports fault injection and pre-registered experiments and issues one of four verdicts: PASS, BLOCK, INCONCLUSIVE or ERROR. It is used for model selection and validation of configuration changes.

Timeline (last 90 days)

Nov 11

The core addition in ARCI v2.0 is the LangChain adapter, which can run create_agent configurations inside the fault-injection regression gate.

Unverified50%
Nov 11

ARCI must fix the task set and decision rules before running, and after running outputs only one of four verdicts: PASS, BLOCK, INCONCLUSIVE, or ERROR.

Unverified50%
Nov 11

The ARCI LangChain adapter example file is about 50 lines, with low integration cost.

Unverified50%
Nov 11

In the pre-registered experiment, the 12B model passed all 96 trials on the 24-task order-desk test suite (96/96).

Unverified50%
Nov 11

In the pre-registered experiment, the 4B model passed 70 of the same 96 trials.

Unverified50%
Nov 11

Under fault-injection conditions, the 12B model's injected faults actually triggered 24/24 times, while the 4B model triggered 18 to 20/24 times.

Unverified50%
Nov 11

The authors' pre-registered prediction was that replacing the 12B model with the 4B model would PASS, but the actual result was BLOCK.

Unverified50%
Nov 11

ARCI currently explicitly reports errors rather than silently passing for these scenarios: asynchronous agents, interrupts, tools with injection parameters, and tool artifacts.

Unverified50%
Nov 11

The experiment was completed on a single laptop using Ollama, took 18 minutes, and required no API key.

Unverified50%
Nov 11

The authors state that the conclusions apply only to the registered test suite, and that this is a single run of 96 paired trials and cannot be used to infer statistical significance.

Unverified50%

2 more timeline events

All Facts (12)

Unverified

ARCI's install command is pip install "agent-reliability-ci[langchain]", the initialization command is arci init --langchain, and the run command is arci run arci.yaml.

50%
Unverified

ARCI is an open-source regression gate for agents.

50%
Unverified

The ARCI LangChain adapter example file is about 50 lines, with low integration cost.

50%
Unverified

In the pre-registered experiment, the 12B model passed all 96 trials on the 24-task order-desk test suite (96/96).

50%
Unverified

In the pre-registered experiment, the 4B model passed 70 of the same 96 trials.

50%
Unverified

Under fault-injection conditions, the 12B model's injected faults actually triggered 24/24 times, while the 4B model triggered 18 to 20/24 times.

50%
Unverified

ARCI must fix the task set and decision rules before running, and after running outputs only one of four verdicts: PASS, BLOCK, INCONCLUSIVE, or ERROR.

50%
Unverified

ARCI currently explicitly reports errors rather than silently passing for these scenarios: asynchronous agents, interrupts, tools with injection parameters, and tool artifacts.

50%
Unverified

The experiment was completed on a single laptop using Ollama, took 18 minutes, and required no API key.

50%
Unverified

The authors state that the conclusions apply only to the registered test suite, and that this is a single run of 96 paired trials and cannot be used to infer statistical significance.

50%
Unverified

The authors' pre-registered prediction was that replacing the 12B model with the 4B model would PASS, but the actual result was BLOCK.

50%
Unverified

The core addition in ARCI v2.0 is the LangChain adapter, which can run create_agent configurations inside the fault-injection regression gate.

50%

Source Articles