CAFA-IVR
A framework for evaluating release gates of AI systems that includes baseline/candidate comparison, metric checks and fail-closed semantics, and supports distinguishing check failures from tool faults through exit codes.
Timeline (last 90 days)
In CAFA-IVR framework v1.0, when the summary CSV contains missing, non-numeric or non-finite metrics, the gate cannot evaluate the metrics yet lets the pipeline continue.
The CAFA-IVR v1.1.0 changelog states: baseline/candidate comparison now fails closed on missing, non-numeric, non-finite or empty enabled metrics, and pipelines that previously let malformed input through now exit with status code 2.
In CAFA-IVR v1.1.0, if a metric is missing, cannot be parsed as a number, parses to infinity or NaN, or the summary file is unreadable or empty, any of these cases will print FAIL, identify the offending input and exit with status code 2.
The pilot dataset bundled with the CAFA-IVR framework (reproduction data for 360 trials) has no key-entity annotations and reports ceer: null in the summary, so it cannot pass its own CEER gate.
CAFA-IVR v1.1.0 provides the --no-ceer-gate option for explicitly opting out of the CEER check.
CAFA-IVR's compare command accepts the --max-wer-delta parameter, but its default value is None, i.e. the WER regression gate is disabled.
The quantities actually gated by the CAFA-IVR framework are the deltas of ASR-IFR and CEER, with WER serving only as an optional diagnostic item.
The CAFA-IVR README states that the default numeric gates are demo examples of the tool rather than universal acceptance criteria, and production teams should govern thresholds themselves according to risk level and historical baselines.
In CAFA-IVR, the default value of --max-asr-ifr-delta is 0.02 and the default value of --max-ceer-delta is 0.01.
All Facts (9)
In CAFA-IVR, the default value of --max-asr-ifr-delta is 0.02 and the default value of --max-ceer-delta is 0.01.
50%UnverifiedIn CAFA-IVR framework v1.0, when the summary CSV contains missing, non-numeric or non-finite metrics, the gate cannot evaluate the metrics yet lets the pipeline continue.
50%UnverifiedIn CAFA-IVR v1.1.0, if a metric is missing, cannot be parsed as a number, parses to infinity or NaN, or the summary file is unreadable or empty, any of these cases will print FAIL, identify the offending input and exit with status code 2.
50%UnverifiedThe pilot dataset bundled with the CAFA-IVR framework (reproduction data for 360 trials) has no key-entity annotations and reports ceer: null in the summary, so it cannot pass its own CEER gate.
50%UnverifiedCAFA-IVR v1.1.0 provides the --no-ceer-gate option for explicitly opting out of the CEER check.
50%UnverifiedCAFA-IVR's compare command accepts the --max-wer-delta parameter, but its default value is None, i.e. the WER regression gate is disabled.
50%UnverifiedThe quantities actually gated by the CAFA-IVR framework are the deltas of ASR-IFR and CEER, with WER serving only as an optional diagnostic item.
50%UnverifiedThe CAFA-IVR README states that the default numeric gates are demo examples of the tool rather than universal acceptance criteria, and production teams should govern thresholds themselves according to risk level and historical baselines.
50%UnverifiedThe CAFA-IVR v1.1.0 changelog states: baseline/candidate comparison now fails closed on missing, non-numeric, non-finite or empty enabled metrics, and pipelines that previously let malformed input through now exit with status code 2.
50%