Release Gate Design: Why Must It Fail-Closed?

The Most Expensive Outcome for a Release Gate: A False Green

Every release gate has four possible outcomes, but only three of them are honest: a correct red (a genuine rollback was needed), a correct green (there really was no problem), and a false red (a false alarm that wastes time but prompts someone to investigate). The fourth—a false green—is the only one nobody monitors, and yet it carries the highest cost.

A false red wastes a single investigation; a false green destroys the entire premise for the gate's existence. Once it happens, you have no way of knowing how many releases went out under "a green light that meant nothing." The question is no longer "why did it roll back?" but "how many releases slipped through under a false green?" In v1.0 of the CAFA-IVR framework, this outcome was reachable: when the summary CSV contained missing, non-numeric, or non-finite metrics (for example, an upstream column rename, a corrupted export file, or a division by zero producing nan), the gate could not evaluate the metric, yet it let the pipeline proceed.

The v1.1.0 changelog is blunt about it: "Baseline/candidate comparison now fails closed on missing, non-numeric, non-finite, or empty enabled metrics. Pipelines that previously passed malformed input now exit with status code 2." That sentence alone deserves reading twice—it acknowledges the flaw, and it is precisely why this article exists.

What Fail-Closed Actually Means

In the v1.1.0 implementation, every enabled check requires a present, numeric-parseable, and finite metric in both the baseline and candidate summaries. The check in _metric_value is mechanical:

• Metric missing or empty → fail
• Metric cannot be parsed as a number → fail
• Metric parses as infinity or NaN → fail
• Summary file unreadable or empty → fail

Any of the above prints CAFA-IVR release gate: FAIL, names the specific offending input, and exits with status code 2—deliberately distinct from status code 1, so that CI systems and wrapper scripts can tell "the gate rejected this release" apart from "the test tool itself is broken." A gate that fails ambiguously will inevitably be learned and ignored; the exit code is itself part of the design.

Most interesting of all: the framework's own bundled pilot dataset fails to pass its own CEER gate. The 360-trial reproduction data lacks key entity annotations, and the summary honestly reports ceer: null. Running the comparison directly:

cafa-ivr compare \
  --baseline empirical/cafa_reproduced_summary_360.csv \
  --candidate empirical/cafa_reproduced_summary_360.csv

Output:

CAFA-IVR release gate: FAIL
 - invalid summary input: empirical/cafa_reproduced_summary_360.csv: required metric 'ceer' is missing

Exit code 2. The gate looks at the project's own reference data and refuses to pass it. This is not a bug—it is exactly the point. A gate that passes input it cannot evaluate will pass yours too.

Fail-closed vs. fail-open is a foundational pair of design patterns in reliability engineering. Fail-closed means that when the system encounters unprocessable abnormal input, a missing dependency, or an internal error, it defaults to a "reject/block" state; fail-open does the opposite, defaulting to an "allow/continue" state. The classic security analogy is a door lock: a magnetic lock that automatically locks when power is lost is fail-closed, while one that automatically unlocks is fail-open. A release gate should naturally be fail-closed, because its job is to block risky releases, not to make a "no problem" assumption for people when it cannot judge. The companion concept is exit code semantics: by Unix/CI convention, 0 means success and non-zero means failure, but different non-zero values can carry different meanings—for example, 1 means "the check did not pass" and 2 means "the check could not be executed / configuration error." Distinguishing the two lets CI systems and wrapper scripts automatically decide whether to rerun, fix the configuration, or notify the owner, avoiding misreading a tool failure as a business regression.

Explicit Opt-Out Beats Silent Skipping

Faced with a missing metric, the other option is "no entity annotations, so skip the CEER check." That is fail-open design, and its appeal is that it never offends anyone. But it is precisely the root cause of false greens happening at scale: six months later, someone removes the annotation step to save time, the gate quietly stops checking entities, and from then on every release lights up green on a check that no longer exists. Nobody notices, because the gate never complains.

So v1.1.0 requires that opt-out be explicit:

cafa-ivr compare \
  --baseline baseline/summary.csv \
  --candidate candidate/summary.csv \
  --no-ceer-gate

--no-ceer-gate declares, in the pipeline definition and in plain sight of everyone: this dataset has no entity annotations, and we accept that the entity check will not run. The difference between a skipped check and an explicitly opted-out check is one line in a configuration file—which is exactly the difference between "we decided" and "it happened." Decisions are auditable; happenings are not.

This design principle generalizes: any silent skip in a quality gate should require an explicit, visible, version-controlled opt-out. If the skip is justified, writing it down costs nothing; if the skip is covering up a regression in the process itself, the act of writing it down will catch it—usually during code review, when someone asks "why does this pipeline have the entity gate disabled?"

Explicit opt-out is a general principle in configuration management: you may turn off a check, but the act of turning it off must be written into version-controlled configuration, appear in code review, and leave a traceable record. What distinguishes it from "silent skipping" is not the outcome but auditability. In CI/CD pipelines, configuration is usually committed alongside the code as YAML or command-line arguments, so any switch like --no-xxx-gate enters the diff, gets seen by reviewers, and can be traced via blame to a specific commit and author. Fail-open skipping, by contrast, happens at runtime; the log may contain only a single easy-to-miss warning, and over time nobody knows that a check has become a mere formality. In software engineering, tests or checks that have long gone unexecuted yet still show as "passing" are called zombie checks, and they are the most common vehicle for false greens.

The WER Gate That Is Off by Default

One more implementation detail ties the whole series together. The compare command accepts --max-wer-delta—the WER regression gate does exist, but its default is None, i.e., disabled. This framework argued in the first article that WER is the wrong unit of measurement, so by default it does not use WER to gate releases. What actually gates releases is the delta in ASR-IFR and CEER—whether the speech path broke more tests, whether entity corruption increased—while WER serves only as an optional diagnostic for teams that want it.

This is not an oversight; it compiles the argument into the default. Defaults are the strongest statement a tool can make—most users never change them—so they should encode the author's genuine beliefs. The author believes WER is diagnostic, not decisive. The defaults say so.

WER (Word Error Rate) is the most commonly used evaluation metric in speech recognition, computed as the proportion of substitutions, deletions, and insertions between the recognized text and the reference text relative to the total number of words. It is intuitive and easy to compare, but as a release gate it has two structural flaws: first, WER is insensitive to semantics, counting recognizing "transfer" as "transf er" the same as recognizing it as "inquiry"; second, WER is a global average and cannot reflect how errors are distributed across key entities (amounts, account numbers, dates)—and in IVR scenarios it is precisely entity errors that cause real business failures. This framework therefore demotes WER to a diagnostic item and instead uses ASR-IFR (the rate of intent/flow failures caused by ASR) and CEER (Critical Entity Error Rate) as gate metrics—the former measures whether speech recognition problems push more test cases down the wrong branch, and the latter measures whether entity corruption increases; both are closer to end-to-end business consequences.

Honest Boundaries

The README states: "The default numeric gates are illustrative examples for the tool demo, not universal acceptance criteria. Production teams should govern thresholds themselves according to risk level and historical baselines." --max-asr-ifr-delta 0.02 and --max-ceer-delta 0.01 are only starting points. A payment IVR and a music IVR should not share the same gate, just as they should not share the same scale of consequences. The framework's job is the mechanism—presence, numeric, finite, fail-closed, explicit opt-out. Your job is the policy: which deltas, which risk level, against which baseline, reviewed how often. A gate whose thresholds belong to no one is a false green in a sharp suit.

For Everyone Building Gates

You don't need this framework to apply this principle. Whatever quality gate you have—test coverage, latency budget, error-rate alerts, model evaluation—ask the "fourth outcome" question: if the metric this gate checks became unavailable tomorrow, would the gate turn red or green? If the answer is green, you don't have a gate; you have a green light wired to an empty line. Fix it the way v1.1.0 did: make the metric's existence part of the check, fail loudly and specifically when it is missing, and make every skip an explicit, visible, reviewable decision.

The red light you investigate is a nuisance. The green light you blindly trust is a failure report waiting for a date.

One-line Summary

In release gates, a "false green" is the costliest outcome, and fail-closed design plus explicit opt-out is how you eliminate it.

Paragraph Summary

The article identifies four possible outcomes for a release gate, of which the "false green" is the only unmonitored yet costliest one: it treats metrics that cannot be evaluated as passing and destroys the entire premise for the gate's existence. CAFA-IVR v1.0 had this flaw, letting pipelines proceed when summary CSVs contained missing, non-numeric, or non-finite metrics; v1.1.0 switched to fail-closed, printing FAIL and exiting with status code 2 on missing, unparseable, NaN/infinite, or empty files—deliberately distinct from status code 1, which signals gate rejection. The article stresses that silent skipping is the root of false greens, and that any skip should be declared via an explicit, visible, version-controlled option like --no-ceer-gate. The WER gate is off by default, embodying the belief that "WER is diagnostic, not decisive," while releases are actually gated on ASR-IFR and CEER deltas. The framework provides the mechanism; teams must govern thresholds and other policies by risk level.