The Third Exit Code for Training Linters: Inconclusive ≠ Failure

An Overlooked CI Truth: A Green Light Isn't Always Trustworthy

In continuous integration (CI) for machine learning training pipelines, the vast majority of failures ultimately boil down to one problem: the pipeline cannot distinguish between "this training run broke" and "I can't read this log." Both end in a non-zero exit code, both wake engineers from their sleep, and one of them is simply a lie.

Continuous integration originated as a core practice in software engineering — developers frequently merge code changes into the main branch, with each merge verified through automated builds and tests. In traditional software, test pass/fail results are typically deterministic: the same input produces the same output. But in machine learning scenarios, CI faces unique challenges: the training process itself is stochastic (random initialization, data shuffling, etc.), evaluation metrics are continuous values rather than boolean judgments, and training log formats vary across frameworks. This means the traditional binary CI model (pass/fail) is inherently unsuited for ML workflows, requiring more granular signals to distinguish genuine problems from environmental noise.

A developer shared his solution on Reddit — a training linter called trainproof. Its core design isn't about the checking logic itself, but about exit codes. The author placed exit codes at the center of the design rather than treating them as an afterthought — a perspective worth reconsidering for every team building MLOps pipelines.

Three Exit Codes: Making "Inconclusive" a First-Class Citizen

The core design of trainproof revolves around three exit codes, each corresponding to a clear signal:

• exit 1 — A rule was triggered; the training is genuinely broken.
• exit 0 — Checks completed with no issues triggered; or only warnings were raised, left for you to triage.
• exit 2 — Inconclusive. Missing fields, unreadable logs, no eval set.

In Unix/Linux systems, processes communicate execution results to callers through exit codes. The convention is: 0 means success, non-zero means failure. But in practice, the POSIX standard allows exit codes from 0-255, and many classic tools already use different non-zero values to convey different types of information — for example, grep uses exit code 1 for "no match found" and 2 for "an error occurred"; diff uses 1 for "files differ" and 2 for "error." However, in CI systems (such as GitHub Actions, GitLab CI, Jenkins), this granularity is typically ignored — pipelines only care about "zero or non-zero," compressing rich semantic information into a crude binary. trainproof reclaims this mechanism by giving exit 2 explicit semantics — effectively restoring the expressive power that exit codes were always meant to have.

The author emphasizes that exit 2 is the truly important one. He makes a pointed argument: a gate that reports "pass" after actually skipping all checks is worse than having no gate at all — because at that point, a green build result is no longer evidence of anything.

This is exactly the blind spot in most CI pipelines today. They tend to crudely fold "check couldn't run" into either "pass" or "fail," and the author argues that both treatments are wrong. A check that didn't run should never look like a check that passed.

Why Deterministic Rules Instead of Model-Based Judgment

Another key decision in trainproof: no models in the process. Every determination is a deterministic rule — it either triggers or doesn't, and prints the specific values it relied on when triggered. Same input, same output, every time.

The author explicitly refuses to place a probabilistic "judge" in a CI gate. His reasoning is practical: an alert you cannot reproduce will eventually be ignored by the team. This is a common pattern in practice — when alert credibility is in doubt, people instinctively develop blindness to red lights, rendering gates effectively useless. This phenomenon is known as "alert fatigue" in the observability domain and is one of the primary causes of on-call team morale collapse. The value of deterministic rules is that every trigger can be precisely reproduced and debugged, and engineer trust in them doesn't erode over time.

A Self-Proving Real-World Case

The most compelling evidence is the story of the tool "catching itself."

trainproof has a check: when every gradient norm in the logs is exactly 0.0, it reports "backward graph severed." Gradient norm is a critical observable metric in deep learning training — it measures the magnitude of the gradient vector computed during backpropagation (typically the L2 norm). During normal training, gradient norms should fluctuate within some reasonable range; if they're persistently zero, it usually means the computation graph was accidentally severed and the model isn't learning at all.

However, a certain framework happens to write this field as 0.0 when gradient clipping is disabled. Gradient clipping is a technique to prevent gradient explosion — when the gradient norm exceeds a set threshold, gradients are proportionally scaled down. Some frameworks (like HuggingFace Transformers) log the post-clipping gradient norm, and when clipping is disabled, the field may be filled with the default value 0.0 rather than being omitted. This kind of framework-level quirk is precisely the root cause of false positives.

So a healthy fine-tuning run of 125,000 steps with good convergence was flagged as FAIL by the author's own tool.

The author's fix is highly instructive — rather than simply adjusting a threshold, he introduced a logical rule: a training run cannot simultaneously be learning and receiving zero gradients. If loss is improving, then those zeros are a reporting artifact, and the check should stand down.

More importantly, the tool explicitly logs "that it stood down, and why" as a visible skip. This leads to the principle the author defends most fiercely:

A check that didn't run must never look like a check that passed.

Therefore, PASS results include structured data listing: which checks ran, which were skipped, and the reason for each skip.

Check Coverage Across the Full Training Lifecycle

trainproof's positioning in the pipeline covers four training phases:

Before GPU

Dataset and tokenizer linting, whether the entry point can import, checkpoint completeness, whether memory and disk meet declared requirements. These checks intercept problems before expensive GPU resources are occupied. In today's environment of high GPU compute costs (a single large model training run can cost thousands to hundreds of thousands of dollars in compute), performing cheap pre-checks before launching training can prevent massive waste — a missing tokenizer file or incomplete checkpoint, if only discovered 30 minutes into training, wastes not just time but scarce GPU hours.

During Training

A single-line HuggingFace callback that warns or aborts on diverging training. HuggingFace Transformers' Trainer class provides a callback mechanism that allows custom logic to be inserted at specific points in the training loop (such as after each step or each epoch). trainproof leverages this mechanism for online monitoring: if loss diverges during training (e.g., suddenly jumps to an extreme value or becomes NaN), training can be terminated immediately rather than waiting until completion.

After Training

From logs you'd write anyway, detect divergence, flatlined metrics, NaN, gradient spikes, overfitting, and other anomalies. "Flatlined" here means loss or evaluation metrics remain unchanged for extended periods, typically indicating too small a learning rate, vanishing gradients, or abnormal optimizer state. Gradient spikes may hint at anomalous samples in the data or instability from an excessive learning rate. These patterns may not be obvious at individual data points but have recognizable characteristics in time series.

Against Baseline

Relative-floor rules — the author notes this is the only method that can catch "happily training on shuffled labels." A model training happily on shuffled labels won't reveal the problem through its own metrics alone; it can only be exposed through baseline comparison.

This approach addresses a subtle but dangerous problem: a model may exhibit seemingly normal training curves on completely wrong data. The classic case is randomly shuffled labels — the model still attempts to fit, loss may decrease (because the model is memorizing noise), gradient norms are normal, no NaN, all surface metrics appear "healthy." The only way to expose this is comparison against a known baseline: if the model's final performance doesn't significantly exceed a random baseline (e.g., exceeding 1/number-of-classes accuracy in classification), something is wrong. This essentially embeds a lightweight data integrity verification into CI.

Engineering Details: Zero Dependencies and Verifiable Contracts

trainproof's engineering implementation reflects the author's obsession with reliability:

• Broad input support: Reads HF's trainer_state.json, Coqui, TensorBoard event files, JSONL, and CSV. HuggingFace Transformers' Trainer class automatically generates trainer_state.json during training, recording complete training state including loss, learning rate, gradient norm, and epoch progress at each logging step. TensorBoard event files are Google's training log format, storing scalars and histogram data in protocol buffers. Supporting these formats means trainproof can adapt to the vast majority of ML training workflows without requiring teams to change existing logging habits.
• Zero dependencies: No torch, no tensorboard, no network connection needed. This design choice is extremely important — fewer dependencies in CI environments mean more reliable builds. If a checking tool itself depends on a multi-GB library like PyTorch, it introduces risks of installation failures and version conflicts, potentially becoming a source of pipeline instability itself.
• Pipeline-friendly: Provides --json output.
• Verifiability: 84 rule IDs, 230 tests, a written contract in CONTRACTS.md specifying what each exit code means and when output might change, plus 38 golden snapshots — ensuring that if a rule quietly stops triggering, the build fails immediately. Golden snapshot testing (also called snapshot testing) is a testing strategy that saves a program's output on known inputs as a "golden standard" file, comparing actual output against this standard on each subsequent run. If output changes, the test fails, and developers must explicitly review and approve the change. This is a check on "the checking system itself" — a meta-level quality assurance ensuring the gatekeeper doesn't quietly go derelict.

The project is MIT-licensed and installable via pip install trainproof. This combination of "zero dependencies + determinism + written contracts" makes it highly suitable for embedding in serious production-grade pipelines.

A Question Worth Every Team Answering

At the end of his post, the author poses the question he truly wants answered: When a check cannot run, what does your pipeline do today?

His observation is that most configurations collapse this situation into either "pass" or "fail," and he believes both are wrong. He's curious whether anyone has already wired a "third state" into their system.

The value of this question transcends trainproof as a tool. It touches on a deep principle in CI/CD and observability design: "no signal" is itself a signal and should not be silently categorized into either end. In the observability domain, this principle is called "distinguishing known unknowns from unknown unknowns" — the former is knowing what you don't know, the latter is not even knowing that you're missing information. Exit 2 essentially transforms "unknown unknowns" into "known unknowns," letting teams at least become aware of their blind spots.

In AI training — a high-cost, long-cycle scenario where results are difficult to judge intuitively — distinguishing "broken" from "unreadable" is especially critical. A false FAIL wastes engineers' trust, while a skip disguised as PASS lets disasters slip through silently. Considering that a single large model training run can last days to weeks, a data problem discovered only on day three means all compute resources from the first three days are completely wasted; and a silent skip disguised as PASS might allow a flawed model to deploy to production, causing business losses far exceeding training costs.

trainproof's answer is simple yet cuts to the heart of the matter: elevate "inconclusive" to a first-class citizen, and leave a traceable record for every silence.

Key Takeaways