trainlint: Static Checks Catch ML Training Bugs That Don't Crash

Data Leakage: The Most Classic Silent Bug

The most dangerous bugs in training code often don't throw errors—the model trains normally, the loss decreases as expected, but the results are just wrong. Recently, an open-source tool called trainlint has attempted to tackle these "silent bugs" with static analysis.

Developer Satvik Jain introduced the project on Reddit: it uses Python's ast module to parse code, matching variable names and API calls to find common anti-patterns in training and evaluation scripts without running the code, requiring no data, models, or GPU.

The author illustrates the severity of the problem with a simple experiment: on a dataset with 5,000 random features and random labels, if you first select the top 20 features on the full dataset and then perform 5-fold cross-validation, accuracy can reach 85% (averaging 87.7% over 20 random seeds). But when feature selection is moved inside a Pipeline, accuracy drops back to 48% (averaging 50.0%).

The theoretical ceiling for any model on random labels is 50%, so the entire 35-percentage-point gap comes from leakage. trainlint can statically identify this "preprocess on the full dataset, then split" pattern—this is its first rule, and also the type of error most likely to produce misleading experimental conclusions.

Data leakage mechanism in cross-validation: When preprocessing steps like feature selection, standardization, or missing-value imputation are performed on the full dataset before splitting into training and validation sets, statistical information from the validation folds "leaks" into the training process through the preprocessing parameters. Taking feature selection as an example: if features most correlated with the target are selected using the full dataset that includes validation-fold labels, the selected features themselves carry information from the validation set—the model has effectively "seen the answers" during validation. The correct approach is to encapsulate preprocessing in a Pipeline so that it only fits on each training fold and transforms on the corresponding validation fold. A simple yardstick for measuring leakage: on purely random labels, the expected accuracy of any leakage-free model is 50%; once it is significantly above this baseline, information leakage is almost certainly present.

Thirteen Rules Covering Training and Evaluation Pitfalls

Besides fit-before-split, trainlint checks thirteen other categories of problems, roughly grouped as follows:

• Mode switching: forgetting to call model.train() during training, leaving eval mode active; using no_grad without switching to eval; fp16 autocast missing GradScaler.
• Loss function issues: manually applying softmax before CrossEntropyLoss; keeping loss as a tensor instead of a scalar; swapping y_true and y_pred argument order; passing hard labels to roc_auc_score.
• Training loop issues: calling zero_grad between backward and step; setting drop_last on the evaluation DataLoader; never shuffling the training DataLoader.
• Engineering hygiene: resumable training scripts not saving optimizer state; missing random seeds.

What these rules have in common: the code runs, and even the loss curve looks normal, but the reliability of the training results has already been compromised.

PyTorch's train/eval modes and torch.no_grad() are two independent mechanisms: model.train() and model.eval() control whether Dropout is active and whether BatchNorm uses the current batch's statistics and updates running averages; whereas torch.no_grad() only disables gradient computation to save memory. If you forget model.eval() during evaluation, Dropout continues to randomly drop neurons and BatchNorm keeps updating statistics, causing metrics to fluctuate and become irreproducible; conversely, leaving eval mode active during training disables regularization. Thus "using no_grad without switching to eval" is a typical silent error—the code runs and loss decreases, but evaluation results are untrustworthy.

Why mixed-precision training needs GradScaler: The smallest positive normal number for fp16 is approximately 6e-5; many gradient values below this magnitude underflow directly to zero, causing parameters to stop updating. GradScaler scales the loss by a dynamic factor before backpropagation so gradients fall within the representable range of fp16, then scales back before optimizer.step(); without it, training may appear normal while some layers "fail to learn."

The Limits of Static Analysis

trainlint is based on name matching and single-file parsing, which means its judgments depend on naming conventions. The author candidly acknowledges several limitations: it only supports PyTorch and scikit-learn; if code names split variables in unusual ways, it may miss or falsely report issues; there is no recall data.

Regarding accuracy, the author conducted two types of validation. Internal testing paired 20 buggy scripts with fixed versions, but since the rules and test cases were written together, this only guards against regressions. More convincing is external validation: on the pytorch/examples repository at a fixed commit, the tool reported 22 issues, of which 20 were real problems and 2 were correct but harmless. Earlier runs had 4 false positives, which have since been fixed.

The author explicitly states that this is only a tutorial code repository, manually judged by himself, with limited samples. This honesty itself deserves recognition—the value of a static checking tool lies not in zero false positives, but in capturing at low cost the patterns that easily slip through manual review.

The difference between static analysis and data-flow analysis: Static analysis means examining source code structure without executing the program. Python's ast module parses source code into an abstract syntax tree, and tools traverse the nodes to identify patterns like "one function is called before another." trainlint uses pattern matching based on naming and call order, which is the lightest-weight tier; data-flow analysis, by contrast, tracks the actual flow of variables through assignments, passes, and aliases, and can determine whether a DataFrame truly originates from the full dataset—but its implementation complexity and difficulty in controlling false positives are much higher. This also explains why the tool relies on naming conventions and only parses single files: it doesn't truly "understand" how data flows; it only recognizes suspicious code shapes.

Use Cases and Limitations

For individual researchers and small-to-medium teams, trainlint can serve as a lightweight pre-commit check, intercepting obvious anti-patterns before code reaches the training cluster. Its zero-dependency, no-runtime requirement makes it easy to integrate into CI pipelines.

But its positioning should be clear: this is not formal verification, nor is it data-flow analysis. It cannot understand complex variable aliases or cross-file data processing logic, nor can it judge whether the experimental design itself is sound. Treat it as the first layer of a safety net, not the final guarantee.

The code and examples for each rule are available in the GitHub repository Satvik-jain/trainlint, and the author welcomes community feedback on false positives and misses.

pre-commit and CI integration: pre-commit is a framework that automatically runs check hooks before git commit; developers declare tools in a configuration file, and commits are blocked if checks fail. Integrating zero-dependency, GPU-free checks like trainlint into pre-commit or CI is extremely low-cost and can intercept obvious problems before code reaches the training cluster. Note its positioning: "false positives" (reporting issues that are actually harmless) erode trust, while "misses" (letting real bugs through) leave hidden risks—so such tools typically serve as the first low-cost filter before manual review and runtime validation, not as a final guarantee of correctness.