trainlint
A static-analysis-based inspection tool for ML training code that uses Python's ast module to parse code and identify common anti-patterns in training and evaluation scripts, such as data leakage and mode-switching errors, without running the code or requiring a GPU.
Timeline (last 90 days)
trainlint is an open-source tool that uses Python's ast module to statically parse code, detecting common anti-patterns in ML training and evaluation scripts without running the code or needing data, models, or GPUs.
On a dataset with 5,000 random features and random labels, if the top 20 features are first selected on the full data and then 5-fold cross-validation is performed, accuracy can reach 85% (87.7% averaged over 20 random seeds); after moving feature selection inside the Pipeline, accuracy falls back to 48% (50.0% average).
The first rule trainlint detects is statically identifying the pattern of "preprocessing on the full dataset first, then splitting the dataset" (fit-before-split).
In addition to fit-before-split, trainlint checks thirteen other categories of issues, covering mode-switching, loss-function, training-loop, and engineering-hygiene categories.
trainlint is based on name matching and single-file parsing, supports only PyTorch and scikit-learn, and has no recall data.
On the pytorch/examples repository at a fixed commit, trainlint reported 22 issues, of which 20 are real issues and 2 are correct but harmless; an earlier run had 4 false positives, which have now been fixed.
trainlint's internal validation uses 20 buggy scripts paired with fixed versions, but the rules and test cases were written together, so it can only prevent regressions.
Static analysis means checking source code structure without executing the program, whereas data-flow analysis traces the actual flow of variables through assignment, passing, and aliasing; trainlint adopts pattern matching based on naming and call order, which is the lightest tier.
The trainlint code and examples for each rule are located in the GitHub repository Satvik-jain/trainlint.
trainlint is positioned as a lightweight pre-commit check or CI integration tool, serving as the first low-cost filter before human review and runtime validation, rather than a final guarantee of correctness.
All Facts (10)
trainlint is an open-source tool that uses Python's ast module to statically parse code, detecting common anti-patterns in ML training and evaluation scripts without running the code or needing data, models, or GPUs.
50%UnverifiedOn a dataset with 5,000 random features and random labels, if the top 20 features are first selected on the full data and then 5-fold cross-validation is performed, accuracy can reach 85% (87.7% averaged over 20 random seeds); after moving feature selection inside the Pipeline, accuracy falls back to 48% (50.0% average).
50%UnverifiedThe first rule trainlint detects is statically identifying the pattern of "preprocessing on the full dataset first, then splitting the dataset" (fit-before-split).
50%UnverifiedIn addition to fit-before-split, trainlint checks thirteen other categories of issues, covering mode-switching, loss-function, training-loop, and engineering-hygiene categories.
50%Unverifiedtrainlint is based on name matching and single-file parsing, supports only PyTorch and scikit-learn, and has no recall data.
50%UnverifiedOn the pytorch/examples repository at a fixed commit, trainlint reported 22 issues, of which 20 are real issues and 2 are correct but harmless; an earlier run had 4 false positives, which have now been fixed.
50%Unverifiedtrainlint's internal validation uses 20 buggy scripts paired with fixed versions, but the rules and test cases were written together, so it can only prevent regressions.
50%UnverifiedStatic analysis means checking source code structure without executing the program, whereas data-flow analysis traces the actual flow of variables through assignment, passing, and aliasing; trainlint adopts pattern matching based on naming and call order, which is the lightest tier.
50%UnverifiedThe trainlint code and examples for each rule are located in the GitHub repository Satvik-jain/trainlint.
50%Unverifiedtrainlint is positioned as a lightweight pre-commit check or CI integration tool, serving as the first low-cost filter before human review and runtime validation, rather than a final guarantee of correctness.
50%