slopsieve: A Diff Quality Checker for the Age of AI Code Slop

When Code Review Gets Drowned in AI Slop

Open source maintainers are facing a new problem. cURL shut down its bug bounty program after being harassed by AI-generated junk patches. The Godot project has been flooded with low-quality PRs. GitHub has even started discussing adding a "kill switch" for PRs. These cases all point to the same pain point: the code looks fine until you actually read it.

A tired late-night reviewer needs to quickly judge the quality of a diff at 1 AM. slopsieve is a tool designed for exactly this scenario.

What Is slopsieve

slopsieve is a zero-dependency Node CLI tool that scores diffs or PRs and detects 12 code hygiene signals. Its design goal is not to determine whether code was written by AI, but to evaluate the hygiene of a diff.

Usage is straightforward:

$ npx slopsieve fixtures/slop.patch

SLOP SCORE: 100/100 - ALARM
4 files · +37 -8 lines

FLAGS
  [30] test weakening        tests/cart.test.js:24 it.skip(...)
  [25] debug leftovers       src/cart.js:25 console.log("checking qty", qty)
  [20] empty catch/except    src/utils.js:12 } catch (e) {}
  [20] trivial dependency    package.json:17 adds 'is-odd'
  [15] commented-out code    3 commented-out lines
  [10] narration comments    6 narration comments
  [10] dead exports          'isQuantityOdd' referenced nowhere in this diff
  [ 5] vague util sprawl     src/utils.js created

The output clearly labels the file and line number where each issue occurs.

unified diff is the standard format used by version control systems to describe code changes, and it is the only input slopsieve needs to parse. It consists of a series of "hunks" (code blocks), each headed by @@ -old start line,old line count +new start line,new line count @@, followed by context lines (prefixed with a space), deleted lines (prefixed with -), and added lines (prefixed with +). The benefit of this format is its natural compatibility with Git, GitHub PRs, and git diff output — the tool only needs to process plain text without connecting to any API or repository. For slopsieve, the key to weighted detection lies in the "intent of the change" rather than the final file content: whether a deleted assert means tests were weakened, or whether a newly added console.log is just debug residue, must be judged from the added and removed lines in the diff. Therefore, its detection scope is naturally limited to the diff itself — callers outside the change, untouched test files, or historical context never enter the scoring.

The 12 Weighted Detection Mechanisms

slopsieve parses the unified diff format and runs 12 weighted checks. Several typical categories of issues can be seen from the example output:

Test weakening (weight 30) — Detects operations like it.skip and deleted assertions. This is the highest-weighted signal, because skipping tests often means the code hasn't truly been verified.

Debug leftovers (weight 25) — Catches console.log, debugger statements, and the like. In the example, console.log("checking qty", qty) is accurately flagged.

Empty catch blocks (weight 20) — The catch (e) {} pattern that swallows exceptions.

Trivial dependencies (weight 20) — Introducing npm packages like is-odd that could be replaced with a single line of code.

Commented-out code, narration comments, dead exports, utility sprawl — Cover other hygiene issues commonly seen in code review.

The tool supports a --fail-at parameter to turn it into a CI gate, and --json output for consumption by other tools. Once integrated into CI, it can automatically block low-quality diffs before merging.

The design logic of weighted scoring deserves a separate explanation. Each category of issue is not simply counted as one vote; instead, it is assigned a weight based on how much it consumes the reviewer's attention and how much it damages code quality: test weakening at 30 points, debug leftovers at 25, empty catch and trivial dependencies at 20 each, and so on. The final slop score is the sum of all triggered items' scores, and reaching a threshold triggers ALARM. The advantage of this linear additive model is transparency — maintainers can immediately see which item pushed the score up, rather than facing a black-box judgment. It also provides a tunable parameter for CI gating: by specifying a tolerance ceiling through --fail-at, teams can decide "how many points is unacceptable" according to their own standards. It's worth noting that the weights are subjectively set heuristics, not statistically calibrated probabilities, so they should be understood as a ranking and alerting tool rather than a precise quality metric.

A Story That Exposed the Tool's Own Bug

The author shared a self-deprecating detail: the regex for detecting debug leftovers, dd\s*\( (used to match Laravel's dump function), accidentally matched isQuantityOdd(, because the string "dd(" was hidden inside "Odd(".

This bug caused the tool to completely misjudge the test fixture. The fix was to add a word boundary, and the test fixture was kept to ensure the regression wouldn't recur. This detail illustrates a typical pitfall of rule-based detection tools: the boundary conditions of regex matching are far more complex than one might imagine.

Word boundary is written as \b in regular expressions, used to assert that one side of a position is a word character (letter, digit, or underscore) while the other side is not, thereby avoiding substring false matches. The author originally used dd\s*\( to match Laravel's dump or dd() debug calls, but that pattern did not require a non-word character before dd, so the trailing dd( inside the identifier isQuantityOdd was treated as a Laravel debug function. The fix was to prepend a word boundary to the pattern so it only matches when dd itself is a complete identifier. This case reveals a typical fragility of rule-based static detection: the shorter the pattern, the more prone to false positives; the more specific, the narrower the coverage. It also explains why such tools usually need accompanying test fixtures — only by freezing previously problematic samples into regression tests can similar boundary conditions be prevented from failing again in future changes.

Not an AI Detector

The author clearly establishes an important boundary: slopsieve is not an AI detector. Humans can write 100/100 garbage diffs, and LLMs can write 0/100 clean code.

This positioning is pragmatic. Rather than trying to determine the origin of code (which is technically unreliable), it's better to focus on quantifiable hygiene signals. The README lists the blind spots of the heuristic rules: two-line catch blocks, callers outside the diff, comment patterns in other languages, and so on.

Impact on the Code Review Ecosystem

slopsieve responds to a real and worsening ecosystem problem. Maintainers' time is consumed by low-quality contributions, while AI tools have lowered the barrier to generating plausible-looking code.

Rule-based detection has inherent limitations, but it provides a zero-cost first line of filtering. For maintainers, using a tool to screen out obvious problems before human review can significantly reduce cognitive burden. The ready-to-use approach of npx slopsieve owner/repo#123 adds virtually no adoption cost.

The long-term value of such tools may lie not in the rules themselves, but in driving the community toward a consensus standard for "what constitutes a high-quality diff."

One-line Summary

slopsieve is a zero-dependency Node CLI that evaluates diff hygiene, using 12 weighted checks to help open source maintainers filter out AI slop patches.

Paragraph Summary

Faced with the reality that AI slop code is plaguing the open source ecosystem — cURL canceling its bug bounty, Godot being flooded with low-quality PRs — slopsieve offers a pragmatic response. It is a zero-dependency Node CLI tool that parses unified diffs, runs 12 weighted checks, scores issues like test weakening, debug leftovers, empty catch blocks, trivial dependencies, commented-out code, and dead exports, and outputs specific files and line numbers. The tool supports --fail-at as a CI gate and --json for consumption by other tools. The author emphasizes that it is not an AI detector — both humans and LLMs can produce high-scoring or low-scoring diffs — and that it only evaluates quantifiable hygiene signals. A regex false-match bug exposed during development also shows that the boundary conditions of rule-based detection are far more complex than one might imagine.