The Frozen-Scan Detector: I Built It, But Nobody Consumes Its Alerts

A single detection command printed three numbers: 15, 15, 3. These three numbers each represent a chain of silent failures, and they also reveal an engineering trap more insidious than "no monitoring at all."

A Failed Pipeline That "Successfully" Ran 21 Times

The author maintains a local Python analysis job that iterates over 3,147 individual stock daily-line JSON files, classifies each stock into one of six technical patterns, and then appends a scan result line to signal_history.jsonl. The next day, it scores the previous day's selections and writes them to a verification log.

This job ran on time at 15:35 every night for 21 consecutive business days, exited with exit 0 each time, and the logs were all green. It wasn't until the author ran his own frozen-scan detector that the truth came out: the date of the latest candlestick in the panel was 2026-09-28, 15 business days behind "today." The pipeline wasn't analyzing the present — it was repeatedly analyzing the same long-frozen past.

Even more telling: three records appeared in the verification log, with target_date advancing forward, yet each stock's return figures were completely unchanged — because the price files' mtime hadn't moved since 2026-09-28 12:10. The system was wrapping old answers in new dates.

Why File Counts Can't Save You

There's a particularly easy-to-overlook detail in this case: the panel directory did contain 3,147 files, but the latest candlestick only showed 3,049 tickers, with the remaining ~100 files stuck at earlier dates. Nothing raised an alert about this.

File count is not a freshness measure. Row counts, run duration, exit codes — these "activity metrics" can all be satisfied on a frozen pipeline. From this, the author draws a key judgment: the only reading a frozen pipeline cannot fake is whether the substance of the output has changed.

In Unix systems, a file's mtime (modification time) records the timestamp of when the file's contents were last modified. For data pipelines, mtime is a cheap and direct piece of evidence for "whether the data was updated": if the price files are re-fetched and written every night, the mtime should advance accordingly. It was precisely by observing that the mtime was stuck at 2026-09-28 12:10 that the author confirmed the price files were never refreshed. Note that mtime can also change if a file is rewritten with identical contents, so it's suitable as supporting evidence, not a content-level criterion.

Fingerprint the Answer, Not the Activity

The solution is to hash the fields that define the "answer," deliberately excluding the fields that define "this run." The author defined SNAP_KEYS = ('dist', 'picks', 'rebound', 'breaks'), and used sort_keys=True during serialization to prevent dictionary reordering from faking a change. At the same time, timestamps like scan_date, verify_date, target_date, generated, run_at, and ts are excluded — the advance of dates cannot fake a content change.

Anyone with the ledger file can independently recompute these fingerprints without needing access to the author's code.

Based on these fingerprints, three checks fall naturally into place:

• panel_stale: Compares the latest candlestick date against the expected last trading day, computes the gap in business days, and triggers when it exceeds a threshold.
• trailing_repeat: Checks the fingerprints of consecutive ledger entries and triggers when the identical trailing length exceeds a threshold.
• verify_frozen: In the verification ledger, if target_date advances while the itemized details remain completely unchanged, it triggers immediately.

Two of these checks also incidentally catch two failure modes the author hadn't anticipated before writing the code: the pipeline can keep reading but the numbers stop moving; or it can keep changing dates but the measurement results stay frozen. Both look like "output is growing," but neither actually is.

sort_keys=True is a parameter of Python's json.dumps that forces dictionaries to be serialized in alphabetical key order rather than insertion order. This is crucial for fingerprint stability: if the same data is constructed into dictionaries with different key orders across runs, without sorting it would produce different JSON strings and different hash values, thereby misjudging "content unchanged" as "content changed" — which is exactly what would mask the freezing problem. Conversely, fixed ordering lets anyone on any machine recompute the same fingerprint, making the ledger auditable.

A Week Later: The Detector Is Right, But It's Decoration

Here is the most honest and most valuable part of this article. The author built the detector, published the tool, wrote up the failure-mode analysis, and then — never wired it into the write side.

Since the detector went live, the write job's cron has appended three more byte-for-byte identical records. The detector returns exit code 2 (FROZEN) every day, but no process consumes that exit code. For a tool that nobody calls, its alerts are equivalent to nonexistence.

From this the author distills the article's core insight: a detector whose verdict is not a precondition is just a dashboard. And a green dashboard sitting on top of a frozen pipeline is worse than no dashboard at all — now there are two things telling you everything is fine: the write side's exit 0, and the detector's very existence.

The fix requires just one line on the write side: call the detector, and if the return code is 2, immediately sys.exit('refusing to append a snapshot I cannot vouch for'). The snapshot should never have been written at all. Logging an alert while still writing stale entries is the same bug with more ceremony.

An exit code is the integer convention a process uses to communicate its result to the caller, typically 0 for success and non-zero for some kind of failure. In shells, cron, CI systems, and schedulers all use exit codes to decide whether to alert or abort. But an exit code only matters if it's consumed: if the scheduler doesn't check it, or subsequent steps don't depend on it, then returning 2 has exactly the same effect as returning 0. This is the technical root of the author's statement that "an unconsumed verdict equals nonexistence" — an alert must be a decision branch in the flow, not a side-channel log.

Six Rules for Any Scheduled Analysis Task

The author distilled his lessons into a reusable checklist:

1. Green is not evidence. "It ran and exited 0" only describes the runner. Ask: if the task had failed, which reading would be different? If the answer is "none," you measured nothing.
2. Ask yourself: what would the output look like if the task had secretly frozen? This question is cheap to answer, and all of the author's reliability bugs live here.
3. Fingerprint the answer, exclude the clock. Timestamps, run IDs, row counts, file counts — they all change when nothing has changed, and they all stay the same when everything has changed.
4. "No input" needs its own exit code. The detector returns 3 INSUFFICIENT when there are no entries to judge, because a checker that checked zero records should never print "healthy" — just as a file scanner that opened zero files should never print "clean."
5. Two consecutive identical days is normal; fifteen is not. Make thresholds explicit, tune them against your own hit rate, and remember that a single repetition is not a signal.
6. Gate, don't report. An unconsumed verdict is decoration. Checks should live as preconditions on the write side, and the failure mode should be "no snapshot written," not "snapshot written plus a warning log."

The Tool's Boundaries and the Unfilled Gaps

scan_freshness_guard.py depends only on the standard library, is 409 lines, and has exit codes 0 HEALTHY, 2 FROZEN, 3 INSUFFICIENT, and 1 error, along with an 18-assertion self-test based on synthetic fixtures. The repo's samples/ directory publishes real frozen ledgers — including the 16-record version with 10 trailing repeats and its verification ledger — so the FROZEN verdict can be reproduced on your own machine without the author's data. The code is released under the MIT license.

The author explicitly states one boundary: this tool detects frozen output, not broken scraping. A data source that returns different garbage every day can pass all three checks. Freshness is necessary, not sufficient.

His own known next gap is the exchange holiday calendar: currently panel_stale only accounts for weekends, treating October exchange holidays as ordinary business days — and that's precisely why the 15-day gap was read as "ordinary Monday drift."

For any analysis agent that genuinely runs every day — stock picking, scheduled scans, any task that "appends a snapshot and exits 0" — the value of this detector isn't in those 400 lines of code, but in the question it forces you to answer: what does your green actually measure?

One-line Summary

A pipeline exited 0 for 21 consecutive days while repeatedly analyzing frozen old data, revealing that activity metrics cannot prove output freshness.

Paragraph Summary

The author's local stock-picking analysis job ran on time for 21 consecutive business days with all-green logs and exit code 0, but a detector found its latest candlestick date was 15 business days behind today — the pipeline had been repeatedly analyzing the same frozen data. "Activity metrics" like file counts, row counts, and run duration can all be faked by a frozen pipeline; the only thing that can't be faked is whether the substance of the output changed. The author therefore hashed the fields defining the answer while excluding all timestamps, designing three checks: panel_stale, trailing_repeat, and verify_frozen. Yet after going live, the detector returned FROZEN daily with no one consuming it, because it was never wired into the write side, degrading into a decorative dashboard. The core conclusion: a verdict must be a precondition for writing, and the failure mode should be refusing to write, not writing plus a warning.