Seal Entropy Before You Refactor: Twenty Identical Runs for a Trustworthy Baseline

Your golden files rot because code isn't the only input. Clocks, random seeds, UUIDs, and hash ordering all leak into output. Before you refactor a messy repository, seal these four entropy sources, then prove the test suite is stable, and only then record a baseline.

This article is Step Zero of refactoring a messy repository. It doesn't discuss the refactor itself, but rather how to earn the right to a "trusted baseline."

Why baselines lie

A characterization test compares bytes. If the bytes change for reasons unrelated to your edit, the test is noise. A noisy test teaches you to re-record the baseline, and re-recording is precisely the channel through which regressions ship.

Most messy repositories have four common entropy sources:

1. Wall-clock reads (datetime.now, time.time, utcnow).
2. Unseeded random number generators (random, NumPy generators, the faker library).
3. uuid4, which reads os.urandom and cannot be seeded.
4. Set and dict iteration order under a random PYTHONHASHSEED.

Find them before designing anything. A single grep is enough to start.

rg -n --glob '!tests/**' \
  'datetime\.now|utcnow|time\.time|random\.|uuid4|secrets\.' \
  src/ | tee entropy-inventory.txt
wc -l entropy-inventory.txt

Treat that count as your blast radius. Ten hits is a fixture problem; two hundred hits is an injection problem, and you should budget for it.

A characterization test (a term popularized by Michael Feathers in Working Effectively with Legacy Code) doesn't assert "correct" behavior up front. Instead, it records the current code's actual output as a reference, and every subsequent run is compared against it. Its value lies in building a regression guardrail for legacy code—as long as the bytes don't change, the behavior hasn't changed. But this also means it compares byte-level output, so any change unrelated to business logic (timestamps, random IDs, container iteration order) gets misjudged as a regression. Another common approach is "approval testing," which writes output to human-reviewable snapshot files—consistent with the canonical JSON snapshot idea in this article.

PYTHONHASHSEED is an environment variable at the CPython interpreter level that controls string hash randomization. Hash randomization has been enabled by default since Python 3.3 to defend against hash collision attacks (HashDoS). The side effect: set and dict iteration order may differ across processes because string hash values change. This variable can only be set via the environment before the interpreter starts—once the process is running, it can't be changed—so it must be set in the runner (CI, pytest launch command), not in a fixture with monkeypatch. Setting it to 0 disables randomization, keeping iteration order consistent across runs.

Step one: Seal all four entropy sources in a single fixture

The test skeleton below is illustrative; I haven't run it against your repository. Treat it as a template, then adjust patch locations to your call sites.

`monkeypatch` is pytest's built-in fixture for temporarily replacing attributes, dict items, or environment variables during a test, restoring them automatically afterward so other tests aren't polluted. Its `setattr` method can replace module-level attributes, but it has limits: if the code under test uses `from datetime import datetime`, the module has already bound the original `datetime` class, and replacing the `datetime.datetime` module attribute won't affect that bound name. This is precisely the root of the limitation mentioned at the end: "module-level patches break libraries that use `from datetime import datetime`."

`autouse=True` makes the fixture apply automatically to every test in its conftest scope, without explicit declaration. Placing it in `tests/characterization/conftest.py` confines the seal to the characterization directory without affecting the whole test suite—echoing the cost-control strategy of "confine to the characterization directory" in Step Four. `uuid.UUID(int=...)` constructs a UUID from an integer, producing a fully deterministic 128-bit identifier with valid formatting, matching the shape of a real `uuid4` to fool code that validates UUID format.
# tests/characterization/conftest.py
import datetime as dt
import random
import time
import uuid

import pytest

FROZEN = dt.datetime(2026, 1, 1, 12, 0, 0, tzinfo=dt.timezone.utc)


@pytest.fixture(autouse=True)
def seal_entropy(monkeypatch):
    # 1. Clock: freeze both naive and aware call sites.
    class FrozenDateTime(dt.datetime):
        @classmethod
        def now(cls, tz=None):
            return FROZEN if tz else FROZEN.replace(tzinfo=None)

        @classmethod
        def utcnow(cls):
            return FROZEN.replace(tzinfo=None)

    monkeypatch.setattr(dt, "datetime", FrozenDateTime)
    monkeypatch.setattr(time, "time", lambda: FROZEN.timestamp())

    # 2. RNG: reseed per test, not per session.
    random.seed(20260101)

    # 3. uuid4: deterministic counter with the same shape as a real UUID.
    counter = iter(range(10_000))
    monkeypatch.setattr(uuid, "uuid4", lambda: uuid.UUID(int=next(counter)))

    yield

Two details matter more than the code itself. First, reseed per test. A session-level seed makes test order part of the baseline. Second, set PYTHONHASHSEED=0 in the runner, not in a fixture—it can't be changed once the interpreter has started.

Step two: Write canonical snapshots, not stdout

Don't hash pytest output; it contains legitimately varying durations and order. Hash the artifacts the tests produce.

def emit(tmp_path, name, payload):
    """Write one canonical JSON file per characterization case."""
    import json
    path = tmp_path / f"{name}.json"
    path.write_text(json.dumps(payload, sort_keys=True, indent=2) + "\n")
    return path

Sorted keys, a single trailing newline, no timestamps in the file. If a snapshot needs a time, read it from the frozen clock so it stays constant.

Step three: Prove stability with twenty runs

One passing run proves nothing. Run the characterization directory twenty times, hashing the concatenated snapshots each time. You must see exactly one unique digest.

for i in $(seq 1 20); do
  rm -rf /tmp/snap && mkdir -p /tmp/snap
  PYTHONHASHSEED=0 pytest -q tests/characterization \
    --snap-dir=/tmp/snap >/dev/null 2>&1
  find /tmp/snap -name '*.json' -print0 \
    | sort -z | xargs -0 cat | sha256sum
done | sort -u | tee digests.txt
wc -l < digests.txt   # must print 1

If it prints more than one, you haven't sealed all the entropy sources. Bisect by running each test file individually. Most failures at this stage are unseeded RNGs inside third-party libraries, not your code.

This loop costs twenty times your suite's runtime. Confine it to the characterization directory, not the whole suite.

sort -z paired with xargs -0 handles NUL-separated filenames to prevent xargs from mis-splitting arguments when paths contain spaces or newlines; find -print0 outputs a NUL-separated list. sha256sum computes a digest over the entire concatenated byte stream, and after sort -u deduplicates, wc -l counts unique digests—the result must be 1.

Why twenty times and not two? It's an empirical threshold: a single run can't distinguish "deterministic" from "coincidentally consistent," and most entropy sources (hash seeds, second-rounded times, reset RNGs) may not show up in a small number of runs. Twenty runs makes common instability surface with high probability while keeping the cost acceptable. To emphasize, this loop verifies stability under the same machine and the same interpreter version; cross-platform or cross-dependency-version differences still need separate handling. If multiple digests print, the bisection procedure is: first run each test file separately to find the one with an unstable digest, then investigate test by test within that file—usually the problem is a global RNG or cache inside a third-party library.

Step four: Now, and only now, record the baseline

After twenty identical digests, copy the snapshots into the repository. Commit them with a one-line diff that does only this. That commit is your reference point for the rest of the refactor.

Rules for the baseline commit:

1. No source changes in the same commit.
2. Record the exact command and seed in the commit message.
3. Note which checks were skipped, and why.

A stable baseline can still be wrong. Determinism doesn't mean correctness; it only means the next diff is attributable.

Step five: Make minimal safe changes

Now start the refactor. One seam per commit. The characterization digest must be byte-identical before and after the change. If it isn't, either the change didn't preserve behavior, or the baseline leaked.

Candidate change	Lines touched	Gate	Verdict
Add a clock or RNG parameter to one function	~10	Same digest	Do first
Move one call site behind an interface	~30	Same digest	Do second
Replace dicts with dataclasses across a module	~150	Same digest	Split into three commits
Rename a public function used by 40 call sites	~90	Same digest + import check	Only if all callers are in-repo
Reorder side-effect calls inside a function	any	Same digest + call-log diff	Needs a second gate

Use a diff budget. If a commit exceeds roughly 40 lines of source change, stop and split it. Large commits hide the line that broke the digest.

Step six: Where free models and free servers fit

Candidate minimal diffs are cheap to generate and expensive to trust. I use MonkeyCode's free model access to draft two or three candidate seams for a single function. The deterministic loop makes the decision, not the model. The free server option is where I run the twenty-run loop, so the repetition cost doesn't land on my laptop.

Disclosure: This article was prepared as part of a promotion for MonkeyCode.

The division of labor is deliberate. Models propose, digest gates dispose. Any candidate that can't pass twenty identical runs—no matter how clean the suggestion reads—isn't a candidate.

Limitations and who should skip this

• Sealing hides real time dependence. Retention windows and expiry logic need a separate, unfrozen integration suite.
• Module-level patching of datetime breaks libraries that use from datetime import datetime. Prefer explicit injection where you can add it.
• Twenty runs catch instability but not wrong expectations. Review one snapshot manually.
• The loop multiplies runtime. On slow suites, cap it at ten and document it in the commit message.
• If behavior is legitimately nondeterministic and untested, skip this method. Seal what you can seal, isolate the rest.

If your test suite can't survive twenty identical runs, the refactor isn't reviewable. Fix the baseline first, then cut a small diff.