research-ledger: A Research Discipline Tool That Keeps AI From Fooling Itself

Using multiple AI models for research, the biggest risk is often not a lack of inspiration, but fooling yourself. An IT support practitioner turned months of hard-won lessons into an open-source skill called research-ledger, attempting to install a "self-deception detector" on AI-assisted research.

Four Self-Deception Traps

The author does IT support by day and explores ideas with Claude, GPT, Gemini, Grok, and other models at night. After several months, the output was a large number of ideas and very few results. He summarized four typical forms of self-deception:

Consensus between models is fake evidence. Three models all saying "this connection holds" feels like evidence, but it isn't. They share a massive amount of training data, and consensus only indicates that you should go verify — not that the conclusion itself is correct.

Metaphor disguised as mechanism. "X in physics is like Y in learning" sounds profound, but usually no one can write out the step that actually transfers.

Thresholds move after results. 0.62 becomes "promising" because you want it to be.

"Not found in the literature" becomes "this is a new discovery." That sentence describes your search, not the world.

Why multi-model consensus is not independent evidence: When Claude, GPT, Gemini, and other models give similar judgments, intuition treats it as multi-source verification. But these models all learn from large-scale web and book corpora, and their training distributions overlap heavily, so they are more likely to share the same set of errors, biases, and popular claims. Statistically, this is "correlated evidence" rather than "independent evidence," and its joint confidence is far lower than people's subjective impression. What actually raises confidence is independent data, reproducible experiments, or formal proofs. Treating multi-model agreement as a trigger to "go verify" rather than a license that "the conclusion holds" is the first gate against self-deception.

What the Skill Does

research-ledger is an MIT-licensed SKILL.md folder that turns any assistant capable of reading skills into a disciplined research partner. The process starts with a "sieve": most ideas are eliminated within five minutes by a short checklist, with no paperwork required.

Ideas that survive enter the translation stage: every metaphor must become a measurable test, or a precise statement with a proof objective. The skill asks only one question — mechanism or topic? If you can't write out the equation or steps of transfer, it's a topic, and it goes into the ledger rather than into an experiment.

Prior retrieval must come with scope. "Not found" must be recorded together with the date, search terms, and most recent sources, and must never be recorded as "novelty."

Pre-registration requires that predictions, thresholds, control groups, and statistical power be frozen before running. In a Git ledger, pushing a separate PREREG commit before the run is the proof.

A pilot is not a confirmatory test. A pilot can tune parameters, but it never gives a PASS.

Blind test rounds are triggered at milestone points: the same seed is sent to two or three models from different labs, in a clean conversation, carrying no project context.

In the end there are five verdicts: DEAD (wrong, tautological, or killed by a control), LANDED (you rediscovered a known bridge — this is calibration, not failure), BRIDGE CANDIDATE (known parts, connection not found, with new predictions), BRIDGE (only holds after passing confirmatory tests, blind test rounds, and independent checks), EXPLANATION (a known mechanism explains the pattern).

The ledger is append-only, and every "doesn't work" is preserved, along with what it showed and under what conditions its lesson holds.

The role of pre-registration and Git commits in research: Pre-registration means writing down and making public the hypothesis, predictions, thresholds, control groups, statistical power, and stopping rules before looking at the results. Its value lies in separating "post hoc explanation" from "ex ante prediction," preventing threshold drift, post hoc selection, and HARKing (hypothesizing after the results are known). In a Git ledger, pushing a separate PREREG commit before running is equivalent to stamping the timeline with a trusted timestamp: anyone can check whether the prediction preceded the data. This is homologous to clinical trials and the registered reports system promoted after the psychology replication crisis, and is essentially using process to constrain confirmation bias in human nature.

Blind test rounds and cross-lab review: A blind test round means that at a milestone, the same problem seed is sent to models from different labs and different vendors, in a clean conversation, carrying no project context. The purpose is to strip away the author's own framing, terminology, and expectations, and let the model answer from an independent perspective. Cross-lab review introduces models with different training sources to check each other, reducing the systemic blind spots of a single model family. But a blind test round can only remove the author's context; it cannot eliminate the training data shared by models. Therefore it is suitable for discovering "author bias," not for proving "world-class novelty."

Where the Rules Come From

Every rule is a scar. The author's own log contains about forty numbered experiments, involving toy quantum models, combinatorics, small neural networks, and numerology of physical constants.

"Zero error" can be meaningless. A model reproduced a set of measured ratios with an error of 0.0000. Then they ran a forgery control: random "fake" target ratios. About 3,700 formula "keys" could fit the real numbers, and just as many could fit the fake numbers. The fit was empty. This became the "lock leak" check.

Translators insert what they already know. When marginal and amateur texts were fed to a model to find ideas mainstream sources had missed, most "insights" were really just the model translating the text into known concepts. Of the 8 that survived, 6 were known science rediscovered by amateurs. Hence the rule: every remaining item is checked against the literature before being combined with anything else.

A control that cannot differ from the treatment proves nothing. In one continual learning test, the "noise control" was algebraically identical to the treatment group, and the test had no power. The rule now is: before freezing, write clearly why the control and treatment cannot coincide by construction.

Equal strength masks everything. Several effects held only when all component weights were equal, and disappeared under heavy-tailed weights. The rule now is: always retest with heavy tails.

The seed style for blind test rounds comes from studying the published problem prompts in openai/math: they say "prove or disprove," never "show"; they explicitly fix conventions, list what is "not" in the setup, give no method hints, and use no analogies.

The "lock leak" check and forgery controls: In numerology or formula fitting, a model may find a large number of formula "keys" that fit the real data. To judge whether these fits carry information, the author constructs random "fake" target ratios as forgery controls. As a result, about 3,700 formulas fit both the real numbers and the fake numbers, showing that the fit is empty. This logic is isomorphic to the null distribution, permutation tests, and randomization controls in statistics: if a method produces results as good on random data, it has not captured real structure. The so-called "lock leak" means there are too many keys and the lock is too loose — anything can open it — so it does not constitute evidence.

What It Caught

Most ideas die, and that is the method working. The ledger is mostly DEAD and LANDED entries. A pile of LANDED verdicts has its own value: it measures what the translator already knows, not how much progress has been made toward a new discovery.

One clean kill. One idea combined a local learning rule (no backpropagation) with "selectivity": only units matching the input update. They pre-registered the kill criteria and a strong control (a Drosophila olfactory model with fixed random features). Result: the selectivity rule silenced and starved the units, learning worse than updating random units; learned features greatly outperformed fixed random features, even when the random model was 16 times wider; and ordinary feedback alignment without gating performed the same as the gated version. The idea was not refuted in general, but in that setting it contributed nothing, so it was recorded along with the conditions of applicability.

Cross-lab review catches different things, and sometimes catches the reviewer itself. The author applied the method to a side project: extracting a machine-readable map of proof techniques from 719 manuscripts in openai/math. Forty extracted "signatures" were blind-reviewed by Claude and Grok using a frozen rubric.

Grok's first attempt: 40 out of 40 "OK," with every rationale field empty, what it itself called a "spot check." It missed an objective error — the source text said 1, but it gave multiplicity 0. This review was marked invalid, not "disagreement."

Grok's second attempt required every "OK" to cite the source text, in four batches of ten: only then was it a real review.

The two reviewers agreed on 35 of 40 items. Each caught things the other missed: Claude was stricter about missing premises, and Grok caught an assertion taken from lines outside the cited range. In the end both passed 32 of the 40 signatures (80%, 95% interval 65–90%). In the earlier set of 10, 6 passed, with an interval of 31–83%, telling the researchers almost nothing.

The lesson went straight into the method: review requires evidence that the reviewer actually read the source, and it requires planting error items that it must catch.

Tests were registered before contamination. Two "needs" from open learning theory problems were frozen for the map before anyone searched the map with them. The test pitted the map against ordinary grep on the same corpus, with blind evaluators. The author's own bet was written into the pre-registration: most likely "undetermined."

Honest Limitations

This method reduces self-deception, but it does not produce discoveries. A blind test round removes the author's own context, not the training data shared by models. The checklist items come from one specific history, and each item is annotated with its conditions of applicability; the skill asks you to evaluate whether it is useful for your problem, rather than applying it blindly.

How to Try It

git clone https://github.com/Freeky7819/research-ledger
cp -r research-ledger/research-ledger ~/.claude/skills/

It also works in Claude.ai (upload the zipped folder) and Codex (the folder contains agents/openai.yaml). Then just talk to it: "Here's an idea...", "Evaluate this paper," "This test gave 0.62, now what?", "Write a blind test round seed for this."

If a rule doesn't apply to you, open an issue with a case. That is exactly what the ledger is for.

Author disclosure: the skill, the experiments, and this article were developed together with AI models (mainly Claude, but also GPT, Gemini, and Grok). The questions and ideas are his; most of the code and quite a bit of the wording were written by them.