Three Models Vote on Fact Extraction: Why Multi-Vendor Redundancy Beats Single-Model Optimization

Running the same task and letting three different AI vendors vote on the answer sounds like overkill. But this project is a campaign-log tool for a tabletop RPG that's been running for years, which changes everything.

Fact extraction is a write, not a read

Transcription is a read. Get it wrong and you've produced one bad sentence the reader skips past. Fact extraction is a write: a fact goes into a normalized store, and every later question-answer is drawn from that store.

This means a wrong fact isn't consumed once. It stays there, persistently wrong, contaminating everything downstream — and a confidently-logged wrong fact looks no different from a correct one. Six months later a player asks "who did we buy the horse from," Scout gives an answer, it's wrong, and there's no way to tell — because the wrong answer is just as confident as every right one.

That asymmetry is the core reason redundancy exists here: one bad write is permanent, one duplicate read costs a few cents, and redundancy is the cheap side of that trade.

Why three models, not one better prompt

The most obvious objection is: this is a prompt problem. Write a better prompt, pick the best model, spend the money once and spend it well. This project's experience doesn't support that judgment.

It isn't that the three models are more or less wrong than each other — they're wrong in different directions. Gemini in particular tends to reach for the source material rather than the transcript in front of it — a tendency already encountered in NotebookLM, where it also had to be steered around with prompts.

This is exactly the point of a model panel. Running the same model three times at temperature zero doesn't give you a panel — it gives you the same answer three times plus a participation trophy. Redundancy only buys you something when the members fail on different things, and different vendors trained on different data are the closest thing to independently-failing samples you can get without building your own model.

What is temperature zero?

When calling a large model, temperature is a sampling parameter that controls output randomness. When set to 0, the model almost always picks the highest-probability next token, making the output as deterministic and stable as possible. Many people, when evaluating model consistency, think: same prompt, same model, temperature zero, run three times — don't I get three independent "conservative" answers? You don't. Temperature zero just makes the same model walk the same highest-probability path every time; three runs overlapping almost entirely is basically inevitable, because they share the same training data, the same weights, the same biases. This kind of "multiple votes" looks like three samples but is really one sample copied three times. What actually lets majority voting produce a gain is having voters fail on uncorrelated things — which is why the article stresses that you must use different vendors' models.

What is a model panel and independent failure?

"Panel" here borrows the statistical concept of a panel: a set of parallel judges from different sources. For majority voting to mathematically improve reliability, each member's errors must be approximately independent — that is, one member erring must not push another member toward the same error. Models from different vendors are shaped by different data, different training pipelines, and different alignment strategies, so their failure modes empirically tend to diverge more — the closest thing to "independent failure samples" you can get without building your own model.

Voting is the easy part

Doing majority voting over a set of facts presupposes you can tell whether two facts are the same fact. That's much harder than it sounds. The author manually arbitrated several cases, one of which is typical: two models extracted the same thirty-second stretch of gameplay, and each produced a statement.

Both statements contain the same three names: kettlesteam, candlefoot, and the stolen voice. Same three things, same relations, undoubtedly the same moment. Hard matching is uncontroversial — but each model also bundled other facts into the same statement, and those don't correspond to each other. One folded in who was present; the other folded in the cost.

So the two facts are highly consistent but not identical — really three independent facts sharing one sentence. In the second case, one model explicitly named something and the other only implied it. Jingle Jangle was inferred, not named, and if the inference is correct it's still a true fact — and nothing in the transcript gives the vote counter any basis to judge whether it's correct.

So do they count as consistent? There's no clean answer, because the question itself is wrong. "Are these two consistent" is really asking how big one fact is, and until you've solved that, majority voting has no well-defined thing to count. Three models can produce three statements describing the same event and yet cast three different votes purely because they segment the event differently. The voting mechanism is about forty lines of code; the fact-granularity problem is the real project.

The fact granularity problem

Majority voting requires a clear list of candidates up front, but the reality is that the boundary of each "fact" is itself drawn by the model. The same stretch of gameplay: model A might split it into one compound statement containing participants, cost, and intent, while model B might split it into three separate small facts. Three votes look like three votes, but behind them are three different segmentation schemes, and the voting objects aren't even on the same layer. In the information-extraction literature this phenomenon is called "granularity mismatch": when evaluating extraction quality, you must first align to a unified minimal semantic unit, otherwise consistency and accuracy rates are meaningless. The author's experience shows precisely that in a multi-model fact-extraction product, the real engineering budget shouldn't go into the voting logic — it should go into defining what an atomic fact is and using rules or models to normalize extraction results to the same granularity. Voting takes forty lines of code; granularity definition is the core contract on which the product stands or falls.

The bill is not a footnote

One inference normally; three per session here, and always. For a product whose survival depends on unit cost, that's a design decision with an invoice attached. Early testing ran on a local Claude backend, deliberately spending already-paid subscription quota rather than API credits — cheap, and deservedly so. Recommended for anyone prototyping: an already-paid subscription is the best price there is.

Testing a panel to find out what it buys you is fine. Shipping it before you know the per-session cost is not. These are two different decisions, and it's easy to confuse them when a spike is running smoothly.

The roster is not the design

Anthropic, Gemini, and OpenAI are today's panel, and it's unlikely that roster survives the next year and a half: vendors change, prices change, models get deprecated with two weeks' notice, and something better suited to this specific task than any of the three will show up. None of that touches the design itself.

What's actually being tested isn't "are these the right three" but whether a panel with a coordination step produces more trustworthy writes than a single call — and whether the disagreements are informative enough to be worth paying for. If that holds, the roster is an implementation detail: swap one member, swap two, run five when stakes are high and one when they're low. The structure is the part that has to be right, and the structure is what this experiment is really interrogating. It's the same instinct as "don't build on a single vendor's shape" — design for the character, not for the incumbents.

It talked to me through my own modules

All of this was made much easier because the panel reached the three vendors through its own system's chat modules rather than raw HTTP calls: chat-anthropic, chat-gemini, and chat-openai hang under go/chat, the shape the extractor was built on.

Three vendors, one interface. Swapping a panel member is a config and dependency change, not a rewrite — which is why the roster is disposable. It also immediately surfaced a rate-limiting gap in one of those modules, which was filed upstream (the author filed it to himself) rather than worked around locally. That's the dogfooding trade: the tools you build will, at the most inconvenient moment, tell you about themselves.

Eating your own dog food

"Dogfooding" means a team uses the product it's building in real work. In this project it's not symbolic: the extractor calls the three vendors through its own three modules under go/chat rather than writing raw HTTP requests. The direct benefit is that swapping a panel member only requires changing config and dependencies — but the more important benefit goes the other way: the product's own defects show up immediately in the author's daily workflow, like that rate-limiting gap, which only stings when you're a user yourself. That's the dogfooding trade in a nutshell: put your own tool to use at the most inconvenient moment, and only then will it honestly tell you where it still falls short.

Project status

Scout is still private and far from finished, and the panel is an investigation, not a delivered feature. What the investigation produced is less a voting mechanism than a discovery: nearly two weeks of asking the wrong question. The assumption was that the problem was picking the right model — when in fact the problem was figuring out what a fact is.

One-line Summary

Lessons from multi-model voting on game-fact extraction: the real hard part is fact granularity, not voting.

Paragraph Summary

This article documents an experiment in Scout, a campaign-log tool for a tabletop RPG, where the author tried having three different AI vendors vote on fact-extraction results. The author argues that unlike transcription — a "read" — fact extraction is a "write": a wrong fact persistently contaminates every downstream query, making redundancy worthwhile. The value of a three-model panel isn't that any one is less wrong — it's that they're wrong in different directions; running the same model multiple times doesn't constitute a valid panel. In practice the real difficulty isn't voting but judging whether two facts are the same one: models segment the same event at different granularities, so three votes are effectively cast on different objects. Tripling inference cost is a design decision for a unit-cost-sensitive product, while which three vendors — that's just a swappable implementation detail.