LLM Reranker Silently Failed All Along: The AI Testing Blind Spot Triggered by 120 Tokens

A seemingly complete LLM reranking feature had never actually run, even though all tests passed. The author of schemagate documents a classic "silent failure" incident—the problem wasn't the model, but that the tests only verified that failure was safe, never that success actually occurred.

What is schemagate

schemagate addresses a core pain point in the text-to-SQL scenario: when a database has hundreds of tables, which few should the model be shown?

Its approach is to first filter with purely mathematical methods—BM25, embedding, table-name matching—then fuse and rank them via RRF (Reciprocal Rank Fusion). If the user provides an API key, the system calls an LLM to rerank the top 20 candidate tables; this is the reranker step.

This architecture starts from a sound premise: mathematical methods are low-cost and highly deterministic, while LLMs are better at fuzzy semantics. Coarse filtering with math first, then fine reranking with an LLM, is a common hybrid strategy in current RAG and retrieval systems.

RRF (Reciprocal Rank Fusion) is a classic method for combining multiple ranking results without training. It takes the reciprocal of each candidate document's rank in each recall path (commonly using the formula 1/(k + rank), with k usually set to 60), then sums the scores of the same document across multiple result sets to get the final score. Its advantage is that differences in score scales returned by different retrievers (for example, BM25's score range versus cosine similarity are completely different) are sidestepped entirely—it relies only on rank, making it robust and requiring no normalization. In schemagate, BM25, embedding, and table-name matching each produce a candidate table list, and RRF is responsible for fusing them into a unified coarse ranking order before handing it to the LLM for fine reranking. Understanding RRF helps grasp the entire pipeline: it determines the boundary of the candidate set that the LLM reranker can see—if a correct table falls out of the top 20 at the RRF stage, no amount of subsequent LLM reranking can save it.

The incident: three calls, zero answers

The author tested the same three questions with --rerank enabled and disabled, using claude-haiku-5-5. The three results were completely identical.

Identical ranking results are not inherently a problem—if the mathematical method already ranked correctly, reranking naturally wouldn't change the results. So the author added logging, wrapping the provider layer to record each call.

The logs showed: the model was indeed called. But it never gave any answer.

Root cause: all 120 tokens were eaten by "thinking"

The problem was in one line of code:

reply = provider.complete(SYSTEM, prompt, max_tokens=120)

For a result that only needs to return a ranking like 3, 1, 7, 2, 5, 120 tokens is more than enough. It really was sufficient when the code was written.

But Claude 5 series models perform internal reasoning (thinking) before answering, and this reasoning consumes tokens that also count toward max_tokens. All 120 tokens were used up by the thinking process, and the model was truncated before it could write the first number.

The provider layer threw an error. The reranker caught it as designed:

except Exception:
    return list(docs)   # never worse than the maths alone

Then it returned the mathematical ranking result. No crash, no warning—what the user saw was exactly the same as with --rerank disabled.

The relationship between Extended Thinking and max_tokens is key to understanding this type of failure. Anthropic's Claude 3.7 and later series introduced thinking capability: before giving the final visible answer, the model generates a segment of internal reasoning content. At the API level, this reasoning (returned as a thinking block) shares the same max_tokens budget as the final reply. In other words, max_tokens is the total upper limit for "thinking + output," not merely a limit on the length of the visible reply. If the thinking phase exhausts the budget, the model is truncated before writing any answer, and the API may return an incomplete response or directly error out. Many pieces of code originally written for pure completion models set max_tokens very small (such as 120), and will silently fail after switching to models with thinking. The usual way to avoid this is to explicitly increase max_tokens, or use a separate thinking.budget_tokens parameter to allocate a budget for reasoning alone.

Why the tests didn't catch this bug

The tests in tests/test_rerank.py were not low quality:

• When the model errors, the ranking is unchanged ✓
• When the model returns garbage, the ranking is unchanged ✓
• The model cannot promote tables the caller lacks access to ✓

All passed.

The problem is: every test verified that "failure is safe," and none verified that "the model actually gave an answer."

When the fallback mechanism is designed to be completely transparent to users, 100% functional failure and normal operation look exactly the same externally. This is a particularly insidious class of failure mode in AI systems—not a crash, but a quiet degradation to the fallback path.

Silent Degradation and observability is the general pattern behind this type of failure. When a feature's fallback is designed to be completely transparent to the user—for example, catching exceptions and returning a baseline result that is "never worse"—the system erases the difference between normal operation and total failure in its external behavior. Such failures do not trigger crash alerts or change the API response structure; they only quietly degrade quality to the baseline level. It is especially common in AI systems because model calls themselves are stochastic, and result fluctuations are easily mistaken for "that's just how it is." The countermeasure is to elevate "whether the fallback path was triggered" to a first-class metric: use structured logs, counters, or traces to record every fallback, every empty model reply, every timeout. Only when the number of degradation triggers appears on the monitoring dashboard does green truly mean healthy.

The fix

Three changes plugged the hole:

Raise max_tokens from 120 to 1024. Leave ample room for both thinking and output.

Add a fake model to the tests that refuses to answer when max_tokens is below 512. If someone lowers this value again, the test will directly fail—turning "the model must have a chance to answer" into an executable constraint.

The benchmark now records whether the model gave an answer for each question, not just the final score.

The author verified the fix on the published PyPI package: version 1.4.4 answered only 1 of 3 questions and changed 0 rankings; version 1.5.0 answered all 3 questions and changed 2 rankings.

Real performance after the fix

Once the feature actually worked, the author measured the reranker's effect on public benchmarks for the first time. The test conditions were quite strict: all 876 tables from Spider dev and BIRD dev were placed in the same pool, and the model did not know which database the question belonged to. A table only counted as a top-5 hit if it appeared entirely within the set of tables used by the correct SQL.

Dataset	# Questions	Pure Math Ranking	After Reranking	Rerank-only Correct	Math-only Correct
spider	1034	80.3%	96.5%	169	1
bird	1534	88.6%	96.7%	125	1

Paired McNemar test, p-value approximately 1e-49 on Spider and approximately 1e-36 on BIRD. This is the largest single-item improvement in the benchmarks file.

The boundaries of these numbers need to be clear: they measure whether the correct tables entered the prompt, not whether the final SQL was correct; only one model was used; Spider and BIRD are public datasets and the model may have seen them during training; the reranker only reranks tables already selected by the mathematical method, and it sees table names and short summaries, not the answer SQL.

The biggest takeaway

What should be tested is "the model has a chance to answer," not "it is safe when the model doesn't answer." If your feature relies on a fallback that may silently degrade, you must be able to see in the logs when it is triggered. Otherwise the feature can be 100% broken and your dashboard will still be all green.

schemagate can be installed and tried via pip install "schemagate[ai]", and reranking can be enabled in the MCP server by setting SCHEMAGATE_MCP_RERANK=1. The code and complete benchmark are open source on GitHub.

One-line Summary

An LLM reranking feature silently failed because max_tokens was consumed by thinking, passing all tests while never actually running.

Paragraph Summary

The author of schemagate documents a classic LLM silent failure: the reranking feature had max_tokens=120 entirely consumed by the thinking process of Claude 5 series models, so the model never produced an answer, and the exception-catching fallback quietly returned the mathematical ranking result, causing the feature to fail 100% with no external sign. The root cause was not the model or test quality, but that the test suite only verified "failure is safe," never that "the model actually answered." After the fix (raising max_tokens to 1024, adding a fake model that refuses low budgets, and logging whether the model answered), the reranker's correct-table hit rate on Spider and BIRD jumped from 80.3%/88.6% to 96.5%/96.7%, with McNemar test p-values around 1e-49. The core lesson: safe fallbacks must log when triggered, and tests must cover the success path.