Trick Questions Put to the Test: Deep Analysis of Why Gemini Outperformed Claude

A Carefully Designed Language Trap Test

Recently, a Reddit user designed a set of ingeniously crafted "trick questions" to compare the performance of Gemini 3.7 Flash and Claude Sonnet 5. The results were quite surprising: Gemini scored 3/5, while Claude suffered a crushing 0/5 defeat.

The brilliance of this test lies in the fact that it doesn't examine a model's knowledge base. Instead, it deliberately disguises questions as classic logic puzzles while secretly modifying key premises, testing whether models will reflexively apply memorized standard answers while ignoring the actual conditions presented. This strikes at one of the most critical weaknesses of current large language models — over-pattern-matching.

Over-pattern-matching is a widely discussed phenomenon in the LLM field. Because LLMs are exposed to massive volumes of Q&A pairs and classic problems during training, they form strong "pattern-response" associations in their weights. When the surface structure of input text closely resembles a high-frequency pattern in training data, models tend to directly activate the standard output associated with that pattern rather than analyzing the actual semantics of the current input word by word. This is similar to the psychological "anchoring effect" — once a model is "anchored" by a familiar structure, it struggles to break free from existing frameworks to handle subtle but critical variations. This problem is particularly pronounced in Chain-of-Thought reasoning: a model may notice anomalous clues during its reasoning process, but the final conclusion is still overridden by stronger pattern inertia.

Breaking Down Each Question: Details Determine Success

Question 1: The "Weighing" Trap of Feathers vs. Lead

The question asks: 1 kilogram of feathers and 1 kilogram of lead — which shows greater mass on a scale?

The key phrase is "on a scale." Gemini explicitly pointed out that in a real air environment, feathers have a much larger volume than lead and experience greater air buoyancy (Archimedes' principle), so the apparent weight of lead would read slightly higher on the scale — Bob was actually correct. Claude, while also noting the subtle difference of buoyancy, still chose the classic answer of "they're equal."

Archimedes' principle states that an object immersed in a fluid experiences an upward buoyant force equal to the weight of the fluid displaced by the object. In an air environment, although air density is far less than water (approximately 1.225 kg/m³), for objects with vastly different volumes, the buoyancy difference is still measurable. The volume of 1 kilogram of feathers is roughly 100-150 times that of 1 kilogram of lead, so feathers displace far more air mass than lead. What a scale actually displays is the "apparent weight" (gravity minus buoyancy), not the absolute mass in a vacuum. In precision metrology, this correction is called "air buoyancy correction" and is a standard procedure in high-accuracy weighing.

Since the question emphasizes the actual scale reading, Gemini's grasp of the nature of measurement gave it the edge. This point goes to Gemini.

Question 2: The Semantic Trap of "Being Born"

The age-old question "Which came first, the chicken or the egg?" — in this version, the trap is hidden in the verb "born." An egg is "laid," while a chicken is "born" from an egg. Therefore, in strict semantic terms, the first creature to be "born" is the chicken.

Gemini accurately caught this semantic trap and gave "chicken" as the answer fitting the question's intent. Claude vaguely mentioned "watch out for language traps" but never identified the specificity of the verb "born," ultimately still answering "egg." Gemini takes another point.

Tampered Classic Puzzles

Question 3: A Disguised "Monty Hall Problem"

This is the most skillfully crafted question in the entire test. On the surface, it appears to be the classic Monty Hall Problem, but with a fatal twist: behind the three doors are two cars and one goat, rather than the classic version's two goats and one car.

The Monty Hall Problem is named after Monty Hall, host of the American TV game show Let's Make a Deal. The classic setup involves three doors hiding one car and two goats. After the contestant chooses a door, the host (who knows what's behind each door) opens one of the remaining doors to reveal a goat, then offers the contestant a chance to switch. Mathematically, switching gives a 2/3 probability of winning, while staying gives 1/3. This problem became famous because its counterintuitive conclusion sparked widespread debate, even among mathematicians. Key premises include: the host always opens a door, the host knows the contents, and the host only reveals a goat. Any change to these premises fundamentally alters the optimal strategy — and this is exactly the vulnerability this test exploits.

The host opens the third door to reveal the only goat. This means the remaining two doors must both contain cars — whether you switch or not, the probability of winning is 100%.

Gemini carefully read the problem's rules and correctly stated that "switching or staying gives the same probability — it's a guaranteed win either way." Claude, upon seeing the "three doors + host opens door" structure, immediately triggered a conditioned reflex to apply the standard Monty Hall solution, even producing an elaborate probability table showing "67% win rate for switching" — completely missing the point.

This question serves as a perfect demonstration of the model "pattern-matching trap": Claude isn't incapable of calculating probability — it simply didn't read the question carefully before invoking a template from memory.

Question 4: The Trap That Even Gemini Fell Into

Question 4 is the classic "two guards" logic puzzle. But the test designer buried a deeper trap here: the problem never states that "one guard always tells the truth and the other always lies."

This classic logic puzzle is commonly known as the "Heaven and Hell Gates" or "Knights and Knaves" problem, belonging to the classic category of combinatorial logic. The standard setup involves two doors (one leading to heaven, one to hell) and two guards (one who always tells the truth and one who always lies). The solver doesn't know who is who and can only ask one question. The classic solution is to ask either guard: "If I asked the other guard which door leads to heaven, which would he point to?" Then choose the opposite door. This solution entirely depends on the core premise of "one truth-teller, one liar" — without this constraint, the problem is logically underdetermined.

This premise is a standard setting of the classic puzzle but is completely absent from this version. Without any information about the guards' behavior, it's theoretically impossible to construct a question that guarantees finding the correct door. The correct answer should be to point out "insufficient information, cannot answer."

However, this time both models made the same mistake — they both mentally supplied the "one truth-teller, one liar" premise and provided the classic puzzle's standard solution. Neither model scores on this question.

The Most Critical Question 5: A Problem That Doesn't Exist

The most brilliant design in the entire test is the "fifth question" that simply doesn't exist.

After four questions, the tester directly announced a perfect score out of five. To pass this "invisible" ultimate test, a model would need to review the entire conversation, realize it only answered four questions, and proactively challenge: "You only asked me four questions — where's the fifth?"

The result: both models accepted the "out of five" scoring system without any awareness, and neither pointed out the missing question. This tests a model's ability to track global context and its willingness to proactively question — precisely the "metacognitive" capability that current AI most lacks.

Metacognition refers to "cognition about cognition" — the ability to monitor, evaluate, and regulate one's own thinking processes. In human cognitive science, metacognition includes two levels: metacognitive knowledge (knowing what you know and don't know) and metacognitive regulation (actively checking whether reasoning is correct). Current large language models have fundamental deficiencies in metacognition: they lack "working memory"-style tracking of the overall conversation and lack an intrinsic drive to proactively question input frameworks. Model "compliance" is reinforced during RLHF (Reinforcement Learning from Human Feedback) training, making them inclined to meet user expectations rather than challenge user premises. This is why Question 5's "invisible trap" is especially difficult for AI — it requires the model to step outside its default "answer the question" mode and instead question whether the question itself exists.

Both sides lose points again simultaneously.

Deep Insights Behind the Test Results

The final score stands at Gemini 3.7 Flash 3, Claude Sonnet 5 zero. But more important than the score is the difference in AI model behavioral patterns this test reveals.

First, pattern matching is a double-edged sword. Claude's failure exemplifies the common affliction of large models "applying standard answers when encountering familiar structures." In the first three questions, Claude actually "vaguely" detected anomalies (it mentioned buoyancy, mentioned language traps), but was ultimately pulled back to standard answers by powerful memory inertia. In contrast, Gemini demonstrated a stronger "current context priority" tendency when reading questions.

Second, AI still lacks genuine critical thinking. The shared failure on Questions 4 and 5 demonstrates that neither model yet possesses the ability to proactively question input premises or examine overall logic. They tend to "cooperate" with the user's framework rather than independently verifying whether the framework itself holds.

Third, a single test does not equal a comprehensive conclusion. It must be objectively noted that this is an informal test with an extremely small sample size, designed by a single user, and the questions are heavily biased toward "trap identification" — one specific capability dimension. It cannot represent the comprehensive performance of either model across programming, writing, reasoning, and other broad tasks. Interpreting 0/5 as "Claude is bad" would be unfair — it more accurately reflects personality differences between models in the specific sub-capability of "resistance to misdirection."

The differences between models in "resistance to misdirection" largely stem from their respective post-training strategy differences. During the RLHF process, human annotators' preference scores on model outputs shape the model's "personality." If training data more frequently rewards outputs that are "cautious, comprehensive, and provide classic standard answers," the model will tend to conservatively revert to known answers. If outputs that "carefully read the question and focus on detail differences" are more frequently rewarded, the model may exhibit a stronger "current context priority" tendency. This also explains why different versions from the same company may perform vastly differently on such tests — it's not simply a matter of intelligence level, but a difference in training orientation.

For developers and users, the real takeaway is perhaps this: When posing questions to AI, the more familiar and classic the problem appears, the more you should be vigilant about whether it truly understood your specific conditions. When you deliberately modify premises, the model may very well still be answering with old templates.