GPT-5.6 Codex Three-Model Showdown: Sol/Terra/Luna Tested on Classic Candy Logic Puzzle

One Logic Puzzle That Reveals the Real Gap Between Models

As OpenAI's Codex series continues to evolve, the question on everyone's mind remains the same: does the new version actually deliver? This evaluation uses a classic "candy logic puzzle" sourced from GitHub to test the real reasoning capability of GPT-5.6 (the current version of Codex), while also checking whether it suffers from the "dumbing down" issue criticized in version 5.5.

The puzzle itself isn't overly complex, yet it's remarkably good at separating capable models from weaker ones. A bag contains three flavors of candy in different shapes (round, star-shaped, etc.), and the tester must draw randomly and use the given conditions to reason toward the correct answer. This type of problem has become a community staple for stress-testing precisely because it sits at the intersection of conditional probability and combinatorial logic — the model must simultaneously track constraints across multiple variables (flavor, shape, draw order) and arrive at a single correct conclusion from limited information. It doesn't rely on memorization or pattern matching; it forces the model to construct a genuine multi-step reasoning chain. For large language models, this hits a known architectural weakness — autoregressive generation naturally favors fluent output but tends to fail in scenarios requiring "backtracking and correction." These multi-constraint puzzles are therefore excellent at exposing the gap between model tiers.

Test Design: Three Model Tiers × Two Reasoning Intensities

The centerpiece of this evaluation is a side-by-side comparison of three differently named model configurations — Sol (Sun), Terra (Earth), and Luna (Moon) — which can be understood as distinct tiers or branches within GPT-5.6 Codex. Naming model tiers after celestial bodies is a common strategy in the AI industry for communicating capability levels. Analogous to Anthropic's Claude Haiku/Sonnet/Opus lineup or Google's Gemini Flash/Pro/Ultra hierarchy, this naming convention implies a gradient from lightweight to flagship — Luna typically maps to a low-latency entry-level model, Terra sits in the middle, and Sol represents the most powerful configuration. The business logic behind this tiered strategy is straightforward: cover diverse user needs with differentiated cost and performance, letting developers choose the right tier based on task complexity.

Each model was tested under two reasoning intensities:

• HIGH: Standard high-intensity reasoning mode
• XH (Extra High): Higher-intensity reasoning mode with extended thinking time

Each configuration was run independently 10 times, observing result stability and accuracy across multiple rounds rather than drawing conclusions from a single lucky run. This "multi-round benchmarking" methodology is far more rigorous than one-shot testing because LLM inference is influenced by the sampling temperature parameter — the same input can produce different outputs, making single tests highly susceptible to randomness. Ten repeated runs provide a reasonable estimate of a model's "expected accuracy," building a statistically credible capability profile. The gap between Sol's high accuracy and Luna's zero accuracy far exceeds statistical noise, making the conclusions quite reliable.

Sol Shines: Near-Perfect Accuracy and Consistency

Among all tested configurations, Sol is the clear standout. In HIGH mode across 10 runs, the first 6 were all correct, demonstrating exceptional stability. XH mode lived up to expectations as well, maintaining perfect scores across multiple runs.

Notably, XH mode takes significantly longer to think, which aligns intuitively with its higher reasoning intensity. This reflects one of the core research directions in modern LLMs — Test-Time Compute: newer models incorporate dynamic reasoning budget mechanisms that allow extended internal "thinking" before producing a final answer, essentially generating intermediate reasoning steps via Chain-of-Thought (CoT). XH mode trades more tokens and compute for logical rigor, at the cost of higher latency and API expenses — but for demanding tasks like logical reasoning and mathematical proofs, the accuracy gains are often well worth it.

Consistency Is the Real Benchmark

When evaluating LLM capability, a single correct answer proves nothing — multi-run consistency is the true differentiator. Sol maintaining near-perfect accuracy in both HIGH and XH modes indicates genuine understanding of the underlying logic structure, not a lucky guess. This directly addresses the community's biggest concern about "dumbing down" — at least at the Sol tier, the answer is a clear no.

Terra and Luna: A Cliff-Edge Performance Drop

Contrasting sharply with Sol's strong showing, Terra and Luna delivered disappointing results.

Terra (Earth) in XH mode answered correctly only 2 out of 10 times, for a 20% accuracy rate. More surprisingly, Terra's HIGH mode was wrong all 10 times.

Luna (Moon) fared even worse: regardless of whether HIGH or XH mode was used, all 10 runs were incorrect, for a 0% accuracy rate.

This data makes the capability chasm between model tiers unmistakably clear. When faced with multi-constraint logical reasoning, Terra and Luna are nearly incapable of producing the correct answer, exposing significant weaknesses in complex reasoning tasks.

Why Is the Gap So Large?

From a technical perspective, this cliff-edge performance gap can be understood along three dimensions:

First, parameter scale effects — smaller models tend to degrade exponentially, not linearly, on long reasoning chain tasks, a consequence of capacity bottlenecks in handling multi-step dependencies. Second, depth of reasoning fine-tuning — flagship models like Sol typically undergo more extensive RLHF (Reinforcement Learning from Human Feedback) and task-specific training for reasoning, while streamlined versions receive far less investment in this area. Third, context tracking ability — multi-constraint logic puzzles require the model to continuously track all known conditions during generation; lightweight models are more prone to "condition forgetting," causing the reasoning chain to break midway. The "dumbing down" phenomenon has been documented multiple times in the AI community, typically stemming from training data ratio adjustments or capability drift caused by over-optimization during RLHF. Sol, as the highest-performing configuration, likely corresponds to a more powerful base model with more thorough reasoning optimization, while Terra and Luna may be streamlined or experimental variants that struggle with high-difficulty logic problems.

Conclusion: Sol Genuinely Delivers — Don't Judge a Model by Version Number Alone

Based on this comprehensive GPT-5.6 Codex candy puzzle evaluation, the conclusion is clear: Sol genuinely delivers. It maintained high accuracy and strong consistency in both HIGH and XH modes, demonstrating solid logical reasoning ability with no evidence of capability degradation.

By contrast, Terra and Luna fell far short of expectations and are largely unfit for complex reasoning tasks. This serves as an important reminder: when choosing between model tiers, don't rely solely on the version number — multi-round testing on real-world tasks is the only reliable way to assess model capability.

For developers and power users, multi-run evaluations using real logic problems like this often provide more actionable insight than official benchmark numbers. Formal benchmarks (such as MMLU, HumanEval, etc.) are comprehensive but carry risks of test set leakage and targeted optimization. Real logic puzzles run multiple times are much harder to "game" and more objectively reflect how a model will perform in actual deployment. If you want to verify the true reasoning capability of a model you're working with, pick a few classic logic problems and run them repeatedly — the results may surprise you.

Key Takeaways