GPT-5.6 Dominates Math but Stumbles on Puzzle Games? Two API Settings Unlock 3x Performance

A Counterintuitive Phenomenon

Recently, OpenAI's GPT-5.6 Sol was used to tackle open problems in mathematics, demonstrating awe-inspiring reasoning capabilities. Yet a fascinating paradox emerged: a model capable of conquering frontier mathematical problems struggles on ARC-AGI-3, a benchmark composed of 2D puzzle games.

ARC-AGI-3 (Abstraction and Reasoning Corpus for AGI) is a general intelligence benchmark designed by François Chollet, creator of Keras, specifically to evaluate AI systems' abstract reasoning and few-shot generalization abilities. Unlike traditional AI benchmarks, each ARC problem consists of a few input-output grid example pairs, requiring test-takers to infer hidden transformation rules from minimal examples and apply them to new inputs. This design deliberately avoids patterns that can be solved through large-scale data memorization, and is considered the "litmus test" for measuring human-like generalization. ARC-AGI-3, as the third-generation version, further increases problem difficulty and diversity, making it one of the most challenging benchmarks for evaluating general AI reasoning capabilities.

This sounds extremely counterintuitive. How can a top-tier model capable of proving mathematical theorems get stumped by simple 2D puzzles? According to the researcher who published this analysis, the root cause isn't the model's intelligence level itself, but rather a problem with the "harness" running the model.

In other words, there's nothing wrong with the model's "brain"—the problem lies in the "nervous system" connecting the brain to the task environment. This discovery carries significant implications for understanding the capability boundaries of today's AI Agents.

The Culprit: Memory Loss in the Running Harness

After deep investigation, researchers found that the core issue blocking GPT-5.6 was: the running harness didn't let the model remember what it had already learned.

In modern AI Agent architectures, the "harness" refers to the middleware software system connecting large language models to external task environments. It manages critical functions including conversation turns, context windows, tool calls, and memory storage. Common Agent frameworks like LangChain, AutoGPT, and CrewAI all attempt to solve these problems. OpenAI's API itself provides multiple state management mechanisms, including the threads concept in the Assistants API, automatic conversation history management, and newer stateful session features. If the framework fails to properly utilize these mechanisms, each API call becomes a stateless independent request, and the model loses the ability to accumulate cognition across multiple interaction turns—which is precisely the predicament GPT-5.6 faced on ARC-AGI-3.

This is key to understanding the phenomenon. Puzzle game benchmarks like ARC-AGI-3 fundamentally test not one-shot static reasoning, but rather the ability to continuously learn and iterate strategies through interaction. Players (or AI) need to gradually understand game rules across multiple attempts, remember lessons from previous failures, accumulate environmental awareness, and then adjust strategies.

Why Math Problems Are Actually Easier for GPT-5.6

By contrast, solving open mathematical problems is typically a form of "deep single-pass reasoning"—the model can conduct long-chain thinking within a continuous, complete context. 2D puzzle games, however, represent a classic "multi-step interactive task": after each operation, the environment provides feedback, and the model must decide its next step based on historical information.

Cognitive science draws a clear distinction between these two problem-solving paradigms: the former belongs to "System 2" deep analytical reasoning, characterized by long-chain logical deduction within a single cognitive framework—such as mathematical proofs and code generation—corresponding to Transformers' powerful ability in single-pass chain-of-thought reasoning. The latter belongs to "exploration-exploitation" interactive learning, requiring continuous revision of mental models through environmental feedback and information integration across multiple time steps. The reinforcement learning field has long recognized this fundamental distinction—even the strongest policy networks cannot effectively learn sequential decision tasks without experience replay and state memory mechanisms. LLM Agents face this exact same fundamental challenge in multi-step interactions.

If the running harness "clears" the model's memory at each interaction turn, or fails to correctly pass previously learned experience to the next round, the model essentially starts "from scratch" at every step. It cannot accumulate experience and therefore cannot form effective strategies. It's like having a mathematical genius play a game, but erasing their memory after every move—even the brightest mind would be paralyzed.

The Dramatic Improvement from Two API Settings

The most exciting finding is that the solution was unexpectedly simple. Researchers reported that merely by enabling two API settings, test scores tripled, while output token consumption decreased by 6x.

This is a win-win result—performance improved dramatically while costs dropped significantly. Although the original source didn't detail the specific names of these two API settings, from context we can reasonably infer they likely relate to mechanisms for context memory retention and reasoning state reuse, allowing the model to continuously retain and access previously acquired knowledge across multiple interaction turns.

From a technical perspective, a large language model's "memory" fundamentally depends on information retained within the context window. In multi-turn interaction scenarios, there are several main memory retention strategies: first, complete history retention, passing all conversation history as prompt input (limited by context length); second, summary compression, where the model or external system summarizes historical information before passing it in; third, API-native session state management, where the server maintains conversation state without requiring the client to repeatedly transmit complete history. Multiple API features OpenAI launched between 2024-2025, including reasoning state caching and persistent threads, were designed precisely to solve these problems. Correctly enabling these features allows the model to reuse intermediate reasoning results across multiple turns, avoiding redundant computation, thus simultaneously achieving performance improvement and cost reduction.

A Dual Breakthrough in Efficiency and Capability

The 6x reduction in output tokens deserves particular attention. It means that when the model can correctly "remember" context, it doesn't need to re-derive and re-explain things in every round, dramatically saving computational overhead. This isn't merely a score improvement—it's a qualitative transformation in reasoning efficiency.

Understanding this number from a token economics perspective: in LLM API business models, billing is typically calculated by input and output token volume, with output tokens usually priced 2-4x higher than input tokens. When a model lacks memory and must re-derive the same content in every interaction turn, it generates massive redundant output tokens—the model needs to repeatedly "talk to itself" to rebuild its understanding of the problem. At GPT-5 level pricing (referencing current o1/o3 series pricing of $15-60 per million output tokens), a 6x reduction in output tokens means order-of-magnitude cost savings in large-scale deployment scenarios. This also explains why inference efficiency optimization is becoming one of the core topics in AI engineering.

For developers actually deploying AI Agents, this number directly impacts costs: the same task might cost only one-sixth of the original price while delivering three times better results. Combined, the cost per unit of performance drops approximately 18x—enough to transform the commercial viability of an AI application project.

Practical Implications for AI Agent Developers

This case reveals a severely underestimated aspect of current AI application development: a model's ability to perform depends heavily on the design of its running harness.

We habitually equate a model's performance directly with its inherent intelligence level. When a model performs poorly on a benchmark, the first reaction is often "this model isn't powerful enough." But this case reminds us the answer might be exactly the opposite—the model is powerful enough, we just haven't given it an environment where it can fully demonstrate its capabilities.

Harness Design Determines the Model's Performance Ceiling

For teams building AI Agents, here are several direct practical implications:

• Prioritize context management: In multi-turn interactive tasks, how you retain, organize, and pass historical information is often more important than which model you choose. Specifically, developers need to find a balance between "complete history retention" (information-complete but high token cost) and "intelligent summarization" (saves tokens but may lose critical details). Next-generation frameworks like MemGPT's hierarchical memory architecture—mimicking operating system virtual memory management to handle LLM context—represent cutting-edge exploration in this direction.

• Leverage API-native capabilities: Many model APIs already have built-in memory and state reuse mechanisms; correctly enabling them may yield benefits far exceeding expectations. Developers should carefully read API documentation regarding session management, caching, and state persistence features, rather than focusing solely on model selection and prompt engineering. Often, an overlooked API parameter might be the cure for a performance bottleneck.

• Cost and performance can be achieved simultaneously: Optimizing the harness doesn't necessarily mean increasing costs; proper configuration may actually improve results while reducing overhead. This overturns the traditional belief that "better performance necessarily requires more computational resources," revealing the leverage effect of engineering optimization in AI applications.

• Benchmark results require careful interpretation: When we see a model's score on a specific benchmark, we should ask: does this score reflect the model's capability ceiling, or its performance under a particular testing framework? Testing conditions, API configurations, and context management strategies can all significantly affect final results.

Conclusion

The paradox of GPT-5.6 Sol "dominating math yet stumbling on puzzle games" ultimately points to a simple but profound truth: an AI system's overall performance is the combined result of model capability and engineering framework.

In an era of increasingly powerful large models, designing running environments that can truly unleash these capabilities is becoming the decisive factor in AI application success or failure. Sometimes, what you need isn't a smarter model, but a framework that better understands how to use it. This case of achieving 3x improvement and 6x cost reduction through just two API settings is the most vivid illustration of this principle.

This also means that in the competitive landscape of AI applications, the real moat may not lie in who can access the strongest foundation model—since model capabilities are rapidly commoditizing—but in who can build the most sophisticated engineering framework, making the same model perform far beyond others' results in their hands. Harness engineering is moving from backstage to center stage, becoming one of the core competitive advantages of the AI era.