GPT-5.6 Inference at 18x Speed: OpenAI's Compute Gamble and Business Logic

Smarter or Faster? A Question Worth Asking

When Sam Altman asked Andrej Karpathy whether large language models should be made smarter or faster, Karpathy answered: "smarter models." That reflects most people's intuition — all else being equal, we prefer a smarter model. When reading benchmarks, we instinctively check intelligence first, speed second.

But does speed really not matter? The reality is that most flagship models today are served within a surprisingly narrow speed range. According to OpenRouter data, Opus 4.8 runs at roughly 60 tokens/s, Fable 5 at around 40 tokens/s, and GPT-5.6 Sol similarly at about 40 tokens/s. This raises a core question: why don't these companies simply push their flagship model output from 60 tokens/s to 750 tokens/s?

The experiential difference between 60 tokens/s and 750 tokens/s is enormous. So why aren't OpenAI, Anthropic, xAI, and Google pursuing both "smarter" and "faster" at the same time?

[Background: The Physical Limits of Inference Speed] The inference speed of large language models (typically measured in tokens/s) is fundamentally constrained by the nature of autoregressive generation: the model can only produce one token at a time, and each token depends on all preceding tokens. This serial dependency means the core bottleneck to speeding things up is memory bandwidth, not raw compute. In modern LLM inference, a GPU's VRAM bandwidth is often the scarcer resource compared to floating-point compute capacity — because generating each token requires reading the entire model's weights from memory. Take a 2-trillion-parameter model: even with FP8 quantization, the weights alone occupy 2TB. Generating a single token requires a full scan of that 2TB. This is precisely why specialized inference chips from companies like Groq and Cerebras can achieve order-of-magnitude speed improvements — they use fundamentally different chip architectures that tightly integrate compute units with memory, dramatically increasing memory bandwidth density rather than simply stacking more compute.

The Hardware Barrier: The Gap Between GPUs and Specialized Chips

The answer starts with hardware costs. The further right you move along the speed axis, the higher the capital expenditure (Capex) required to serve at scale — and it all comes down to the cost of the chips.

There's a clear hardware divide: on one side, general-purpose GPUs from NVIDIA and AMD; on the other, specialized inference chips from Groq and Cerebras. The price difference is staggering — a Cerebras system can easily cost 20 to 50 times more than a comparable GPU setup.

[Background: Cerebras WSE Chip Architecture] The Cerebras WSE (Wafer Scale Engine) is the largest chip in the world, with the unique design of turning an entire wafer into a single chip rather than packaging multiple chips together. The WSE-3 measures approximately 46,225 mm², compared to just ~814 mm² for an NVIDIA H100. The key advantage of this design is its massive on-chip SRAM capacity (44GB on the WSE-3), with access latency far lower than the HBM (High Bandwidth Memory) that NVIDIA GPUs rely on. For inference workloads, if model weights can fit entirely in on-chip SRAM, the memory read latency per token generation drops by an order of magnitude, enabling extreme speeds. The trade-off is obvious: a single WSE-3 may cost millions of dollars, and wafer-scale manufacturing yield control is extraordinarily complex — which directly drives that 20–50x cost gap. OpenAI's decision to rent rather than buy Cerebras compute reflects that very cost pressure.

Interestingly, OpenAI offers GPT-5.6 Sol in two ways simultaneously: a traditional GPU-based version running at 40–50 tokens/s, and a Cerebras-powered version running at up to 750 tokens/s. If achieving an 18–20x speed boost requires such enormous capital expenditure, why would OpenAI still do it?

The Legacy of "Fast but Dumb"

In fact, OpenAI previously released a model exceeding 1,000 tokens/s — GPT-5.3 Codex Spark. But the general consensus was that it was "fast but dumb." The reason: to run faster, it sacrificed intelligence — typically by making the model smaller.

This created a long-standing assumption: as you move right along the speed axis, total market demand gradually dries up. Vendors tend to push small models into the high-speed range because it's more economically efficient; users, meanwhile, would rather use a 50 tokens/s flagship than something 20x faster but much less capable.

By deploying its largest, most intelligent model at both ends of the speed spectrum simultaneously, OpenAI is now upending that long-held assumption.

The Economics of Compute: Throughput vs. Responsiveness

OpenAI hasn't disclosed GPT-5.6 Sol's parameter count, but it's reasonable to assume at least 2 trillion parameters, possibly up to 4 trillion. Getting a model of that scale from 40 tokens/s to 750 tokens/s is no small feat.

To understand this, consider an NVIDIA chart with throughput on the vertical axis and responsiveness on the horizontal axis.

For example, if you commit to serving every user at 340 tokens/s, using NVIDIA B300 chips in a 1-megawatt data center, you can achieve roughly 450,000 tokens/s total throughput — that's approximately 1,324 concurrent active users, each getting up to 340 tokens/s. As long as simultaneous users don't exceed that number, you can honor your SLA commitment.

[Background: SLA and Concurrent Capacity Planning] An SLA (Service Level Agreement) is a formal commitment between a cloud provider and its users regarding service quality — in AI inference contexts, this typically covers latency, throughput, and availability. The concurrent active user count calculation described here involves a key engineering concept: the trade-off between TTFT (Time to First Token) and TPOT (Time Per Output Token). When a provider commits to high speeds (e.g., 750 tokens/s), they need not only the hardware to support it but also enough capacity to avoid slowdowns during peak concurrent demand — requiring significant compute redundancy. It's worth noting that real users are not always in an active generation state; there are large idle intervals between requests (think time), so the true number of concurrent active users is often a fraction of total online users. OpenAI must precisely model this distribution when planning pricing and capacity, in order to offer high-speed SLA guarantees without operating at a loss.

Switch to Cerebras chips, however, and the picture changes dramatically. Assume the same 450,000 tokens/s total output, but now each user gets 750 tokens/s — concurrent active users drop to just ~600, more than half fewer than the 340 tokens/s scenario. Double the speed, cut concurrent capacity in half.

The hardware cost difference is even more striking: assuming GPT-5.6 Sol has 2 trillion parameters (a conservative estimate), you'd only need 14 B300 NVIDIA chips to support it, versus 90 WSE3 Cerebras chips. The cost gap is enormous. So is this move worth it for OpenAI?

The Business Logic: Speed as a Tool to Burn Through Quotas

This is where things get genuinely interesting. Codex already offers a "1.5x speed" option, explicitly noting that it consumes more usage. The key word: "usage."

To grow revenue, OpenAI needs to convert users from free subscriptions to $200/month Pro memberships. One of the most effective ways to drive that conversion is to make users burn through their quotas faster, constantly nudging them to upgrade to higher tiers.

The $10 Billion "Renting Compute" Bet

Back to the GPU vs. Cerebras cost gap: this time, OpenAI's capital expenditure isn't in hardware purchases — they didn't buy Cerebras chips. Instead, they signed a $10 billion agreement with Cerebras to rent 750 megawatts of compute capacity in their data centers, with a contract running through 2028.

That gives OpenAI roughly 30 months to make that $10 billion deal pay off. A simplified back-of-the-envelope calculation: at $200/month for a Pro membership, 30 months generates roughly $6,000 per user in revenue — meaning OpenAI theoretically needs to convert about 1.6 million users to Pro. Against its 1 billion monthly active users, that's roughly a 0.16% conversion rate.

The fastest way to drive that conversion is to accelerate token quota consumption and charge a premium for the "fast processing" enabled by Cerebras chips. Just as the 1.5x speed option consumes extra usage, Cerebras chips capable of 750 tokens/s will drain quotas at a much faster rate — depending on how OpenAI optimizes pricing for that hardware.

Of course, OpenAI's revenue isn't limited to subscriptions. It also includes enterprise customers and the API. Low-latency applications may call Cerebras-backed inference via API to support workflows requiring fast reasoning — all of which collectively drive demand for Cerebras compute.

The Overlooked Third Dimension: Token Efficiency

There's one more factor we haven't discussed — token efficiency, or token cost. It adds yet another dimension to the equation and could once again upend the economics.

An increasing number of models (such as xAI's latest Grok 4.5) are becoming more token-efficient, completing the same tasks with fewer tokens. In a sense, token efficiency and token speed are in tension — frontier labs fundamentally want to encourage more token consumption to generate more revenue, while improvements in token efficiency directly compress that opportunity.

[Background: Token Efficiency and the Reasoning Model Trade-off] Token efficiency refers to the number of tokens a model consumes to complete a given task. It's a new dimension that has gained significant attention alongside the rise of reasoning models. Early Chain-of-Thought prompting techniques improved model accuracy but also dramatically increased output token counts — some reasoning models might generate thousands of "thinking tokens" to solve a single math problem. This verbose reasoning chain gave rise to the problem of "token over-consumption," prompting researchers to develop more compact reasoning paths. Techniques include chain compression, reasoning step pruning, and training models via reinforcement learning to "reach the right answer in fewer steps." Improvements in token efficiency in models like Grok 4.5 are partly the result of knowledge distillation — compressing the reasoning capabilities of large models into more compact representations. The tension with commercial interests is real: consuming fewer tokens means a better user experience but also lower billed revenue. Frontier labs must find a dynamic balance between product reputation and revenue generation.

Speed, intelligence, and efficiency — intertwined, they form the most compelling game playing out in the AI inference race. The trend we're seeing is that frontier labs are competing not just on whether their models are smarter, but on whether they're faster and more cost-efficient. This three-dimensional competition has only just begun.

Key Takeaways