OpenAI's First Inference Chip Jalapeño Benchmarked: Does It Really Crush NVIDIA's GB300 in Efficiency?

From Buyer to Definer: A New Playbook for Compute Competition

For the past few years, the core mission of AI companies could be summarized in a single sentence: figure out how to buy more NVIDIA chips. Now, the rules of the game are changing. The most capable model companies are no longer content to simply consume compute — they're starting to define it themselves.

OpenAI has just published the first benchmark results for Jalapeño (Spanish for "chili pepper"), its debut in-house inference chip. The chip has a clear mandate: it's not responsible for training the next generation of GPT models. Instead, it's purpose-built for inference — the layer that actually delivers answers when a user asks a question, an agent calls a tool, or a model generates tokens one by one.

This distinction matters enormously. Training determines what a model can learn; inference determines whether those capabilities can be delivered to every user quickly, cheaply, and reliably enough to matter. The more powerful the model, the greater the inference demand; the more steps an agent takes, the more latency and energy costs accumulate at every layer. What Jalapeño is designed to address is the reality of hundreds of millions of real requests every day: how much electricity they consume, how many machines they occupy, and how long users have to wait.

Benchmark Results: 1.5x to 1.9x Performance per Watt

OpenAI used Semi-Analysis's public benchmark suite InferenceX, running tests on three open models: GPT-OSS 120B, DeepSeek^Ad R1 670B, and Kimi K2 1T. These models come from different teams and vary widely in scale, which lends some credibility to the claim that the architectural advantages aren't just tailored to OpenAI's own models.

The headline numbers can be summarized as follows:

• At peak throughput, Jalapeño delivers 1.5x to 1.9x more AI work per watt compared to the reference systems
• End-to-end latency is approximately 28% to 59% of the reference systems
• In interactive, low-latency workloads, performance reaches 2.1x to 4.1x that of the reference systems
• Rated at 700W, with sustained real-world power draw not exceeding 550W (compared to GB200's 1,200W rating and GB300's 1,400W rating)

What's particularly notable is OpenAI's framing for the comparison. Rather than simply benchmarking peak FLOPS per chip, the question becomes: at a response speed users find acceptable, how much useful work can you get done per 1,000 watts? This framing more accurately captures the real tension in inference businesses — a system can stack up huge numbers of requests to achieve high aggregate throughput, but individual users may wait a long time; or it can concentrate resources on fewer requests to achieve extremely low latency, at the cost of overall utilization.

OpenAI describes Jalapeño's current state as sitting on the "Pareto frontier," meaning it's now very difficult to find a configuration that improves one metric without sacrificing another.

What's Behind That 104.3x Figure?

The widely circulated "104.3x" figure appears in the DeepSeek R1 670B results, but it does not mean Jalapeño is 100x faster than GB300 in all scenarios. What it actually means: when throughput speed is normalized to approximately 169 tokens per second — the level GB300 could previously achieve — Jalapeño delivers 12,258 tokens per kilowatt, versus 118 for GB300, a ratio of roughly 104.3x. When comparing peak throughput per watt at each system's own maximum, the lead is actually around 1.7x.

Taking GPT-OSS 120B as an example, Jalapeño achieves a minimum inter-token latency of 0.69ms (1,459 tokens per second for a single user), versus 1.69ms for GB200; end-to-end latency is 1.03 seconds versus 1.80 seconds. On Kimi K2 1T, Jalapeño's peak throughput per watt is approximately 1.5x that of GB300, with latency of 1.56 seconds versus 5.31 seconds.

InferenceX is a inference benchmark suite developed by Semi-Analysis — an independent research firm focused on chips and data centers — designed to simulate real-world business scenarios rather than pure peak compute contests. Its core metric is the latency-throughput tradeoff curve: the x-axis sets a maximum acceptable response latency for users, while the y-axis measures how many requests a system can handle at that constraint and how much energy it consumes. This testing approach forces systems to make real tradeoffs between "quality of service" and "resource utilization," rather than inflating peak throughput by batching requests. By contrast, traditional FLOPS or TOPS metrics only reflect theoretical compute ceilings and fail to capture memory bandwidth bottlenecks, scheduling overhead, and network transfer latency — factors that are often decisive in real-world inference. Understanding this testing framework is a prerequisite for reading Jalapeño's numbers accurately.

The Advantage Isn't Just the Chip — It's Full-System Co-Design

Why does Jalapeño pull ahead so significantly in high-interactivity, low-latency regimes? The answer isn't just the chip itself. Large model inference breaks down into roughly two phases: the Prefill phase, where the system processes the prompt and is more compute-bound; and the Decode phase, where the model generates tokens one by one and is more memory bandwidth-bound. Meanwhile, model state must move between cores, chips, and racks — no matter how fast the compute units are, if the data is still in transit, everything waits.

Jalapeño was designed from the start with the chip, memory, networking, software, and rack treated as a unified whole. Model state — including the KV Cache — can be explicitly placed and kept resident, with the system dispatching appropriate resources based on the differing demands of Prefill and Decode. A large connectivity domain keeps more work within the same system, reducing the idle time caused by moving data across devices.

One commentator offered a useful analogy: it's like a kitchen designed specifically for large language models. The traditional approach is to buy the best stove, refrigerator, and conveyor equipment separately and then assemble them. OpenAI instead studied what dishes need to be prepared every day first, then decided where to store ingredients, how staff should move, and how much heat each step requires. No single component fully explains the advantage, but from the moment the order enters the kitchen to the moment it's served, waiting and moving are both reduced.

KV Cache (Key-Value Cache) is a critical mechanism for understanding large model inference efficiency. When generating each new token, a Transformer architecture needs to "look back" at the Key and Value vectors corresponding to all previous tokens. If these were recomputed at every step, the computational cost would grow quadratically with context length. KV Cache works by computing these vectors once and storing them in high-speed memory for subsequent steps to read directly. This means the access speed and capacity of the cache directly determine the performance bottleneck in the Decode phase. For long conversations and multi-turn agentic tasks, KV Cache can occupy tens of gigabytes of memory; in multi-user concurrent scenarios, how to efficiently share, evict, and schedule the cache across different requests is one of the central design challenges of inference system architecture. Jalapeño's treatment of KV Cache placement and retention as a system-level design objective is a targeted optimization aimed squarely at this bottleneck.

AI Designing AI: A Feedback Loop That's Closing

Perhaps even more noteworthy: AI isn't just what this chip will serve in the future — it directly participated in building it. OpenAI reports that from initial design to tape-out, the team took only nine months. Models at different stages helped engineers explore implementation options, shortened the design-measure-verify cycle, and contributed to optimizing arithmetic circuits.

When the team needed to support three open-weight models that weren't originally in the production plan, Codex — working with GPT-Astra — brought them to high-performance operation within two months. In certain Attention and MoE modules of GPT-OSS, AI-generated implementations ran 1.5x to 1.8x faster than the versions written by human experts — it's worth emphasizing that this result applies only to the selected modules, not to the entire model at the same margin.

This reveals a new development loop: models help design chips, chips accelerate models, and new models continue to optimize software running on those chips. It also explains why Jalapeño's significance extends beyond a single chip — OpenAI is connecting models, products, service software, chips, memory, networking, and data centers into a complete, integrated technology stack. Real user workloads tell the team what to optimize in the next generation of hardware; better hardware lowers cost and shortens response times; better experiences drive more usage and feedback; and every layer of change feeds back into the next design iteration.

MoE (Mixture of Experts) is a design paradigm widely adopted in today's large-scale models. Unlike dense models that activate all parameters on every inference pass, MoE divides a model into multiple "expert" sub-networks, with each token dynamically routed by a gating mechanism to only a small number of experts during the forward pass. This allows a model's total parameter count to far exceed the number of parameters actually activated at inference — DeepSeek R1 has 670B total parameters, but only a fraction are activated during any given inference. The inference challenge with MoE architectures lies in the unpredictable compute paths introduced by routing decisions, potentially severe load imbalances across different experts, and the need to distribute expert weights across multiple devices — all of which compound the complexity of memory access and cross-device communication. AI-assisted optimization of MoE module implementations is precisely the kind of problem where the search space is combinatorially explosive, making it an ideal domain for AI to find high-efficiency solutions that humans would struggle to enumerate manually.

Business Logic: Competing for How Much Intelligence Each Dollar Can Deliver

From a business perspective, OpenAI isn't competing for an isolated benchmark trophy — it's competing for control over how much useful intelligence each dollar can deliver. If a model requires fewer attempts and fewer output tokens for the same task, software reduces unnecessary context, and a chip completes inference at lower energy cost, total costs fall simultaneously at multiple levels.

OpenAI offered another illustrative example: on Artificial Analysis's coding agent index, GPT-5.6 running at maximum inference intensity achieved a new high score while using 54% fewer output tokens than another leading model. That number has nothing to do with chip efficiency, but it demonstrates that cost optimization can happen at many different layers.

The Jevons Paradox is unavoidable here: total usage doesn't necessarily fall when efficiency improves. Tasks that were previously too expensive to be worth attempting may become viable once costs drop — enterprises can have agents review more contracts, or provide customized analysis for more customers. So even if Jalapeño dramatically improves energy efficiency, it won't leave OpenAI suddenly flush with spare compute. The more likely outcome is that new demand rapidly consumes the saved capacity.

The Jevons Paradox takes its name from 19th-century British economist William Stanley Jevons. In 1865, he observed that after steam engine efficiency improved, Britain's total coal consumption rose rather than fell — because greater efficiency made coal-powered production more economically attractive, triggering an explosion in demand. This phenomenon has recurred repeatedly across energy economics and technology: semiconductor efficiency doubles every few years, yet global chip power consumption keeps rising; data compression technology improves continuously, yet internet traffic surges in parallel. Applied to AI inference, every reduction in per-inference cost doesn't reduce total compute demand — it unlocks application scenarios that were previously shelved as too expensive, ultimately driving overall usage to converge at a higher level. This is important context for understanding why AI infrastructure investment continues to accelerate rather than plateauing as efficiency improves.

Has NVIDIA's Moat Disappeared? It's Too Early to Say

Declaring NVIDIA's moat gone is clearly premature.

First, what has been disclosed represents early engineering results. OpenAI is still completing production certification, refining software, and preparing for scale-up, with plans to begin deploying Jalapeño into its own compute infrastructure before the end of 2026. Between a clean benchmark curve and the reliability, yield, and supply capacity of a long-term production environment, there's still an entire road to travel.

Second, Semi-Analysis verified the InferenceX testing process at OpenAI's lab, but did not run the full suite and did not test AgentX — the benchmark more oriented toward long-context, multi-turn interactions. How much of Jalapeño's advantage carries over to genuinely complex agentic tasks remains to be seen with more data.

Third, OpenAI itself hasn't described its in-house silicon strategy as a wholesale replacement for NVIDIA. The company's compute portfolio still includes partners such as Microsoft, NVIDIA, Amazon, AMD, Broadcom, Cerebras, CoreWeave, and Oracle. Frontier-scale training, large-scale batch inference, and continuously running agents won't be handled by a single type of chip.

So the competition Jalapeño introduces isn't "OpenAI suddenly stops buying NVIDIA" — it's that OpenAI is no longer only a buyer in the hardware market. It now has a first-party path that can develop in parallel with its partners, and it can write real model workloads directly into chip architecture. Even if this path only handles a portion of inference demand, it could meaningfully shift OpenAI's cost structure and negotiating position.

A second-generation Jalapeño is already in deep development, and a third generation is beginning to take shape. What will ultimately determine the landscape isn't a single benchmark run, but whether the "model-software-chip-energy" feedback loop can iterate faster than traditional hardware cycles while reliably serving real users. If the answer is yes, the scarcest resources in the future won't just be advanced process nodes, memory, and power — they'll also include who understands their own models best and can translate that understanding all the way down to the chip and the data center.