OpenAI Jalapeño Inference Chip First Look: The Battle Is Won or Lost Per Token Per Watt

OpenAI's In-House Inference Chip Debuts — and Rewrites the Rules

OpenAI has just released the first benchmark results for its in-house inference chip, Jalapeño. The chip's package thermal design power is rated at 700W, but OpenAI specifically notes that sustained power draw during testing never exceeded 550W. That detail matters — when reading the charts, keep in mind that the Mixed TPS/kW metric is normalized to 700W, while 550W reflects the actual operating state during the benchmarks.

The core of this comparison is no longer the familiar multiply-accumulate throughput or peak FLOPS. Instead, it shifts to a far more pragmatic frame: how many tokens can a system process at a fixed power budget, and how low can latency go? One important caveat upfront: these numbers don't include the full rack's CPU, networking, or cooling overhead — and certainly not a data center's final PUE.

When we write code or chat with an AI, we never feel raw TFLOPs. What we actually feel is how fast the first word appears and whether generation stutters. Engineeringly, that maps to TTFT (Time to First Token) and TBT (Time Between Tokens). A GPU's theoretical compute is the ceiling, but in practice, memory bandwidth, KV Cache management, queuing strategies, and cross-chip communication all drag performance down. By publishing results that anchor the core metric to "token delivery rate under a given power and latency budget," Jalapeño's first public numbers are clearly closer to the reality of online serving than raw FLOP counts ever were.

The Team and the Timeline: Clear Division of Labor, but Schedule Claims Deserve Scrutiny

The collaboration has a clear structure: OpenAI defines the architecture, Broadcom handles chip implementation and networking, and Celestica manages board and system integration. As the model vendor running real production workloads, OpenAI knows better than anyone which operators are slow and how agents consume context — and it translates those insights into chip requirements.

OpenAI claims the journey from initial design to Tape Out took just 9 months. Semi Analysis, counting team ramp-up time, puts the figure closer to 16 months. Either way, "fastest development cycle" is OpenAI's own framing — worth noting, but not to be treated as an industry audit standard.

The credibility of these benchmarks deserves careful scrutiny. The tests used Semi Analysis's InferenceMAX framework, and a third party was indeed on-site running tests alongside engineers — but the data was still provided by OpenAI, the full test suite was not run, and no raw logs have been made public. To be precise, these are "vendor results verified on-site by a third party" — external independent replication is not yet possible. The test scenarios were also strictly limited to 8K input and 1K output, with Speculative Decoding disabled and no multi-turn long-context Agentic tasks included.

Tape Out is a critical milestone in chip development — the moment when the design team submits the finalized layout data to the foundry to begin the actual lithography and manufacturing process. Before this point, the team must complete RTL coding, logic synthesis, timing closure, physical implementation (place and route), and extensive simulation and verification. After Tape Out, there are still packaging, testing, and yield ramp stages before a chip can handle real production workloads — typically several more months. The 9-month or 16-month figure refers to the span from design start to Tape Out, not to full production readiness. Broadcom's role in this collaboration — including its experience with advanced 2nm-class processes and its deep relationship with TSMC — was a key external factor in compressing the development timeline.

Breaking Down the Three Benchmarks: Real Advantages, but the Boundaries Must Be Clear

All three benchmark groups measure Mixed TPS/kW — mixed throughput divided by power consumption. Note the word "Mixed": it combines both input and output token processing into a single number. This reflects overall system work per unit of power, but it does not directly translate to how quickly users see output. For pure output speed, you'd need to look at the output-only throughput figures that InferenceMAX reports separately.

Group 1: gpt-oss 120B vs. GB200. Jalapeño's peak throughput-per-watt is approximately 85,448 vs. 44,960 — about 1.9x. End-to-end latency drops from 1.80 seconds to 1.03 seconds, a ~43% reduction. Minimum TBT falls from 1.87ms to 0.69ms. Lower TBT means smoother, more fluid output.

Group 2: DeepSeek^Ad R1 670B vs. GB300 (MXFP4 precision). Throughput-per-watt is approximately 1.7x (19,641 vs. 11,781). End-to-end latency drops from 5.99 seconds to 1.65 seconds — a reduction of roughly 72.5%.

Group 3: Kimi K2 1T vs. GB300. Throughput-per-watt is approximately 1.5x (18,195 vs. 11,862). End-to-end latency drops from 5.31 seconds to 1.56 seconds — about 70.6% lower. Trillion-parameter models are extremely sensitive to cross-chip data movement, and the cost of shuffling weights, activations, and KV Cache across different memory tiers adds up quickly.

The conclusion holds within these bounds: under the defined models, request lengths, and operating environments, Jalapeño does achieve higher mixed throughput and lower latency. But directly translating "1.9x" into data center power savings is a stretch — different vendors test with different concurrency levels, sequence lengths, and network topologies. OpenAI attributes the gains to hardware-software co-design, but the published data doesn't decompose how much queuing, memory access, or communication each contribute. "The results are the results" — engineering attribution still requires a question mark.

MXFP4 is a 4-bit precision variant within the MX (Microscaling) floating-point format standard promoted by Microsoft, Intel, and others. Compared to FP16 or BF16, each MXFP4 value occupies only 4 bits, theoretically compressing memory footprint and bandwidth requirements to roughly one-quarter, while allowing hardware to pack more compute units into the same die area. The key design insight of the MX format family is the use of a shared scaling factor — a group of values shares a common exponent block — which preserves sufficient dynamic range at very low bit widths and reduces quantization error. Running DeepSeek R1 670B at MXFP4 precision in Group 2 means the model weights were quantized to that format, which does affect model quality. Cross-precision performance comparisons are meaningful in engineering contexts, but when assessing "which chip is stronger," you need to verify that both sides are using equivalent precision — otherwise throughput differences may partly reflect quantization gains rather than pure hardware advantage.

Design Philosophy: Treating Prefill and Decode as Two Separate Problems

Understanding Jalapeño's design logic requires splitting inference into two phases.

Prefill phase: You hand the model a long prompt, and it needs to rapidly digest it and build up internal state. This step is purely compute-intensive — matrix operations are massive and easy to saturate compute units with. If you approach this with the mindset used for training large models, it's easy to focus only on this half.

Decode phase: Once the request enters the continuous generation loop, the model starts producing tokens one by one. Per-step compute drops off a cliff, but each step requires repeatedly reading weights and state from memory. The bottleneck shifts from compute to memory bandwidth and access latency. It's like an assembly line where the first stage involves bulk preparation, but the second stage requires fetching a single part from a distant warehouse for every single bolt you tighten — the moment that channel gets congested, production halts and TBT spikes. OpenAI itself emphasizes that decode is severely memory-bandwidth-bound.

The key to breaking this constraint is KV Cache. To generate the next token, the model needs to query its accumulated memory of previous tokens (keys and values) — and the longer the context, the larger that memory grows. If it has to be shuffled between multiple chips, all the bandwidth gets consumed moving data around while compute units sit idle. Jalapeño's core solution is to keep KV Cache and related state as local as possible. Of course, this is the ideal scenario — whether tail latencies (P95, P99) remain stable under production conditions is something that can only be proven in live environments.

KV Cache size grows linearly — or faster — with context length. Take a typical 7B-parameter model: processing a 32K-token request can require several gigabytes of KV Cache, while the model weights themselves are only about 14GB. As concurrent requests pile up, KV Cache quickly fills GPU memory, forcing the system to offload some state to slower storage tiers (such as CPU memory or NVMe). The latency penalty on every read-back causes noticeable TBT jitter. This is why chip vendors designing inference accelerators treat on-chip SRAM capacity, HBM bandwidth, and cross-chip interconnect speed as primary constraints — not because compute is insufficient, but because the cost of "moving memory" has become increasingly difficult to absorb. Jalapeño's design choice to anchor KV Cache locally is fundamentally a system-level tradeoff: trading more local storage capacity for lower data-movement latency — space for time.

Networking Baked Into the Architecture: Full-Stack Control Is the Endgame

In production, even the strongest silicon is only as good as the system around it. A request comes in, gets routed, processed, cache-split, and finally network-distributed — any single link in that chain can create the perception of lag. Jalapeño integrates networking directly into the chip architecture, keeping requests within a single interconnect domain as much as possible. Anyone who builds infrastructure knows that idle compute units are almost always waiting for data. Today the bottleneck is rarely raw compute — it's cache and data scheduling that hasn't been resolved.

OpenAI's push into chip design is fundamentally a play for full-stack control. It builds the models, writes the serving stack, and now reaches down into memory and networking: if a particular kernel class runs slowly in production, the next-generation hardware can be changed directly; if Agent tasks are bottlenecked by waiting, scheduling policy can follow immediately. Its ultimate metric is no longer raw FLOPs, but "useful intelligence per hundred million dollars" — burning fewer watts, running more concurrent requests on the same power budget. That's the bill that actually shows up on the product side.

The Competitive Landscape — and a Few Sober Reminders

NVIDIA's position remains solid. The GB200 NVL72 liquid-cooled rack with its 130TB/s communication bandwidth represents an extraordinarily high system engineering moat. Jalapeño's initial home will clearly be OpenAI's own data centers, running real production workloads to break it in. It's also worth noting explicitly: the published appendix only compares against GB200 and GB300 — the soon-to-ship Vera Rubin is absent from the comparison. This data represents only a current snapshot, far from the basis for any definitive verdict.

Every major player is now carving out custom inference silicon: Google's TPUs pile on on-chip SRAM, Microsoft's Maia emphasizes large memory capacity and Ethernet interconnects, and AWS Trainium chases total machine interconnect throughput. Testing environments are too different to draw direct cross-vendor comparisons, but the trend is unmistakable — cache, bandwidth, interconnect, and Serving software co-design are now the absolute core battleground of modern inference systems.

As for the multi-gigawatt accelerator partnership between OpenAI and Broadcom, that's a long-term roadmap extending to the end of 2029. It does not mean first-generation Jalapeño capacity is already at scale. Chip tape-out, advanced packaging, liquid cooling, power delivery, land acquisition, and data center construction are all required — every one of them. The compute arms race is long past the stage where press releases matter. What counts as real capacity is equipment that gets physically racked in a data center and powers on on schedule.

One more claimed advantage worth examining: AI-assisted chip design. OpenAI says that for certain specific operator implementations, AI-generated code outpaced human experts by 1.5x to 1.8x in speed. But this applies only to specific steps. Chip verification remains brutally labor-intensive — running simulations, performing synthesis, checking timing. AI-generated candidate code still has to pass extremely rigorous regression testing at the end.

One final counterintuitive point: higher efficiency doesn't necessarily mean the data center uses less power. Jevons Paradox tells us that as inference costs fall and speeds rise, people become more willing to use it liberally — persistent Agents, multi-turn conversations, and background tasks all proliferate — and total power demand may actually surge. Looking at energy consumption requires accounting for both per-unit efficiency and the explosion in total demand together.

Jevons Paradox originates from the 19th-century British economist William Stanley Jevons, who observed that after steam engine efficiency improved, Britain's total coal consumption didn't fall — it climbed dramatically, because higher efficiency made steam power economically viable in far more contexts. This pattern has repeated itself throughout IT history: CPU efficiency doubles every few years, yet total data center power consumption keeps rising; mobile network speeds increase, and users consume proportionally more data. The logic in inference chips is identical: when the cost of each API call drops low enough, use cases previously constrained by expense — persistent personal Agents running in the background, automated code reviews triggered every few minutes, multi-model Agentic workflows chained together — all get unlocked and deployed at scale. Accurately assessing a new chip generation's real impact on data center energy consumption therefore requires modeling demand-side elasticity, not just looking at the power figure per unit of workload.

Closing Thoughts: The Evaluation Standard Has Already Changed

The signal from this first round of benchmarks is clear: the criteria for judging inference chips have shifted decisively toward tokens per watt, end-to-end latency, and system utilization. What should you watch for next? Watch for open raw logs. Watch for real Agentic task tests on long contexts. And watch for P99 tail latency numbers from production environments.

For everyday users, this may feel like little more than "faster text generation." But at the infrastructure level, an entire evaluation framework has turned over. What matters most to you — time to first token, or maximum system throughput?