OpenAI's In-House Chip Jalapeño: 1.8–4x Better Performance Per Watt Than Existing Chips

What Is Jalapeño: A Chip Built for Inference

OpenAI has publicly unveiled its in-house inference chip, Jalapeño, for the first time. In an interview, chip lead Richard Ho revealed that the chip outperforms NVIDIA's current flagship processors on key tests of efficiency and response speed. The core goal is straightforward: deliver more intelligence to users at a lower cost.

Jalapeño is an inference chip, not a training chip. That positioning alone signals OpenAI's strategic thinking. Richard Ho explained that OpenAI's compute demand is in an explosive growth phase, and the primary driver is on the inference side — as weekly active users climb rapidly, the compute pressure on the model serving layer far exceeds that of training.

On the training side, OpenAI continues to rely on its strong partnership with NVIDIA, which it considers a provider of excellent training chips. But on the inference front — the fastest-growing battleground — OpenAI has decided to step in and close the compute gap itself.

Notably, Jalapeño won't replace existing suppliers. Instead, it will serve as part of OpenAI's chip "fleet," working alongside chips from NVIDIA, Cerebras, and others to handle different workloads.

Training and inference represent two fundamentally different phases of AI chip demand. Training requires repeated iterative computation over massive datasets, demanding extremely high floating-point throughput. A single training run can last weeks or months, with relatively stable and predictable chip utilization. Inference, on the other hand, is the process of deploying a trained model to serve real-time user requests — every conversation, every code completion triggers an inference call. As products like ChatGPT scale their user base, inference request volumes can surge several times over in a very short window, placing entirely different demands on concurrent processing capacity and response latency. Because inference workloads are bursty, highly real-time, and directly tied to user-facing pricing, leading AI companies widely view custom inference chips as a core lever for controlling marginal costs.

High Throughput and Low Latency Together: An Industry First?

The aspect Richard Ho is most proud of is Jalapeño's ability to deliver both high throughput and low latency on a single chip — something he says has not been done before in the industry.

Both metrics carry real value:

• High throughput means serving massive volumes of tokens to large numbers of customers at lower cost;
• Low latency enables extremely fast response speeds, making AI agents more reactive and interactions like coding smoother.

Historically, these two goals have required hardware design trade-offs. Jalapeño claims to deliver both, which is a genuine boost to OpenAI's product experience.

The tension between throughput and latency stems from the logic of hardware resource allocation. High throughput is typically achieved through batching — grouping multiple users' requests into a single batch for simultaneous computation, dramatically increasing total tokens processed per unit time. But individual requests must wait for the batch to fill before execution begins, causing Time to First Token (TTFT) to rise. Low latency demands that requests be processed the moment they arrive, in small or no batches, which reduces hardware utilization and raises the marginal cost per token. The traditional approach is to deploy different chip models for different business scenarios, or to switch between modes via software on the same chip. Jalapeño claims to have resolved this tension at the architectural level. If true, it could significantly simplify infrastructure planning for OpenAI, which needs to support both high-concurrency API services and low-latency interactive products simultaneously.

1.8–4x Performance Per Watt: Token Prices Could Drop

The most closely watched figure from the interview is Jalapeño's performance per watt, which Richard Ho says is 1.8 to 4 times higher than existing chips. He specifically noted that performance per watt is a good proxy for the cost of a single token — the higher the performance, the lower the per-token cost.

To ensure fair comparison, OpenAI used an open-source benchmarking tool called InferenceX. Richard Ho emphasized that choosing an open-source solution was deliberate, meant to produce a neutral and impartial comparison.

He was candid: "This is a very good chip, and it should bring costs down significantly." Once Jalapeño enters production, token prices for customers are expected to fall accordingly. Reducing inference costs and shrinking infrastructure overhead has long been a core objective that OpenAI has been pushing toward.

Multi-Chip Strategy: How Are Workloads Allocated?

With options including NVIDIA, Cerebras, and its own Jalapeño, how does OpenAI decide which workload runs on which chip?

Richard Ho describes this as the "optionality" the company is creating. OpenAI will make deliberate decisions about which product, which SKU, and which chip is best suited for which model — offering users a range of model options that each carry different cost and latency profiles.

Behind this fine-grained compute scheduling is a consistent OpenAI strategy: use the most appropriate hardware to deliver more intelligence to users at the lowest possible inference and infrastructure cost.

Observations and Takeaways

OpenAI developing its own inference chip is another signal — following Google's TPU and Amazon's Trainium/Inferentia — that leading AI companies are moving toward custom silicon. This reflects a growing industry consensus: as model capabilities begin to converge, whoever can drive inference costs lower will hold the advantage in large-scale service delivery.

Some caution is warranted: the 1.8–4x performance-per-watt improvement comes from OpenAI's own benchmarks. While open-source tooling was used, real-world production performance will still need third-party validation once the chip enters mass production. For everyday users, the most direct thing to look forward to is likely further reductions in API call pricing.

Google's TPU (Tensor Processing Unit) has been in internal use since 2016 and is now in its sixth generation — the most mature precedent for custom AI silicon in the industry. Amazon's Trainium focuses on training, while Inferentia focuses on inference; both are available externally through AWS. These examples show that taking a custom chip from tape-out to large-scale production typically takes years, with challenges spanning yield rates, supply chains, and software ecosystem compatibility. OpenAI has yet to disclose Jalapeño's foundry partner, process node, or production timeline — information that will directly determine whether its cost advantages can translate into price changes users actually feel. Custom chips also require companion compilers, scheduling frameworks, and operator libraries; the maturity of the software stack often matters more than the hardware itself in determining real-world performance.