Hot Chips Conference: The AI Chip Arms Race Escalates Across the Board

A New Battleground in the AI Compute War

Every year, the Hot Chips conference serves as a bellwether for the semiconductor industry. Held annually at Stanford University since 1989, Hot Chips is one of the most influential technical symposiums in the semiconductor world. Unlike CES or Computex, which focus more on commercial launches, Hot Chips emphasizes deep technical discussions at the chip architecture level, attracting chip design engineers, architects, and academic researchers. Historically, many groundbreaking processor architectures—from Intel's Pentium series to AMD's Zen architecture, from ARM's big.LITTLE to Apple's M-series chips—have had their technical details publicly disclosed here for the first time. The conference's core value lies in giving the industry early insight into computing architecture trends for the next 2–3 years, making it a critical window for observing the direction of the technology landscape.

This year's edition of the aptly named "Hot Chips" conference drew its heat from the heavyweight players involved. From OpenAI's debut of its custom chip "Jalapeño," to Cerebras's wafer-scale behemoth CS-5, Groq's third-generation inference-focused LPX, and Apple's M6—this year's conference brought together virtually all the core forces in today's AI hardware competition.

This is more than just a technology showcase; it's a microcosm of the reshaping of the AI industry landscape. When compute power becomes the critical bottleneck determining the upper limits of large model capabilities, whoever commands more efficient and cost-effective chips gains the upper hand in the next round of competition.

Multiple AI chips showcased at the Hot Chips conference

OpenAI Jalapeño: A Software Company's Hardware Ambitions

The most eye-catching reveal was OpenAI's first public disclosure of its in-house chip project, "Jalapeño." As a company renowned for its algorithms and models, OpenAI's foray into chip design sends a clear signal: deep dependence on NVIDIA GPUs has become a strategic vulnerability.

NVIDIA GPUs command over 90% of the AI training market, an advantage rooted in more than a decade of building its CUDA software ecosystem. CUDA is NVIDIA's parallel computing platform, offering a complete development toolchain, optimized libraries, and a massive developer community. Nearly all mainstream deep learning frameworks (TensorFlow, PyTorch, etc.) are deeply optimized for CUDA, creating a formidable ecosystem moat—even if competitors release superior hardware, the cost of developer migration remains prohibitively high. NVIDIA's data center GPUs like the H100 and A100 can cost tens of thousands of dollars per unit, and during supply shortages, they become nearly impossible to obtain. This market dominance yields handsome profits but also subjects AI companies that depend on these products to enormous cost pressures and supply chain risks.

Why Is OpenAI Building Its Own Chips?

Training and running ultra-large-scale models like the GPT series consumes astronomical amounts of compute. High-end AI accelerator cards have been in chronic short supply, keeping prices stubbornly high. By developing custom chips, OpenAI stands to gain advantages on three fronts:

Cost control: Chips tailored to its own model architecture can dramatically reduce per-unit inference and training costs
Supply chain independence: Reducing reliance on a single vendor avoids being held hostage
Hardware-software co-optimization: Deep coupling between chips and models achieves higher energy efficiency than general-purpose GPUs

Of course, this path is far from easy. Chip design involves lengthy development cycles and massive investment—even well-resourced tech giants have stumbled here. Whether OpenAI can successfully translate its software advantages into the hardware domain remains the biggest open question.

Cerebras CS-5: Pushing Wafer-Scale Computing to the Extreme

If mainstream manufacturers are pursuing chip density and interconnect efficiency, Cerebras is taking a radically different "brute force elegance" approach—the Wafer-Scale Engine.

Breaking the Physical Boundaries of Traditional Chips

Cerebras's core idea: instead of dicing a wafer into many small chips and then interconnecting them, why not turn the entire wafer into one giant chip? In traditional chip manufacturing, a 12-inch silicon wafer is cut into hundreds of individual chips, because defects are inevitable in semiconductor fabrication—the larger the wafer, the lower the yield. Cerebras's Wafer-Scale Engine turns an entire wafer into a single chip with an area exceeding 46,000 square millimeters (a typical GPU is around 800 square millimeters). This creates three major engineering challenges: first, yield—Cerebras addresses this through redundant design and fault-masking techniques; second, thermal management—a chip this large can dissipate tens of thousands of watts, requiring specially designed liquid cooling systems; and third, packaging and power delivery, requiring thousands of power contact points and high-bandwidth I/O interfaces.

This design delivers three key advantages:

Ultra-high-bandwidth on-chip communication: Data doesn't need to traverse slow external buses between multiple chips, dramatically reducing latency. Despite the challenges, wafer-scale design completely eliminates the inter-chip communication bottleneck in multi-chip systems. When processing ultra-large models requiring massive parameter exchanges, this can theoretically yield communication efficiency improvements of tens of times
Massive parallel cores: A single "chip" integrates far more compute units than conventional products
Simplified system architecture: Reduces the complexity of multi-chip coordination

The fifth-generation CS-5 further improves performance and energy efficiency on this foundation. For scenarios involving giant models, this integrated compute solution holds unique appeal, particularly in reducing communication overhead from model sharding.

Groq 3 LPX: Purpose-Built for AI Inference

Unlike the arms race in training chips, Groq has placed its bet on "inference"—the stage closest to real-world applications. In deep learning, training and inference are two fundamentally different computational stages. Training processes massive datasets, continuously adjusting billions or even hundreds of billions of parameters through backpropagation algorithms, with extremely high computational intensity—often requiring thousands of GPUs working in concert for weeks or even months. Inference uses already-trained models to make predictions on new data; while each individual computation is smaller, latency sensitivity is critical—every conversation with ChatGPT, every AI-powered search, involves an inference operation behind the scenes. From a cost structure perspective, a large model may only be trained once but called upon billions of times for inference, so the cumulative compute consumption and cost of inference often far exceed that of training. This difference has given rise to different chip architectures optimized specifically for training or inference.

The third-generation LPX (Language Processing Unit) continues Groq's technical trajectory in the low-latency inference space.

The Enormous Potential of the Inference Market

As large models move from labs into real-world deployment across industries, inference costs are gradually surpassing training costs as the primary ongoing operational expense. Groq's deterministic low-latency architecture is particularly well-suited for real-time application scenarios sensitive to response speed, such as conversational AI, code assistants, and search augmentation.

Groq's LPU architecture abandons some of the complex scheduling mechanisms found in traditional GPUs, using software to pre-orchestrate data flow, thereby achieving extremely high throughput and predictable latency. In the AI chip space, energy efficiency (how many operations can be completed per watt of power) is becoming a more critical metric than raw performance. Data center operators have found that electricity and cooling costs for AI servers already account for over 40% of total cost of ownership (TCO), while mobile devices are further constrained by battery capacity. This is driving chip design to shift from "compute faster" to "compute more efficiently." Optimization techniques include: adopting more advanced process nodes (such as 3nm, 2nm) to reduce transistor power consumption; using dedicated instruction sets instead of general-purpose compute logic; leveraging sparsity techniques to skip zero-value operations; and dynamic voltage and frequency scaling (DVFS) to adjust power based on workload. For example, Groq claims its LPU achieves over 10x the energy efficiency of GPUs with equivalent compute power for inference tasks.

This single-scenario focus has enabled impressive tokens-per-second generation speeds in specific benchmarks. Against the backdrop of exponentially growing AI compute demand, improvements in energy efficiency directly determine the economic viability and environmental sustainability of AI services.

Apple M6: The Quiet Revolution of Consumer-Side AI

Among the lineup of data center-class chips, Apple's M6 appears understated yet profoundly significant. It represents an alternative technical path—bringing powerful AI capabilities down to edge devices.

The Strategic Value of On-Device AI

Apple has consistently emphasized on-device processing, and the M6 is expected to further enhance its Neural Processing Unit (NPU) performance, enabling more AI tasks to be completed locally. An NPU (Neural Processing Unit) is a hardware accelerator specifically designed for neural network inference, typically integrated into the system-on-chip (SoC) of mobile devices and PCs. Unlike general-purpose CPUs or GPUs, NPUs are specifically optimized for the massive matrix multiplications and convolution operations common in neural networks, employing low-precision computation (INT8, INT4, or even lower) and sparsity techniques to dramatically reduce power consumption while maintaining accuracy. Apple has integrated a neural engine since the A11 chip, and by the M-series it has gone through multiple generations of evolution, with performance improving from hundreds of billions of operations per second to trillions. This enables iPhones and Macs to run AI functions like image recognition, voice processing, and real-time translation locally. With the rise of on-device large models, NPU compute demands are experiencing explosive growth.

The value of this strategy is clear:

Privacy protection: Sensitive data never needs to leave the device or be uploaded to the cloud
Low-latency response: Local processing eliminates network round-trip delays, as well as the network latency and per-call billing costs of cloud inference
Offline availability: No dependency on a persistent network connection

While the industry broadly focuses on the cloud-based large model arms race, Apple has chosen to democratize AI capabilities across billions of endpoint devices—a differentiated competitive approach that is far closer to everyday users.

Conclusion: A Diversified AI Chip Competitive Landscape Is Taking Shape

This year's Hot Chips conference has clearly outlined the diverging trends in the AI chip industry: OpenAI is pursuing vertical integration for self-reliance, Cerebras is challenging physical limits with its wafer-scale approach, Groq is drilling deep into inference efficiency optimization, and Apple is focusing on making on-device AI ubiquitous.

This diversification precisely reflects the increasingly segmented demands of AI applications. No single chip can dominate every scenario. The future compute ecosystem will likely be a layered landscape where training, inference, cloud, and edge each serve their specialized roles. For the industry as a whole, fiercer chip competition means faster technological iteration and lower compute costs—ultimately benefiting all AI application developers and users.