Google TPU v8t and v8i Deep Dive: The New Era of Separated Training and Inference AI Chips

Google TPU v8 Series Launch: Training and Inference Chips Officially Split

At Google Cloud Next, Google officially unveiled two new AI chips: the TPU v8t and TPU v8i. Google Vice President Amin Vahdat and several industry experts discussed the release in depth, revealing the key differences between the two chips in terms of architecture design and application scenarios.

This marks the first time Google has explicitly differentiated between training and inference chips within the TPU product line, signaling that specialized AI chips have entered a new phase of refined design.

TPU v8t vs. TPU v8i: What Makes Each Chip Different?

TPU v8t: Purpose-Built for Large-Scale AI Training

The "t" in TPU v8t stands for "training," specifically targeting large-scale AI model training scenarios. This chip features deep optimizations in high-bandwidth memory, floating-point computation capabilities, and more, with the goal of accelerating the training of GPT-class large models.

High-Bandwidth Memory (HBM) is one of the core components of modern AI training chips. HBM vertically stacks multiple layers of DRAM chips and interconnects them using Through-Silicon Via (TSV) technology, delivering bandwidth far exceeding traditional DDR memory. Current mainstream HBM3E specifications can provide over 1TB/s bandwidth per stack. During large model training, billions or even trillions of parameters need to be rapidly shuttled between the chip and memory, making memory bandwidth a critical bottleneck for training speed. Additionally, BF16 (Brain Floating Point 16) is a data format Google designed specifically for deep learning—it retains FP32's exponent range while reducing mantissa precision, doubling computational throughput with virtually no loss in model training quality. Optimizations in such floating-point formats are also an important means of training acceleration in the TPU v8t.

From a naming strategy perspective, Google is shifting its TPU product line from "one chip for everything" to "dedicated chips for dedicated tasks." This transition reflects the reality that as AI model parameters leap from tens of billions to trillions, general-purpose chip designs can no longer adequately meet training scenarios' demands for extreme computational power.

TPU v8i: The Cost-Efficiency Choice for Inference Workloads

The "i" in TPU v8i stands for "inference." With the explosion of generative AI applications like ChatGPT, inference-side compute demand is rapidly escalating—by industry estimates, inference now accounts for over 60% of total AI compute consumption.

Google launched a dedicated inference chip precisely because inference workloads differ fundamentally from training in their computational characteristics: inference prioritizes low latency, high throughput, and energy efficiency over peak floating-point performance. Training is an iterative process requiring forward propagation, loss function computation, backpropagation of gradients, and parameter updates—involving massive matrix multiplications and gradient accumulations with extremely high demands for floating-point precision and memory capacity. Inference, by contrast, applies a trained model to new data, requiring only a single forward pass. Generative AI inference has its own unique characteristics—for large language models, generating each token requires recomputing the KV Cache in the attention mechanism, making inference a memory bandwidth-bound rather than compute-bound task. Therefore, inference chips typically invest more in low-precision compute units like INT8 or even INT4, and optimize on-chip cache architectures to reduce memory access latency. Only by making these targeted optimizations at the chip architecture level can the cost per inference truly be reduced.

Google's Strategic Layout for Self-Developed AI Chips

From 2016 to 2025: Nine Years of TPU Evolution

Google was the earliest among tech giants to bet on self-developed AI chips. When the first-generation TPU debuted in 2016, the industry still had considerable skepticism about its prospects. Nine years later, the TPU has iterated to the v8 series, powering Google's core businesses including Search, YouTube recommendations, and the Gemini large model.

Looking back at TPU's evolution is itself a microcosm of changing deep learning hardware requirements. The TPU v1, released in 2016, was a pure inference chip using a Systolic Array architecture for 8-bit integer matrix operations, primarily accelerating inference tasks within Google's data centers. TPU v2 (2017) first added training capabilities, introducing floating-point computation support and HBM memory. TPU v3 (2018) doubled compute power and introduced liquid cooling. TPU v4 (2022) was the first to deploy Optical Circuit Switch (OCS) technology, enabling thousands of chips to form ultra-large-scale Pods for distributed training. TPU v5e and v5p in 2023 were respectively differentiated for efficiency and performance, which can be seen as the prelude to the v8 series' official training-inference split.

Compared to competitors entirely dependent on NVIDIA GPUs, Google has accumulated significant cost and performance advantages through TPUs on its own cloud platform and internal AI R&D. This "produce and consume internally" model gives Google stronger autonomy over the AI compute supply chain.

Training-Inference Separation: An Industry Trend Taking Shape

Designing training and inference chips separately is not Google's invention. Amazon had already launched two product lines: Trainium (training) and Inferentia (inference). But Google's formal adoption of this strategy in the TPU v8 series further validates that training-inference separation is the prevailing direction in AI chip design.

The logic behind it is clear:

• Training chips require large memory bandwidth, strong floating-point compute, and high-speed chip-to-chip interconnects
• Inference chips prioritize latency performance, throughput efficiency, and performance per watt

It's worth understanding in depth that in large-scale distributed training, inter-chip interconnect bandwidth often determines overall training efficiency more than single-chip compute power. When model parameters exceed single-chip memory capacity, model parallelism or pipeline parallelism strategies must be employed to partition the model across multiple chips, requiring frequent exchange of activation values and gradient data between chips. Google TPU's ICI (Inter-Chip Interconnect) is a custom high-speed interconnect bus that allows TPU chips to communicate directly without going through the host CPU. In comparison, NVIDIA GPU clusters rely on NVLink and NVSwitch to achieve similar functionality, with the latest NVLink 5.0 providing 1.8TB/s bidirectional bandwidth per GPU. Interconnect topology design—such as 3D Torus, fully-connected Fat-Tree, etc.—directly impacts cluster communication efficiency and scalability.

The two types of requirements differ enormously, and using a single chip to address both inevitably means compromising on one end. With separate designs, Google Cloud customers can choose the most suitable hardware based on their actual workloads, achieving better price-performance ratios.

What Does TPU v8 Mean for the Cloud Computing Competitive Landscape?

In an era where AI compute is in short supply, cloud providers with self-developed chips hold a natural advantage. The TPU v8 series launch will directly strengthen Google Cloud's competitiveness in the AI workload market—especially when competing with AWS and Azure for large AI customers, where differentiated hardware solutions could become a decisive competitive advantage.

The current competitive landscape in the AI chip market can be summarized as:

• NVIDIA dominates through its CUDA ecosystem and GPU performance
• Google builds differentiated advantages within its own ecosystem through the TPU series
• Amazon serves AWS customers with Trainium and Inferentia
• Microsoft is also accelerating deployment of its self-developed AI chip Maia

NVIDIA's dominance in the AI chip market stems not only from hardware performance but even more from the deep moat of its CUDA software ecosystem. Since CUDA (Compute Unified Device Architecture) launched in 2007, it has accumulated over 4 million developers and tens of thousands of acceleration libraries. Mainstream deep learning frameworks like PyTorch and TensorFlow have the most mature support for CUDA, and acceleration libraries including cuDNN, cuBLAS, and TensorRT cover the complete workflow from training to inference. This means any chipmaker challenging NVIDIA must not only be competitive in hardware performance but also provide a sufficiently complete software stack to lower developers' migration costs. Google's counter-strategy is to build its own software ecosystem through the JAX framework and XLA compiler, enabling developers to run models on TPUs relatively seamlessly.

On the Microsoft front, its self-developed AI chip Maia 100—first announced in November 2023—uses TSMC's 5nm process, integrates over 105 billion transistors, and is designed specifically for large language model training and inference. Maia's launch signals that Microsoft is no longer fully dependent on NVIDIA GPUs to power Azure AI services, instead seeking to establish self-sufficient supply capabilities for critical compute resources. Simultaneously, Microsoft also released Cobalt 100, an Arm-based general-purpose CPU for routine computing tasks in Azure data centers. This layout forms a three-way standoff with Google's TPU strategy and Amazon's Graviton+Trainium+Inferentia combination, reflecting how hyperscale cloud providers are systematically reducing their dependence on third-party chip vendors.

This compute arms race is shifting from pure performance stacking toward refined optimization for specific workloads. For AI developers and enterprise users, more chip choices mean fiercer price competition and lower usage costs.

Conclusion: The Era of Specialized AI Chips Has Arrived

The launch of Google TPU v8t and v8i sends a clear signal—AI chip design philosophy is moving from "jack-of-all-trades" to "master of one." By building dedicated chips for training and inference respectively, Google has not only improved the overall efficiency of its own AI infrastructure but also provided a clear reference for the industry's chip design direction.

As Google continues to reveal more technical details at Cloud Next, the actual performance data, software ecosystem support, and pricing strategies of the TPU v8 series will remain focal points of sustained industry attention.

Key Takeaways

• Google unveiled two new chips at Cloud Next: TPU v8t (training-dedicated) and TPU v8i (inference-dedicated)
• The separation of training and inference chips reflects the industry trend toward refined optimization of AI workloads
• The self-developed chip strategy strengthens Google Cloud's differentiated competitive advantage in the AI compute market
• The TPU v8 series marks a new phase in AI chip evolution from general-purpose to specialized design