Microsoft Maia 300 Launches: 96-Accelerator Racks and a Custom Inference Silicon Roadmap

Microsoft officially launched its third-generation custom AI inference accelerator, Azure Maia 300, on October 12. Unlike previous chip announcements, this time Microsoft put rack-level parameters and power envelopes at the very front of the announcement — 96 accelerators, a 167-kilowatt rack, 27.7TB of HBM capacity, 1.7PB/s of aggregate memory bandwidth. These numbers are themselves a message: Microsoft is talking about data center capacity, not just a chip.

The rack is the unit of product

Scott Guthrie, Executive Vice President of Cloud + AI, defined Maia 300 as "a complete intelligent system designed for today's emerging AI workloads," covering long-context inference, latency-sensitive conversational experiences, and sustained multi-step agent tasks. More important is the deployment cadence: Microsoft plans to scale the Maia family to multi-gigawatt deployments starting in 2027 and make it the primary inference system for Microsoft products and services.

A chip announcement that opens with power and rack counts is essentially making a capacity claim. Maia 200 is already carrying production workloads including Microsoft Copilot, with GitHub Copilot and Microsoft Foundry to follow by year-end. OpenAI models, Microsoft's in-house MAI models, and mainstream open-source model families all run on this silicon, spanning the United States, Europe, and Australia.

Multi-chiplet packaging and HBM4 memory

Maia 300 uses a multi-chiplet design: the compute chiplet is built on TSMC's N2P process, while memory controllers and I/O functions sit on separate dies, letting each function use the technology best suited to its role.

Memory configuration is the decisive factor for inference compute. Eight HBM4 stacks provide 288GB of capacity and 18TB/s of bandwidth. Microsoft reports this as 2.5x the memory bandwidth and 33% more capacity than Maia 200. The package has also shifted from a unified memory organization to a locality-aware design across chiplets, where data placement follows the compute unit that consumes it.

SiliconANGLE's read is worth noting: the 288GB capacity matches NVIDIA's Rubin accelerator, but memory bandwidth is about 20% lower. The two vendors converge on capacity and diverge on per-accelerator bandwidth — which is precisely the design space buyers actually face.

Chiplet design refers to breaking a SoC into multiple functionally independent dies, each using the most suitable process node, then interconnecting them into a single logical whole through advanced packaging. In traditional monolithic designs, compute, memory controllers, and I/O must share the same process, yet I/O circuits benefit far less from advanced nodes than logic does — forcing uniformity drives up cost. Maia 300 hands the compute chiplet to TSMC's N2P and puts memory controllers and I/O on mature-node dies, a textbook application of this approach. The tradeoff is latency and power overhead from cross-die communication, which makes interconnect density and locality design within the package critical — hence the phrase "data placement follows the compute unit that consumes it."

HBM4 (fourth-generation high-bandwidth memory) is JEDEC-standardized stacked DRAM placed adjacent to accelerator dies via a silicon interposer, trading an extremely wide bus for bandwidth far beyond GDDR. Compared with HBM3E, HBM4 doubles the interface width to 2048 bits and supports higher per-stack capacity and data rates. For inference, HBM matters not just for bandwidth but for capacity — model weights, KV cache, and activations must all reside in memory, and capacity determines how long a context and how much concurrency you can serve. Maia 300 achieves 288GB and 18TB/s with eight stacks, a configuration that directly sets the upper bound on the model sizes it can serve.

Fitting 96 accelerators into a 167-kilowatt rack

Maia 300 integrates 96 accelerators within a 167-kilowatt envelope, distributed across 24 four-accelerator trays, each containing host processors, memory, storage, and cooling. Folding the host into the accelerator tray is the key to doubling accelerator density, because standalone host trays disappear from the floor plan.

Rack HBM bandwidth grows 5.1x, while rack power grows 2.6x — which works out to a 1.95x improvement in HBM bandwidth per watt, close to the roughly 2x figure Microsoft publishes per accelerator. Teams thinking in tokens per watt will find this number more useful than any peak metric, because the power envelope is what data centers bill for and what facilities must cool.

Cooling is the other half of the power envelope. Liquid cooling handles more than 80% of the rack's thermal load, coming from accelerators, voltage regulators, and switch ASICs, and the design can connect to facility water or adapt to air-cooled rooms via a heat exchanger unit. These two options directly determine which buildings can host this capacity.

Power density and PUE provide the context for understanding the 167-kilowatt figure. Traditional air-cooled racks have long been capped at 10 to 20 kilowatts, and even high-density air-cooled solutions rarely exceed 40 kilowatts. 167 kilowatts represents nearly an order-of-magnitude jump in per-rack power, and air cooling physically cannot remove heat concentrated at that level — liquid cooling therefore shifts from optional to mandatory. "Facility water" refers to directly tapping the data center's chilled water loop, while a "heat exchanger unit" adds a closed loop between the rack and the facility, allowing older air-cooled rooms to host the system.

This shift also changes data center planning logic. Where capacity was once scaled by rack count and floor area, it is now scaled by megawatt power capacity and cooling capability — and the weakest of power, distribution, and cooling determines the compute that can actually be deployed. PUE (Power Usage Effectiveness) measures the ratio of a data center's total energy consumption to IT equipment consumption, and liquid cooling typically reduces PUE significantly because the cooling system itself draws less power. For buyers, tokens per watt is more informative than peak compute precisely because electricity and heat dissipation are the bulk of long-term operating cost.

A 12,288-accelerator superpod over Ethernet

The scale-up network is where Maia 300 diverges most from conventional GPU clusters. Microsoft's AI network controller runs a custom AI transport over Ethernet, with a two-tier topology: in the intra-rack semi-switched fabric, each accelerator distributes connectivity to direct partners within the tray and to switched paths reaching the other 94, all electrically connected; the second tier, optical switching, connects up to 128 racks into a single superpod.

The engineering choice worth calling out is that bandwidth allocation between intra-rack and cross-rack traffic remains open after system design is complete. It becomes a parameter set by job packing, and accelerators can connect to remote peers in the superpod at 750GB/s. Tensor parallelism can stay within a single rack, while expert parallelism and pipeline parallelism span racks, and the scheduler can absorb a failed tray by drawing on hot spare capacity in adjacent racks.

This is scheduling capability expressed as network design. Capacity once trapped within a single rack boundary can now be used by jobs packed elsewhere, raising the effective utilization of the same silicon.

Scale-up vs. scale-out are the two tiers of AI cluster networking, and understanding them is essential to reading the superpod design. Scale-up connects multiple accelerators into one high-bandwidth, low-latency "big chip," used for parallelism strategies like tensor parallelism that communicate frequently at every layer; scale-out connects large numbers of nodes into a cluster, used for data parallelism and pipeline parallelism, with comparatively relaxed bandwidth and latency requirements. In conventional GPU designs, the scale-up domain is typically confined to NVLink or similar high-speed interconnects within a single rack, with cross-rack traffic forced onto slower scale-out networks — creating a hard capacity boundary.

What makes Maia 300 distinctive is extending the scale-up domain to up to 128 racks — 12,288 accelerators — over two tiers of Ethernet. Electrical connections within the rack ensure low latency, optical switching between racks maintains bandwidth, and the whole system is built on Ethernet rather than proprietary interconnects. The expert parallelism mentioned (distributing different experts of a MoE model across devices) and pipeline parallelism (splitting the model by layer) can therefore span racks, while the most communication-intensive tensor parallelism stays within a single rack — a tradeoff between communication overhead and scale.

Whose baseline is the 40% tokens-per-megawatt improvement measured against?

Microsoft's efficiency headline is this: Maia 300 should deliver 40% more tokens per megawatt than leading GPU systems, and reach 7x the rack-level token throughput of Maia 200 in a 96-accelerator configuration. Maia 200, meanwhile, is recorded as delivering 1.7x more tokens per watt than leading GPU infrastructure for built-in AI in Teams, Copilot, and Cowork.

The two halves of this statement carry different weight. The 7x rack-level figure and the 2.5x per-watt performance are comparisons within Microsoft's own product line. The 40% figure crosses vendor boundaries, and Microsoft's post does not name the comparison system, model, sequence length, or batching configuration. The Verge reported the same figure the same day, with the same information missing.

Cross-vendor numbers should be treated as the starting point for questions. A useful question is: which baseline, which model, which context length, and were both systems' serving stacks tuned? The answer determines whether that figure transfers to your workload.

Tokens per watt is displacing FLOPS as the core efficiency metric of the inference era, because the bottleneck for large-model inference has shifted from compute to memory bandwidth and power. A single decode step reads all model weights, so actual throughput is often limited by memory bandwidth rather than compute units — and power directly corresponds to the total deployable compute. Dividing throughput by power yields what a data center can truly deliver under a given power capacity.

That is precisely why cross-vendor efficiency claims warrant extra scrutiny. Model size, context length, batch size, quantization precision, parallelism strategy, and software stack optimization in the measurement conditions can each swing the number by multiples. The same hardware can differ by more than 2x in tokens per watt under different serving stacks, so an undisclosed-baseline 40% advantage could reflect architectural differences or simply two systems tuned to different degrees. For buyers, the practical approach is to benchmark on both platforms with your own model and context length rather than trusting vendor cross-comparisons.

Capacity planning around rack power

For inference buyers, this announcement works as a planning template even if the chip never enters your facility. Capacity is now priced in racks and megawatts, and the calculation runs: required tokens per second, divided by tokens per watt at the context length your workload uses, gives you a power figure — and then you hit cooling limits and facility limits.

Two of Microsoft's own numbers change how this calculation behaves. Doubling accelerators per rack means fewer racks carry the same load, and per-watt bandwidth improvements push more of the bottleneck onto cooling and power. Liquid cooling deserves its own line item in the plan: a rack that removes more than 80% of its thermal load via liquid cooling needs either facility water or a heat exchanger.

Microsoft says it will share more details at Ignite 2026 in November. Until then, the published rack tables and two architecture diagrams are the parts of this announcement technical readers can verify.

One-line Summary

Microsoft launches its third-generation inference accelerator Maia 300, declaring in units of 167-kilowatt, 96-accelerator racks that the AI data center capacity race has entered the megawatt era.

Paragraph Summary

Microsoft has officially launched its third-generation custom AI inference accelerator, Maia 300, unusually opening with rack-level parameters: 96 accelerators, a 167-kilowatt power envelope, 27.7TB of HBM, and 1.7PB/s of aggregate bandwidth. The chip uses a multi-chiplet design with a compute die on TSMC N2P and eight HBM4 stacks delivering 288GB of capacity and 18TB/s of bandwidth. Its scale-up network is a two-tier Ethernet fabric connecting up to 128 racks into a 12,288-accelerator superpod, with electrical connections within racks and optical links between them, and bandwidth allocation set dynamically by job packing. Microsoft claims 40% more tokens per megawatt than leading GPU systems but does not disclose the baseline. Maia 200 already carries Copilot workloads in production, and Microsoft plans to scale to multi-gigawatt deployments starting in 2027.