Why Do AI Data Centers Consume So Much Power? From GPU Heat to the Nuclear Energy Dilemma

Since 2019, more than a thousand massive warehouse-scale campuses have emerged around the world. Each complex spans nearly 100,000 square meters, consumes water like a small town, and draws electricity like a small city. These are the AI data centers that power the training and inference of large language models like Claude and ChatGPT. They resemble the traditional data centers that have served the internet since the 2000s, yet they introduce an entirely new set of challenges and possibilities.

Inside a Data Center: How Your Question Gets Answered

AI data centers are typically divided into several functional zones. The Network Operations Center (NOC) manages data traffic and power consumption; a power infrastructure comprising transformers, gas generators, and battery backups ensures uninterrupted electricity; and a connection hub known as the "Meet-Me Room" routes incoming data toward the heart of the facility — the server rooms.

When you ask a large language model a question, your prompt enters one of these server rooms. The mechanism by which the model generates a response is complex, but at its core it is performing massive amounts of mathematical computation. To handle these calculations, data centers rely on tens of thousands of server racks, each stacked with server trays carrying memory chips and GPUs.

Inside these high-performance microchips, thousands of tiny compute cores work in parallel, performing trillions of mathematical operations per second. While most queries require only a handful of GPUs to answer, data centers must already process tens of thousands of prompts per second — a rate that is growing rapidly. More critically, training a large language model requires hundreds of thousands of GPUs working in concert, which is precisely why these vast server arrays are indispensable.

GPUs (Graphics Processing Units) were originally designed for rendering video games, and their strength in "parallel computation" happens to align perfectly with the demands of deep learning. Unlike CPUs, which process complex logic sequentially, GPUs integrate thousands of relatively simple compute cores that can simultaneously apply the same operation to large numbers of values in a matrix. The inference and training of large language models are fundamentally dense matrix multiplications, making GPUs the central compute unit of AI data centers. Today's leading AI training clusters rely heavily on high-end GPUs like NVIDIA's H100 and H200, with individual units priced above $30,000, while a large training cluster may require tens or even hundreds of thousands of them. This extreme dependence on GPUs is what makes AI data centers vastly larger, more power-hungry, and more expensive than traditional internet data centers.

The Cooling Problem: GPU Heat That Can Melt Circuit Boards

This high-intensity computation produces a central challenge — heat. Even in temperature-controlled server rooms, the heat generated by GPUs is sufficient to melt server circuit boards, and the copper wires connecting these chips continuously dissipate energy during transmission, increasing both power consumption and heat output.

Engineers are tackling this challenge on multiple fronts. On the chip cooling side, they continue to use proven approaches — such as cooling towers that absorb and evaporate heat — while also introducing newer methods, like running cold-water pipes behind server boards or submerging entire trays in non-conductive cooling fluid.

At the same time, they are replacing inefficient copper wiring with fiber optics. Copper carries data via electrical signals, while fiber optics carry it via light. Many rack tops are fitted with optical transceivers that convert electrical signals into optical ones; these signals travel through fiber with far greater data fidelity and near-zero power loss compared to copper.

Integrated Photonics: Can It Cut Power Consumption in Half?

As photonic technology becomes further integrated inside servers, it could also resolve another major bottleneck. The space on a GPU available for copper connections is limited, which constrains how much data a chip can transfer at once. Integrated photonic chips, by contrast, can leverage different wavelengths of light to simultaneously carry multiple data streams over just a few optical pathways.

This technology can even enable GPUs to interact directly with memory chips on adjacent trays and racks, distributing the thermal load and allowing larger models to run at lower power. Some experts believe integrated photonics could reduce data center power consumption by nearly 50%. Yet even that may not be enough.

Silicon Photonics integrates optical components with conventional silicon-based semiconductor processes, enabling the generation, transmission, modulation, and detection of light on a single chip. Its core advantage lies in the fact that photons carry no electric charge and generate almost no heat as they propagate through a medium. A single optical pathway can simultaneously carry dozens of data signals at different wavelengths via wavelength-division multiplexing (WDM), offering far greater bandwidth density than copper interconnects. A major bottleneck in current AI chips is the "memory wall" — the speed at which data can be moved between the GPU and memory cannot keep up with the speed of computation, leaving vast amounts of processing power idle as it waits for data. Integrated photonic interconnects can dramatically increase inter-chip bandwidth and reduce latency, enabling GPUs to more efficiently access memory distributed across multiple trays and racks, addressing this bottleneck at its root. Intel, IBM, and numerous startups have invested heavily in this area, but large-scale production still faces challenges around yield rates for photonic-electronic integration and packaging costs.

The Energy Black Hole: By 2028, the Equivalent of Eight New York Cities

A single GPU draws roughly the same power as 16 light bulbs. Multiply that across every GPU in a data center, and the total is enough to power 300,000 homes.

Meanwhile, companies are racing to build increasingly "intelligent" AI — and so far, that has generally meant increasingly large AI. Even as efficiency improves, larger models still require larger data centers and more electricity. Some projections suggest that by 2028, U.S. data centers alone could consume electricity equivalent to eight New York Cities. In fact, some AI companies have already proposed building dedicated nuclear power plants to supply the next generation of data centers.

The reality is stark: we do not have enough energy to sustain this pace of expansion. Either we hit the brakes, or we find ways to "compute more intelligently."

The Way Out: Smaller Models, or Computing with Light?

One approach is to move away from "do-everything" systems. Current large language models are designed to answer any question, but you don't need a model that understands Shakespeare to help you fix your dishwasher. We could build small language models focused on specific domains, housed in compact micro data centers.

Another direction is optical computing — future large language models might perform their mathematical operations directly using light itself, dramatically cutting energy consumption.

But there is a more fundamental paradox here: if our demand for AI continues to grow, no amount of efficiency improvement will be sufficient. Ultimately, this is not a technical problem — it is a social one. How many resources is society willing to invest in building smarter machines?

Microchips power not only data centers but every electronic device we use today, from smartphones and cars to home appliances and medical devices. The energy bill for AI is ultimately a test of the trade-offs an entire society is willing to make.

Optical computing refers to using photons rather than electrons to directly perform mathematical operations. In matrix multiplication scenarios, optical processors can complete calculations at the speed of light through physical phenomena such as interference and diffraction, with energy consumption theoretically orders of magnitude lower than electronic chips. Companies like Lightmatter and Luminous Computing are already developing photonic tensor processor prototypes. However, optical computing currently faces challenges including insufficient precision, difficulty integrating with existing digital systems, and high manufacturing costs — large-scale commercialization remains a considerable distance away. Achieving the goal of "computing more intelligently" will require the coordinated advancement of multiple technical approaches — optical computing, novel memory architectures, sparsification algorithms, and more — rather than any single breakthrough.