Quadrillion-Parameter Transformers: Training Feasibility and Engineering Challenges

From Trillion to Quadrillion: A Leap of an Order of Magnitude

A recent discussion on Hacker News caught attention with a striking title: Quadrillion-parameter transformers and how to train them. The post currently has only 7 points and 1 comment, not particularly popular, but the question it points to is far from trivial—when model parameter scale jumps another three orders of magnitude from trillion to quadrillion (10^15), can existing training paradigms still hold?

The value of this question isn't about "some company is about to train a quadrillion-parameter model tomorrow," but rather that it forces us to re-examine the physical boundaries of scaling laws: memory, bandwidth, energy consumption, parallelism strategies—each layer hits a wall at some order of magnitude.

What a Quadrillion Parameters Means

Let's break down the numbers first. A dense model with a quadrillion parameters, even stored in FP8 precision, would require roughly 1000 TB of memory just for the weights themselves—far exceeding the capacity of a single machine or even a single pod. Even with MoE sparse architectures to reduce activated parameters, the storage and communication overhead corresponding to total parameters remains an unavoidable hard constraint.

For reference, frontier models currently discussed publicly are mostly within a trillion parameters. From trillion to quadrillion is a span of three orders of magnitude, not a routine version iteration. This also explains why the first reaction to this topic on Hacker News tends to be "first ask how to store it, then ask how to train it."

MoE (Mixture of Experts) is a sparse activation architecture: the model contains multiple independent "expert" feed-forward networks, and each token is routed to only a few of them via a router. Therefore, the total parameter count can be far larger than the parameters actually activated per token—for example, 1 trillion total parameters with only tens of billions activated at a time. This makes "inflating total parameters while keeping computation manageable" possible, and it's an unavoidable structure in quadrillion-scale discussions. The cost is that routing decisions themselves need training, expert load tends to be imbalanced, and when experts are distributed across different devices, cross-node all-to-all communication arises—the larger the scale, the harder the scheduling.

Several Possible Training Paths

Regarding quadrillion-parameter training, the original discussion didn't lay out a systematic technical solution, but combined with current mainstream approaches to large-scale training, several repeatedly mentioned directions can be outlined:

Extreme Parallelism and Sharding

The combination of tensor parallelism, pipeline parallelism, data parallelism, and sequence parallelism is almost inevitable. The problem is that when parallelism is pushed to extremes, communication overhead quickly eats into computational efficiency, and gradient synchronization latency becomes a bottleneck. How to maintain acceptable MFU across tens of thousands or more accelerators is the first question to answer.

Sparsification and MoE

Dense models are nearly infeasible at quadrillion scale, and MoE is widely seen as the means to inflate total parameters while keeping computation manageable. But MoE brings new problems—routing balance, expert load skew, cross-node all-to-all communication—that get harder to tune as scale grows.

Low Precision and Memory Hierarchy

FP8 and even lower-precision training, offloading weights to host memory or external storage, tiered checkpoint management—all of these shift from "optimizations" to "necessities" at this scale.

MFU (Model FLOPs Utilization) measures the proportion of total peak compute actually used for model forward and backward computation during training. In large-scale distributed training, gradient synchronization, pipeline bubbles, and all-to-all communication all cause accelerators to idle, making MFU a core metric for judging whether a parallelism strategy is worthwhile. Industry large-scale training MFU typically falls in the 30%–50% range; when parallelism is pushed to tens of thousands of accelerators, communication overhead significantly depresses this value. So in the quadrillion-parameter scenario, "can acceptable MFU be maintained" is equivalent to "is the parallelism scheme truly feasible."

A More Realistic Question: Is It Worth It

A sharper question than "how to train" is "why train." Whether the scaling law's benefit curve can continue at quadrillion parameters has no public evidence to support it. Data-side constraints are equally obvious—the total amount of high-quality tokens is finite, and when parameter scale far exceeds data scale, whether the model can effectively utilize that capacity is questionable.

Discussions in technical communities like Hacker News often turn to cost-benefit at this point: the cost of a single training run, the feasibility of inference deployment, and whether it can deliver substantial improvements over a carefully tuned medium-scale model. The original post has limited information and offers no conclusions, but this is precisely where readers should make their own judgment.

Scaling law refers to the empirical regularity that model loss decreases as a power law with parameter count, data volume, and compute, and has been the theoretical basis for "scaling up" large models over the past few years. But power-law returns diminish: the loss reduction gained per order of magnitude gets smaller and smaller, and the law typically holds under the premise of sufficient training data. When parameter scale far exceeds the total amount of high-quality tokens, the model may not be able to fully utilize its capacity, and whether the benefit curve can still be extrapolated is debated. This is precisely the most fundamental uncertainty in quadrillion-parameter discussions—it's not an engineering problem, but a question of whether an empirical regularity can continue to hold across three orders of magnitude.

Beyond Engineering: The Value of a Thought Experiment

Viewed as a thought experiment, the significance of quadrillion parameters lies in calibrating boundaries. When researchers seriously work through "if we were to do it, where are the bottlenecks," they're effectively drawing a roadmap for next-generation infrastructure: what level interconnect bandwidth needs to reach, how storage tiers should be designed, how fault tolerance and checkpoint strategies should evolve.

For engineers focused on large-scale training, the value of such discussions isn't in the conclusions but in how specifically they break down the problem, making clear the engineering cost behind each order of magnitude. Although the original post is brief, the question it raises is big enough to deserve serious attention.

(Note: The original material contains only the title and engagement data, with no specific technical solutions or data provided. This article analyzes the problem indicated by the title and does not cite unverified specific figures beyond the source material.)

Background Supplement

FP8 and Low-Precision Training: FP8 refers to 8-bit floating-point formats (such as E4M3, E5M2), which can halve the memory footprint of weights and activations compared to FP16/BF16. Its feasibility depends on fine-grained scaling factors (per-tensor or per-block scaling) to suppress numerical overflow and underflow. At quadrillion-parameter scale, low precision isn't just a speedup technique but a necessary prerequisite for fitting weights, optimizer states, and activations into available memory tiers; accompanying this are optimizer state sharding (such as ZeRO) and tiered strategies that offload inactive parameters to host memory or NVMe.