Compute-Optimal ≠ Cluster-Optimal: The Real-World Dilemmas of Distributed LLM Training and How to Overcome Them

Introduction: The Gap Between Theoretical Optimality and Engineering Reality

In the field of large language model (LLM) training, "Compute-Optimal" has long been regarded as the gold standard. Since DeepMind's Chinchilla paper, the industry has widely accepted a core principle: given a fixed compute budget, model parameter count and training data volume should maintain a certain optimal ratio to achieve the best performance.

However, a reality that an increasing number of practitioners have come to recognize is that compute-optimal does not equal cluster-optimal.

Behind this seemingly simple assertion lies a profound gap between theory and engineering practice. When we take the optimal formulas on paper and execute them on real GPU clusters, we discover that the factors truly determining training efficiency extend far beyond "floating-point operation counts" alone.

What Is Compute-Optimal? A Deep Dive into the Chinchilla Scaling Law

The Core Idea Behind the Chinchilla Scaling Law

The concept of compute-optimality stems from a fundamental question about training resource allocation: if you have a fixed compute budget (typically measured in FLOPs), should you spend more of that compute on increasing model size, or on feeding the model more training data?

Chinchilla's research concluded that many prior large models (such as GPT-3) were actually "undertrained" — they had too many parameters relative to the amount of training data. The optimal strategy should scale model parameters and training token count at roughly the same rate. This finding profoundly influenced the design paradigm of subsequent models.

In 2022, DeepMind published a paper titled Training Compute-Optimal Large Language Models, which conducted systematic experiments across more than 400 models of varying scales, revealing a serious imbalance in how the industry had been allocating model size versus data volume. The paper's core finding was that for a compute budget C, the optimal parameter count N and training token count D should satisfy the approximate relationships N∝C^0.5 and D∝C^0.5 — meaning both should grow at similar rates as the compute budget increases. This directly challenged the prevailing "large model, less data" paradigm exemplified by GPT-3, and gave rise to Chinchilla (70B parameters, 1.4T tokens), which significantly outperformed Gopher (280B parameters, 300B tokens) with the same compute budget. This means that blindly scaling up parameter count without matching it with sufficient training data is effectively a waste of the compute budget.

Theoretical Assumptions and Limitations of Compute-Optimality

However, compute-optimality is fundamentally a theoretical abstraction. It reduces the entire training process to a single variable — "total FLOPs" — and implicitly assumes:

• All compute can be fully utilized
• Communication overhead between hardware is negligible
• Memory capacity is not a bottleneck
• Cluster size can scale freely

FLOPs (Floating Point Operations) is the basic unit for measuring computational workload. In Transformer models, the FLOPs for a single training step is approximately 6ND (where N is the parameter count and D is the number of training tokens), covering both forward and backward pass computations. However, FLOPs fundamentally only measures "how many mathematical operations need to be performed," not "how long it takes to perform them." Actual execution time also depends on memory bandwidth (whether data can be delivered to compute units in time), the degree of parallelism in compute units, the efficiency of data flow through