#scaling law

Quadrillion-Parameter Transformers: Training Feasibility and Engineering Challenges

Starting from a Hacker News discussion, this analyzes the training feasibility of quadrillion-parameter transformers: memory and communication hard constraints, MoE sparsification and extreme parallelism engineering paths, and realistic questions about scaling law returns and data bottlenecks.