Hierarchical Residuals (HiRes): Decoupling Residual Width from Compute Width for a 1.3x FLOPs Efficiency Gain

HiRes decouples residual width from compute width, with a fitted scaling law showing ~1.30x FLOPs efficiency improvement.
In a standard Transformer, the residual stream width is forced to equal each layer's compute width d, so widening the residual requires proportionally widening attention and FFN, causing linear compute growth. The paper proposes Hierarchical Residuals (HiRes), grouping layers into blocks that keep standard d-dimensional PreNorm layers, with wider D-dimensional PostNorm outer residual connections between blocks and learnable read/write projections bridging d and D at boundaries. The mechanism uses only dense projections and normalization, requiring no custom kernels. Experiments show consistent gains over vanilla residuals across sizes and depths, with a fitted scaling law giving about 1.30x FLOPs efficiency; pretraining OLMoE-1B-7B on 500B tokens reduces training loss by 0.0208 and improves downstream accuracy by 1.14 points, matching VWN, mHC, and AttnRes in performance but with lower activation memory and I/O overhead.
Transformer residual connections are nearly ubiquitous, but they carry an implicit constraint: the residual width that runs through the model's depth must match the compute width inside each layer. A new paper proposes Hierarchical Residuals (HiRes), a minimal architectural change that separates these two dimensions—the residual state can be wider while each layer's compute width stays the same. Ablation experiments show that this decoupling delivers consistent gains across model sizes and depths, and the fitted scaling law points to a 1.30x FLOPs efficiency improvement.

The Problem: Residual Width Is "Tied" to Compute Width
In a standard Transformer, each sublayer's input and output share the same dimension d, and the residual addition also happens in this d-dimensional space. This means that if you want to widen the residual stream to improve gradient propagation and information retention, you are forced to widen each layer's attention and FFN as well, and compute cost grows linearly with it.
The paper treats this implicit coupling as a design waste: the residual stream's job is to carry state across layers, while compute width determines a single layer's expressiveness—the two could have different optimal values. HiRes's starting point is to untie this constraint without introducing any special kernels or infrastructure changes.
PreNorm vs. PostNorm are the two ways of arranging the relative positions of residual connections and normalization layers in a Transformer. PreNorm means normalization comes before the sublayer (attention or FFN) and the residual addition comes after normalization, i.e., x + Sublayer(Norm(x)); PostNorm places normalization after the residual addition, i.e., Norm(x + Sublayer(x)). The key difference is whether normalization exists on the residual path: in PreNorm the residual stream is an unnormalized "highway" where gradients can flow back losslessly, making training more stable—this is the mainstream choice for modern large models; PostNorm normalizes after the residual addition, offering greater expressiveness but being more sensitive to initialization and learning rate during deep training. HiRes keeps PreNorm inside blocks to preserve single-layer training stability, while applying PostNorm to the outer residual between blocks—precisely leveraging PostNorm's additional normalization constraint during cross-block propagation to maintain numerical stability in the wider D-dimensional residual space.
How HiRes Works: PreNorm Inside Blocks, PostNorm Between Blocks
HiRes's structure can be understood in three layers:
- Grouping: Several consecutive Transformer layers are grouped into a block; inside the block, layers remain standard d-dimensional PreNorm layers, with the single-layer structure completely unchanged.
- Outer widening: Between blocks, a wider D-dimensional (D > d) PostNorm outer residual connection is used, and the residual state is carried across blocks in this wider space.
- Read/write projections: At block boundaries, learnable read/write projections bridge the two dimension spaces, writing d-dimensional layer outputs into the D-dimensional residual stream and reading them back to d dimensions for the next block.
The entire mechanism consists only of dense projections and normalization—no fancy operators. This is key for deployment: it means no custom CUDA kernels are needed, nor any infrastructure-level optimization, and existing training stacks can run it directly.
Read/write projections are two sets of learnable linear transformations introduced by HiRes at block boundaries, whose role is to convert between the d-dimensional and D-dimensional spaces. Specifically, the write projection maps the d-dimensional layer output from the previous block into the D-dimensional residual stream, and the read projection maps the D-dimensional residual state back to d dimensions as input to the next block. Since D > d, write is an up-projection and read is a down-projection, forming an asymmetric linear bridge. Both are ordinary dense matrix multiplications with parameter counts of d×D and D×d, negligible relative to each layer's attention and FFN parameters. The cleverness of this design is that the residual stream can accumulate cross-block information in D-dimensional space while each block internally still performs standard computation in the familiar d-dimensional space, so the optimal values of the two can be tuned independently.
Experiments: Scaling Law Points to 1.30x FLOPs Efficiency
The paper compares HiRes with vanilla residual addition across model sizes and depths, and the conclusion is consistent superiority over the baseline rather than an incidental gain in some configuration. By fitting a scaling law to the results, the authors estimate a 1.30x FLOPs efficiency improvement—under the same compute budget, HiRes reaches a lower loss.
Larger-scale language model pretraining validation also supports this: pretraining OLMoE-1B-7B on 500B tokens, HiRes achieves 0.0208 lower training loss and 1.14 percentage points higher downstream accuracy. For the pretraining stage, both numbers fall in a practically meaningful range.
Scaling laws describe the power-law relationship between model loss and parameter count, data volume, and compute (FLOPs), typically of the form L = a·N^(-α) + b·D^(-β) + c. In architecture comparisons, researchers often fit scaling-law curves for both architectures separately and then compute the ratio of FLOPs required by each at a fixed loss—this is the "FLOPs efficiency improvement factor." A factor of 1.30 means that to reach the same loss, HiRes needs only about 77% of vanilla's compute. Note that the scaling law is an extrapolated fit over multiple experimental scale points, and its coefficients are affected by the fitting range, data distribution, and hyperparameter configuration, so this number should be understood as a trend estimate rather than a precise guarantee. The concrete pretraining results on OLMoE-1B-7B provide single-point empirical support; only the two together allow a fairly complete assessment of the credibility of the gains.
Comparison with Other Residual-Widening Methods
Widening residuals is not an entirely new idea—VWN, mHC, and AttnRes have all explored similar directions. HiRes's difference lies in implementation cost: the paper claims performance comparable to these methods but with lower activation memory and I/O overhead. This is consistent with its design philosophy—trading standard operators and dense projections for engineering simplicity rather than compensating for overhead with specialized kernel optimization.
In other words, HiRes turns "residual width" into a scaling variable that can be tuned independently. When compute width is limited by hardware or latency budgets, this offers an additional optimization path: widen the residual stream to improve efficiency without proportionally scaling up each layer's compute.
A Brief Observation
HiRes's value may lie not in any specific number but in its reexamination of a design assumption that has been accepted by default. Residual connections have been carried over since the ResNet era, and their width's consistency with layer width is rarely treated as a tunable option. As scaling laws gradually approach compute bottlenecks, "decoupling" changes like this—trading a minimal architectural cost for a new tunable dimension—are often more cost-effective than simply stacking parameters. Of course, the 1.30x efficiency gain comes from a fitted scaling law, and actual gains will vary with model size, data, and training configuration; validation on your own workload is still needed before deployment.
One-line Summary
HiRes decouples residual width from compute width, with a fitted scaling law showing roughly 1.30x FLOPs efficiency improvement.
Paragraph Summary
In a standard Transformer, the residual stream width is forced to match each layer's compute width as the same d, so widening the residual requires proportionally widening attention and FFN, causing linear compute growth. The paper proposes Hierarchical Residuals (HiRes), which groups several layers into a block, keeps standard d-dimensional PreNorm layers inside the block, uses a wider D-dimensional PostNorm outer residual connection between blocks to carry state, and bridges d and D at boundaries with learnable read/write projections. The entire mechanism consists only of dense projections and normalization, requiring no custom kernels. Experiments show it consistently outperforms vanilla residuals across sizes and depths, with a fitted scaling law giving about 1.30x FLOPs efficiency improvement; pretraining OLMoE-1B-7B on 500B tokens reduces training loss by 0.0208 and improves downstream accuracy by 1.14 percentage points. It performs comparably to VWN, mHC, and AttnRes but with lower activation memory and I/O overhead.
Related articles

Gemini 4 Helium Coming Soon? Signals of Google Loosening Subscription Restrictions
Google team member hints releases will continue; community speculates Gemini 4 Helium (Flash Lite) may be next. Subscription ban controversy gets a response as product lead says "Soon," suggesting restrictions may be lifted. This article covers Gemini developments, OpenAI Coding Agent and risk management, Doubao real-time audio-video API, and compute business dynamics.

AI-Assisted React Development in Practice: Optimizing a Video App and Handling Errors
A 14-year veteran developer demonstrates AI-assisted React video app optimization: compressing video from 80MB to 5MB (~94%) via ImageKit, plus reliable error handling and retry mechanisms. Includes planning, code understanding, and verification methods for working with AI.

Incident Response in the Age of AI Attackers: A 6-Month Retrospective and the Four Pillars
AI attackers are making attacks on AI infrastructure more frequent. This article explains MarkTechPost's four-pillar incident response playbook based on a 6-month retrospective, with Fireworks AI and Klue case studies on implementation.