GAP Architecture: A Gated Transformer Block That Merges Attention and MLP

Since the Transformer architecture was introduced, the attention sublayer and the feed-forward network (MLP) sublayer have been stacked alternately as two independent modules. A researcher on Reddit (rickey1990) has proposed an alternative architecture called GAP (Gated Attention Projection), which attempts to merge these two steps into a single gated block. The author ran controlled experiments on WikiText-103 and open-sourced the code, paper, and speed benchmarks.

Design Motivation: Why Merge Attention and MLP

Each layer of a standard Transformer consists of an attention sublayer and an MLP sublayer in series, each with its own parameters and residual path. GAP's idea is to use a single gated structure to simultaneously perform token-to-token attention interaction and per-token feature transformation, thereby reducing redundant projection matrices and intermediate activations.

According to the author, the goal of this design is not to pursue higher final accuracy but to explore parameter efficiency: under the same training token budget, can fewer parameters carry comparable language modeling capability? This "structural merging" idea is similar in direction to recent gated linear attention and state space models, but GAP retains the form of standard attention, with changes concentrated in how the block is organized internally.

Gated Linear Attention (GLA) is a class of models that introduce gating mechanisms into linear attention variants. Standard softmax attention requires computing similarities between all token pairs, with complexity growing quadratically with sequence length; linear attention instead reduces complexity to linear by approximating with kernel functions or rewriting the attention computation. GLA builds on this by introducing a data-dependent gating vector for each channel to control the retention and forgetting of historical information, similar in form to the forget gate in RNNs. Such structures are more efficient than standard attention for long-sequence modeling, but typically require carefully designed gating initialization and normalization, otherwise training tends to be unstable. The difference between GAP and GLA is that GAP retains the form of standard attention, with changes concentrated in merging the attention and MLP steps into a single block rather than replacing the computation of attention itself.

State Space Models (SSM) are a class of models that describe sequence evolution using hidden-state recurrence; their continuous-time form can be discretized into linear recurrence, thereby processing long sequences with linear complexity. Representative work such as Mamba introduces an input-dependent selection mechanism into SSMs, bringing their language modeling performance close to Transformers. SSMs and gated linear attention share mathematical structure: both use data-dependent gates to control hidden-state updates, and both can be viewed as gated linear recurrence. GAP is compared to these directions because they all attempt to replace the two-stage "attention + MLP" stacking with a more compact intra-block structure, but GAP does not adopt a recurrent form—it still maintains explicit attention interaction between tokens.

Experimental Setup and Key Data

The author used the WikiText-103 dataset, training each model on 100 million scored training tokens, with a standard Transformer as the control. The core results include:

• GAP20 uses 19.91% fewer parameters than the standard Transformer;
• A smaller configuration can reduce parameters by 34.31%, but that version performs worse relative to the standard Transformer than the current model, so the author gave up 14% of the parameter savings to widen the GAP model, ultimately retaining about 20% fewer parameters; this configuration is referred to as GAP20 in the tests;
• On final language model accuracy, the standard Transformer performs better;
• In a CPU benchmark, compiling GAP20 with torch.compile improved training throughput by about 19.9% compared to the uncompiled version.

These numbers indicate that GAP has some room for parameter efficiency, but the trade-off in accuracy is clear. The author himself emphasizes that this is an experimental architecture and does not claim it is overall superior to the Transformer.

Scored training tokens refers to the number of tokens that participate in loss computation, typically excluding padding, masked positions, or tokens used only as context without prediction. In language modeling, measuring the training budget by scored token count rather than total raw tokens allows fairer comparison across models with different context lengths or masking strategies. 100 million scored training tokens means that each model actually used about one hundred million prediction targets for parameter updates during training. This scale is smaller than the full training runs commonly used on WikiText-103, so the results are better suited as a preliminary comparison of architectural feasibility rather than a final performance conclusion.

How to Interpret These Results

With about 20% fewer parameters and slightly lower accuracy, GAP does not necessarily have an advantage in "performance per parameter," but it may have reference value in scenarios with limited GPU memory or storage. More noteworthy is the detail that the 34.31% parameter-reduction version performed poorly: it suggests that the capacity loss from gated merging is not linearly compensable, and simply widening the model can only partially recover performance.

The throughput gain from torch.compile is an engineering-level benefit, distinct from the architecture's own language modeling capability. The author lists it separately, which also indicates that the current stage focuses on validating design feasibility rather than chasing leaderboard rankings.

torch.compile is a compilation frontend introduced in PyTorch 2.x. It captures the computation graph at the Python level through TorchDynamo, then hands it to a backend (such as Inductor) to generate optimized kernel code, thereby reducing framework scheduling overhead, fusing element-wise operations, and improving throughput. It does not change the model's mathematical definition, so accuracy should be consistent before and after compilation; the difference appears only in training or inference speed. This also explains why GAP20's 19.9% throughput improvement should be classified as an engineering benefit: it measures the runtime efficiency of the same architecture under compilation optimization, which is an independent dimension from the parameter efficiency brought by gated merging itself.

Open Questions and Directions for Improvement

In the post, the author openly solicits feedback on the design, mathematical derivation, and directions for improvement, and provides code with configurable model sizes. For researchers interested in efficient Transformer structures, the following questions are worth further validation: whether the initialization of the gating function and the placement of normalization affect training stability; whether the parameter advantage holds at larger data scales; and whether mixed stacking of merged blocks and standard blocks can balance accuracy and efficiency.

The repository is at github.com/rickey1990/Gated-Attention-Projection, containing the architecture code, speed benchmarks, and research paper.