Attention Mass
The proportion of total attention weight that a model allocates to specific positions in the input sequence, such as evidence spans, when making predictions. It measures the degree of attention concentration rather than the length of the context window.
Timeline (last 90 days)
Attention mass remaining on the evidence explains at most 58% of the variance in retrieval loss across 126,473 trials, while log length explains at most 8%.
Attention mass refers to the proportion of the total attention weights that the model allocates to specific positions (such as the evidence span) when predicting, and can be computed by reading the attention maps of each layer and aggregating them onto the evidence tokens.
For teams building long-context applications, the actionable check is to monitor the model's attention mass on the evidence, not just input length or recall.
All Facts (3)
Attention mass remaining on the evidence explains at most 58% of the variance in retrieval loss across 126,473 trials, while log length explains at most 8%.
50%UnverifiedAttention mass refers to the proportion of the total attention weights that the model allocates to specific positions (such as the evidence span) when predicting, and can be computed by reading the attention maps of each layer and aggregating them onto the evidence tokens.
50%UnverifiedFor teams building long-context applications, the actionable check is to monitor the model's attention mass on the evidence, not just input length or recall.
50%