Length Generalization
The ability of a model that has seen only shorter sequences during training to handle longer sequences at test time, where the test distribution is systematically shifted relative to the training distribution (longer sequences), posing challenges to mechanisms such as positional encoding and attention masking in Transformers.
Timeline (last 90 days)
The paper proposes a "position realizability theory" to explain the dependence of length generalization on positional encoding and derives widely documented empirical patterns.
Length generalization refers to the ability of a model that has only seen shorter sequences during training to handle longer sequences at test time.
All Facts (2)
The paper proposes a "position realizability theory" to explain the dependence of length generalization on positional encoding and derives widely documented empirical patterns.
50%UnverifiedLength generalization refers to the ability of a model that has only seen shorter sequences during training to handle longer sequences at test time.
50%