GPT-2 uses learnable positional embeddings added to token embeddings; if the K/V ordering in the cache is misaligned with the positions assumed by the query vector, then when the sequence length is greater than 1 a mismatch occurs between query positions and cache positions, changing the visible set under the causal mask and thereby changing the softmax weights and output.