StreamingLLM
A KV cache management method designed for streaming inference scenarios. It supports infinite-length text generation by retaining attention sink tokens and recent tokens, while keeping GPU memory usage bounded.
Timeline (last 90 days)
Common KV cache eviction methods include H2O, ScissorHands, and StreamingLLM, whose core logic is to assign an importance score to each token and remove low-scoring tokens.
Attention Sinks were systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper, referring to the phenomenon where Transformers assign a large proportion of attention weights to the first 1–4 tokens at the beginning of a sequence.
The attention sink phenomenon was systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper: Transformers allocate a large proportion of attention weights to the 1st–4th tokens at the beginning of the sequence.
The attention sink phenomenon was systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper, referring to the tendency of Transformer models to assign a large proportion of attention weights to the first few tokens at the beginning of a sequence (typically tokens 1–4).
Attention Sinks were systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper, referring to the phenomenon where Transformer models allocate a large proportion of attention weights to the first 1–4 tokens at the beginning of a sequence.
All Facts (5)
Attention Sinks were systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper, referring to the phenomenon where Transformer models allocate a large proportion of attention weights to the first 1–4 tokens at the beginning of a sequence.
70%UnverifiedCommon KV cache eviction methods include H2O, ScissorHands, and StreamingLLM, whose core logic is to assign an importance score to each token and remove low-scoring tokens.
50%UnverifiedAttention Sinks were systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper, referring to the phenomenon where Transformers assign a large proportion of attention weights to the first 1–4 tokens at the beginning of a sequence.
50%UnverifiedThe attention sink phenomenon was systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper: Transformers allocate a large proportion of attention weights to the 1st–4th tokens at the beginning of the sequence.
50%UnverifiedThe attention sink phenomenon was systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper, referring to the tendency of Transformer models to assign a large proportion of attention weights to the first few tokens at the beginning of a sequence (typically tokens 1–4).
50%