FlexBudget
An experimental methodology for fairly comparing KV cache eviction policies under a unified compute budget and fixed kernel conditions. It uses a training-free implementation written purely in core PyTorch, with the aim of separating algorithmic contributions from differences in engineering implementation.
Timeline (last 90 days)
AMD engineer Huan Ran presented a scheme called FlexBudget in a poster session at PyTorch Conference North America 2026, aiming to solve the methodological problem of rules and implementations being entangled when comparing KV cache eviction strategies.
The core idea of FlexBudget is to fix the kernel and memory budget and replace only the KV cache eviction rule itself, so that different strategies compete under completely fair conditions.
FlexBudget is training-free and can be used without any additional training or fine-tuning.
FlexBudget is implemented entirely natively in PyTorch and does not rely on custom CUDA extensions.
All Facts (4)
The core idea of FlexBudget is to fix the kernel and memory budget and replace only the KV cache eviction rule itself, so that different strategies compete under completely fair conditions.
50%UnverifiedFlexBudget is training-free and can be used without any additional training or fine-tuning.
50%UnverifiedFlexBudget is implemented entirely natively in PyTorch and does not rely on custom CUDA extensions.
50%UnverifiedAMD engineer Huan Ran presented a scheme called FlexBudget in a poster session at PyTorch Conference North America 2026, aiming to solve the methodological problem of rules and implementations being entangled when comparing KV cache eviction strategies.
50%