HSQ-A
An attention optimization technique for long-context inference that selects high-impact keys through lightweight scoring and stores values with FP8 quantization, substantially compressing KV-cache memory usage.
Timeline (last 90 days)
HSQ-A uses a lightweight scoring network to select about 15% of high-impact keys and stores the remaining values in FP8 format, thereby compressing the KV-cache from the 0.8 GB/k common in dense 70B models to 0.12 GB/k, a compression ratio of about 85%.
For a 750,000-token contract document, the original KV-cache is about 90GB, which after the HSQ-A compression pipeline drops to the order of about 10GB and can fit into the remaining VRAM of an A100-40GB.
HSQ-A's aggressive pruning may introduce compression artifacts, discarding subtle long-range dependencies, and warrants particular caution on code and formal-logic tasks.
All Facts (3)
HSQ-A uses a lightweight scoring network to select about 15% of high-impact keys and stores the remaining values in FP8 format, thereby compressing the KV-cache from the 0.8 GB/k common in dense 70B models to 0.12 GB/k, a compression ratio of about 85%.
50%UnverifiedFor a 750,000-token contract document, the original KV-cache is about 90GB, which after the HSQ-A compression pipeline drops to the order of about 10GB and can fit into the remaining VRAM of an A100-40GB.
50%UnverifiedHSQ-A's aggressive pruning may introduce compression artifacts, discarding subtle long-range dependencies, and warrants particular caution on code and formal-logic tasks.
50%