TurboQuant
The core vector quantization algorithm used by TurboVec, designed to achieve a more favorable balance between compression ratio and retrieval accuracy.
Timeline (last 90 days)
On models with TurboQuant enabled, the community's actual runs generally set the Hot Cache Limit to 10% on 128 GB machines.
TurboQuant 3.5-bit matches full precision in quality, with KV cache memory usage at about one quarter (validated by independent research based on vLLM).
On oMLX 0.6.4, TurboQuant KV Cache cannot be stacked with VLM MTP, and in actual testing it does not bring generation speedup.
For sliding-window models such as Gemma 4, TurboQuant KV cache and KV cache quantization have been added.
All Facts (4)
On models with TurboQuant enabled, the community's actual runs generally set the Hot Cache Limit to 10% on 128 GB machines.
50%UnverifiedTurboQuant 3.5-bit matches full precision in quality, with KV cache memory usage at about one quarter (validated by independent research based on vLLM).
50%UnverifiedOn oMLX 0.6.4, TurboQuant KV Cache cannot be stacked with VLM MTP, and in actual testing it does not bring generation speedup.
50%UnverifiedFor sliding-window models such as Gemma 4, TurboQuant KV cache and KV cache quantization have been added.
50%