Semantic Cache
Semantic Cache is a caching technique that reuses inference results for similar queries by matching embedding vector similarity. It is used to reduce the cost of calling large models.
Core Facts
Timeline (last 90 days)
Semantic Cache reuses inference results via an embedding similarity threshold; a threshold that is too high causes missed hits, while one that is too low misjudges semantically different questions as the same.
Caching in large model services is generally divided into two levels: semantic caching and prompt prefix caching (KV Cache).
Semantic caching converts queries into vectors via an embedding model and performs similarity search in the vector space, returning cached results directly when the similarity exceeds a set threshold.
All Facts (3)
Caching in large model services is generally divided into two levels: semantic caching and prompt prefix caching (KV Cache).
75%UnverifiedSemantic caching converts queries into vectors via an embedding model and performs similarity search in the vector space, returning cached results directly when the similarity exceeds a set threshold.
60%UnverifiedSemantic Cache reuses inference results via an embedding similarity threshold; a threshold that is too high causes missed hits, while one that is too low misjudges semantically different questions as the same.
50%