Zstandard

By combining the Zstandard compression algorithm with the Pingora proxy framework, it is expected to save petabytes of cache storage space.

Zstandard is an open-source compression algorithm from Meta (formerly Facebook).

The core advantage of Zstandard lies in achieving an excellent balance between compression ratio and compression/decompression speed.

Compared to traditional gzip/zlib, zstd typically delivers a higher compression ratio while maintaining extremely fast decompression speeds.

Zstandard supports training compression dictionaries, which can further improve the compression ratio for large numbers of similar small files.

Data in caching systems tends to follow a write-once, read-many access pattern, so a higher compression level can be applied at write time while relying on high-speed decompression at read time to guarantee response latency.

Embedding zstd compression logic into the Pingora proxy layer allows compression and decompression operations to be completed natively within the request processing pipeline, without relying on external services or additional process scheduling.

Compression is essentially trading compute for storage, and when using high compression levels it is necessary to evaluate whether CPU overhead will become a new bottleneck.

Introducing a new compression format into an existing system requires consideration of backward compatibility, including how to handle existing uncompressed cache entries and how to support both formats simultaneously during a gradual rollout.

The actual effectiveness of cache compression is highly dependent on the specific data distribution and cache hit patterns.