Xet
Xet is Hugging Face's storage layer technology that implements file-level incremental uploads based on Content-Defined Chunking (CDC). It is built into the Hugging Face CLI and Hub, and reduces bandwidth consumption from redundant data transfers by ensuring that only modified chunks are uploaded.
Timeline (last 90 days)
Xet is Hugging Face's storage layer technology, which can be understood as Git applied to file chunks rather than entire files.
Xet is already built into the Hugging Face CLI and Hugging Face Hub, requiring no additional installation.
Xet splits each file into chunks of approximately 64KB, with split points determined by the content itself.
Before uploading, the Xet client queries the Hub for the fingerprints of existing chunks, and chunks that already exist are not transmitted.
Parquet's default write method disrupts Xet's deduplication logic, because its columnar encoding, Row Group boundaries, and Footer are rewritten each time, causing the byte layout to change dramatically.
For a 100MB Parquet file with 3,000 rows added near the beginning, nearly the entire file must be re-uploaded with default settings, whereas enabling content-defined chunking requires uploading only approximately 6 chunks.
Re-sorting a file to accelerate queries causes every chunk to change, resulting in a full-file upload; the correct approach is to sort first, then write, and only append afterward.
Switching compression algorithms, changing page settings, or default parameter changes after a library upgrade will cause Xet to treat the same data as a new file, and the first upload after an upgrade will be a full upload.
The demonstration uses approximately 1.5 million rows of New York City taxi trip CSV data totaling 191MB; the first upload transfers the complete file, and after appending 100 rows only the newly added portion is uploaded.
All Facts (9)
Xet is Hugging Face's storage layer technology, which can be understood as Git applied to file chunks rather than entire files.
50%UnverifiedXet is already built into the Hugging Face CLI and Hugging Face Hub, requiring no additional installation.
50%UnverifiedXet splits each file into chunks of approximately 64KB, with split points determined by the content itself.
50%UnverifiedBefore uploading, the Xet client queries the Hub for the fingerprints of existing chunks, and chunks that already exist are not transmitted.
50%UnverifiedParquet's default write method disrupts Xet's deduplication logic, because its columnar encoding, Row Group boundaries, and Footer are rewritten each time, causing the byte layout to change dramatically.
50%UnverifiedFor a 100MB Parquet file with 3,000 rows added near the beginning, nearly the entire file must be re-uploaded with default settings, whereas enabling content-defined chunking requires uploading only approximately 6 chunks.
50%UnverifiedRe-sorting a file to accelerate queries causes every chunk to change, resulting in a full-file upload; the correct approach is to sort first, then write, and only append afterward.
50%UnverifiedSwitching compression algorithms, changing page settings, or default parameter changes after a library upgrade will cause Xet to treat the same data as a new file, and the first upload after an upgrade will be a full upload.
50%UnverifiedThe demonstration uses approximately 1.5 million rows of New York City taxi trip CSV data totaling 191MB; the first upload transfers the complete file, and after appending 100 rows only the newly added portion is uploaded.
50%