HF Buckets
HF Buckets is a cloud storage service provided by Hugging Face. It integrates Xet incremental upload technology to support efficient storage and management of large datasets. File operations are performed through CLI commands, enabling streamlined handling of large-scale data assets on the Hugging Face platform.
Timeline (last 90 days)
Hugging Face Buckets are compatible with the Amazon S3 protocol, allowing any tool that supports the S3 API to read from and write to a Bucket, but incremental deduplication uploads are only achieved when writing with HF toolchain.
The copy and sync commands in Buckets will directly overwrite existing content; once data is finalized, it should be copied into a dataset repo to preserve version history.
The demonstration uses approximately 1.5 million rows of New York City taxi trip CSV data totaling 191MB; the first upload transfers the complete file, and after appending 100 rows only the newly added portion is uploaded.
All Facts (3)
Hugging Face Buckets are compatible with the Amazon S3 protocol, allowing any tool that supports the S3 API to read from and write to a Bucket, but incremental deduplication uploads are only achieved when writing with HF toolchain.
50%UnverifiedThe copy and sync commands in Buckets will directly overwrite existing content; once data is finalized, it should be copied into a dataset repo to preserve version history.
50%UnverifiedThe demonstration uses approximately 1.5 million rows of New York City taxi trip CSV data totaling 191MB; the first upload transfers the complete file, and after appending 100 rows only the newly added portion is uploaded.
50%