DEVOURED
Hugging Face Storage Buckets (Website)
Hugging Face is launching dedicated storage buckets that use content-defined chunking to deduplicate model artifacts and reduce data transfer costs.
What: Hugging Face Storage Buckets offer per-TB pricing with integrated CDN support and Xet-based deduplication, allowing developers to sync datasets and checkpoints without Git overhead.
Why it matters: This signals a move by Hugging Face to become a full-service platform for ML lifecycle management, positioning itself as a more specialized alternative to generic S3-compatible cloud storage.
Takeaway: If you are managing large model checkpoints or datasets, use the `hf sync` command to leverage their content-defined chunking and deduplication features.
Deep dive
- Content-Defined Chunking: A technique used by Xet to split files into variable-sized chunks based on data patterns rather than fixed offsets, significantly improving deduplication across evolving datasets.
- Deduplication: By identifying identical data segments across versions, only changed chunks (e.g., 5% of weight updates) are uploaded.
- No Git Overhead: Bypassing traditional Git storage limits allows for handling massive binary artifacts without the associated commit history bloat.
- Built-in CDN: Accelerates download speeds for distributed training clusters across different regions.
- Per-TB Pricing: A simplified billing model designed specifically for AI/ML workloads compared to complex cloud object storage egress and request fees.
Decoder
- Deduplication: A process that eliminates redundant copies of data by storing only unique chunks and using references for duplicates.
- Artifact: In software/ML, any tangible byproduct produced during the development process, such as model weights, logs, or processed dataset snapshots.
Original article
Storage built for AI teams
Store models, datasets, and artifacts with simple per-TB pricing. Xet deduplication. Included CDN. No git overhead.
- Per-TB pricing with built-in CDN and deduplication speedups.
- No Git constraints: commit-free sync and fast object updates.
- Designed for ML workflows: datasets, checkpoints, model artifacts.
# Create a storage bucket
$ hf buckets create acme-corp/training-data
✓ Bucket created: hf://buckets/acme-corp/training-data
✓ Visibility: private · Region: us-east-1
# Sync training data to the bucket
$ hf sync ./checkpoints/ hf://buckets/acme-corp/training-data
Scanning local files... 12,847 files (2.4 TB)
Xet dedup: 62% deduplicated : uploading 912 GB (saved 1.5 TB)
█████████████████████████ 78% 714/912 GB · 2.1 GB/s · ETA 1m 34sNext-gen large-scale storage for AI
Xet uses content-defined chunking to break files into byte-level chunks and deduplicates across your entire bucket. When you retrain a model and only 5% of weights change, only that 5% is re-uploaded.
- Raw + processed dataset: stored once, billed once*
- 4x less data per upload, verified with real-world workloads
*Requires Enterprise or Enterprise Plus plan
Traditional S3 Upload
8 / 8 chunks uploaded
VS
XET Deduplicated Upload
1 / 8 chunks uploaded
Gray = already stored · Purple = only the changed chunk
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi