bk99.de entertain the web since 1997

bk99.de
HF · StorageCurated

XetHub brings block-based storage to the Hub

Summary

XetHub joins Hugging Face to version large model and data set files more efficiently. Content-based deduplication is meant to transfer changed blocks instead of complete files. Xet works with block-based, content-addressed storage.

Ideas

  • ML versioning needs deduplication below the level of huge monolithic files.
  • Content addressing can share storage and transfer across many similar model versions.

Insights

  • ML storage has to move large, similar artefacts efficiently without giving up reproducibility.

Facts

  • The technology is being integrated into the infrastructure of the Hugging Face Hub.

Critique

  • A new storage layer increases protocol and vendor dependency as long as alternative tools are missing.

Recommendations

  • Measure upload, checkout, cache requirements and recovery with your real model revisions.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2025

Uploads and downloads from chunk to block

The article explains the conversion of large Hub transfers to block-based Xet storage. Deduplication and parallel transfer are meant to handle changes to large files efficiently. The new architecture splits files into smaller content-addressed blocks.

HF · 2026

Mutable storage buckets in the Hub

Storage buckets add S3-like object storage for checkpoints, logs and intermediate states to the version-oriented repositories. The data remains accessible via browser, script and command line. Buckets are S3-like and are backed by the Xet storage layer.

HF · 2025

Xet has arrived on the Hugging Face Hub

The Hub activates Xet as the storage layer for model and data set artefacts. Users keep Git-like workflows, while transfer and deduplication happen at block level. Xet is gradually replacing Git LFS as the storage path for large Hub files.