bk99.de entertain the web since 1997

bk99.de
HF · StorageCurated

Xet has arrived on the Hugging Face Hub

Summary

The Hub activates Xet as the storage layer for model and data set artefacts. Users keep Git-like workflows, while transfer and deduplication happen at block level. Xet is gradually replacing Git LFS as the storage path for large Hub files.

Ideas

  • Infrastructure changes are accepted more easily when existing workflows are preserved.
  • Shared blocks reduce redundant copies of similar model revisions.

Insights

  • ML storage has to move large, similar artefacts efficiently without giving up reproducibility.

Facts

  • The hf_xet extension speeds up uploads and downloads for compatible clients.

Critique

  • Performance and availability now also depend on a more complex storage layer close to being proprietary.

Recommendations

  • Document client requirements and keep a tested fallback ready for automated builds.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2026

Mutable storage buckets in the Hub

Storage buckets add S3-like object storage for checkpoints, logs and intermediate states to the version-oriented repositories. The data remains accessible via browser, script and command line. Buckets are S3-like and are backed by the Xet storage layer.

HF · 2025

Uploads and downloads from chunk to block

The article explains the conversion of large Hub transfers to block-based Xet storage. Deduplication and parallel transfer are meant to handle changes to large files efficiently. The new architecture splits files into smaller content-addressed blocks.

HF · 2024

XetHub brings block-based storage to the Hub

XetHub joins Hugging Face to version large model and data set files more efficiently. Content-based deduplication is meant to transfer changed blocks instead of complete files. Xet works with block-based, content-addressed storage.