Mutable storage buckets in the Hub
Summary
Storage buckets add S3-like object storage for checkpoints, logs and intermediate states to the version-oriented repositories. The data remains accessible via browser, script and command line. Buckets are S3-like and are backed by the Xet storage layer.
Ideas
- Not every ML artefact fits into immutable repository versions ready for publication.
- Mutable object storage complements reproducible releases but must not replace them.
Insights
- ML storage has to move large, similar artefacts efficiently without giving up reproducibility.
Facts
- They are intended for checkpoints, optimiser states, logs and traces.
Critique
- Mutable objects make reproducibility harder if builds do not reference exact contents.
Recommendations
- Define retention, versioning and the handover of bucket artefacts into permanent releases.
References
Read the original article on Hugging Face
Links to the original source and the Web Archive open in a new tab.