bk99.de entertain the web since 1997

bk99.de
HF · DataCurated

Streaming data sets a hundred times more efficiently

Summary

The new streaming architecture reads only the required ranges of remote data and plans access via Parquet metadata. This reduces requests and data transfer for large data sets. For selected workloads, the article reports up to a hundredfold increase in efficiency.

Ideas

  • Efficient streaming needs knowledge of which byte ranges a query actually requires.
  • Metadata planning can reduce network traffic more than merely larger download buffers.

Insights

  • Data access becomes scalable when formats, metadata and query patterns are designed together.

Facts

  • The Parquet structure and range requests limit the data transferred.

Critique

  • The peak figure depends on the data set and access pattern and does not apply to every format.

Recommendations

  • Measure requests, transferred bytes and cache hits under your typical iteration order.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2023

Analysing more than 50,000 Hub data sets with DuckDB

The article uses Parquet metadata and DuckDB to evaluate many Hub data sets directly with SQL. Large data sets do not have to be loaded completely onto your own machine for this. DuckDB can query Parquet files directly over HTTP.

HF · 2025

Content-defined blocks for Parquet

The article adapts content-defined chunking to the internal structure of Parquet files. Similar table versions can therefore share blocks, even though file offsets have changed. Parquet organises data in column chunks and row groups.

HF · 2025

Training static embeddings four hundred times faster

The article distils high-quality sentence representations into very small static embedding tables. This produces extremely fast baselines for search and classification without transformer inference. Training static embeddings is described with up to a 400-fold speed-up.