bk99.de entertain the web since 1997

bk99.de
HF · DataCurated

Analysing more than 50,000 Hub data sets with DuckDB

Summary

The article uses Parquet metadata and DuckDB to evaluate many Hub data sets directly with SQL. Large data sets do not have to be loaded completely onto your own machine for this. DuckDB can query Parquet files directly over HTTP.

Ideas

  • Columnar formats allow analysis close to the metadata instead of complete downloads.
  • SQL makes data inventories accessible for reproducible quality checks.

Insights

  • Data access becomes scalable when formats, metadata and query patterns are designed together.

Facts

  • The Hub provides automatically generated Parquet versions for numerous data sets.

Critique

  • Automatically converted Parquet files inherit the quality and licence problems of their source data.

Recommendations

  • Start with metadata and sample queries before you take over a large data set completely.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2025

Streaming data sets a hundred times more efficiently

The new streaming architecture reads only the required ranges of remote data and plans access via Parquet metadata. This reduces requests and data transfer for large data sets. For selected workloads, the article reports up to a hundredfold increase in efficiency.

HF · 2025

Content-defined blocks for Parquet

The article adapts content-defined chunking to the internal structure of Parquet files. Similar table versions can therefore share blocks, even though file offsets have changed. Parquet organises data in column chunks and row groups.

HF · 2024

Serving thirty LoRA models with one server

Text Generation Inference loads many LoRA adapters on top of a shared base model and serves them together. This avoids complete model copies and improves utilisation. The demonstration serves 30 LoRA models through one TGI instance.