bk99.de entertain the web since 1997

bk99.de
HF · DataCurated

Content-defined blocks for Parquet

Summary

The article adapts content-defined chunking to the internal structure of Parquet files. Similar table versions can therefore share blocks, even though file offsets have changed. Parquet organises data in column chunks and row groups.

Ideas

  • Knowledge of the file format improves deduplication compared with blindly chosen byte boundaries.
  • Stable block boundaries make incremental data versions cheaper to store and transfer.

Insights

  • Data access becomes scalable when formats, metadata and query patterns are designed together.

Facts

  • Content-defined chunking sets boundaries based on content rather than fixed positions.

Critique

  • Optimisation close to the format has to keep pace with new Parquet variants and compression methods.

Recommendations

  • Measure the deduplication rate and CPU cost with typical changes to your data sets.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2025

Streaming data sets a hundred times more efficiently

The new streaming architecture reads only the required ranges of remote data and plans access via Parquet metadata. This reduces requests and data transfer for large data sets. For selected workloads, the article reports up to a hundredfold increase in efficiency.

HF · 2023

Analysing more than 50,000 Hub data sets with DuckDB

The article uses Parquet metadata and DuckDB to evaluate many Hub data sets directly with SQL. Large data sets do not have to be loaded completely onto your own machine for this. DuckDB can query Parquet files directly over HTTP.

HF · 2026

Security incident at Hugging Face in July 2026

Hugging Face describes a break-in in which an autonomous agent combined a prepared data source with insecure execution paths. Internal data sets and credentials were reached; according to what was known at the time, public Hub artefacts were not manipulated. The attack combined a malicious data set with remote code execution and template injection.