bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

How BLOOM inference was optimised

Summary

The report follows the optimisation of a 176-billion-parameter model from memory planning to distributed execution. It shows that production inference is a systems problem of parallelisation, kernels and communication. BLOOM is too large for the memory of a single common GPU.

Ideas

  • With very large models, the distribution of weights and cache determines the possible parallelism.
  • Optimisation needs reliable profiles instead of guesses about the most expensive part.

Facts

  • The solution distributes the model computation across several accelerators.

Critique

  • The architecture shown is optimised for BLOOM and the hardware used, not universally.

Recommendations

  • Profile preprocessing, prefill, decoding, communication and memory separately.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2026

Asynchrony in continuous batching

The article extends continuous batching with asynchronous preparation and output so that CPU work blocks the GPU scheduler less. Overlapping pipeline phases increase utilisation with many simultaneous requests. Tokenisation, scheduling and output can be overlapped with GPU computation.

HF · 2025

Continuous batching explained from scratch

The article builds a scheduler that adds new requests to running batches between individual decoding steps. This keeps accelerators better utilised despite answers of different lengths. Continuous batching schedules requests at token level instead of per fully completed batch.

HF · 2025

Lessons from gpt-oss for faster Transformers

The article transfers optimisations from OpenAI’s open gpt-oss models to the Transformers runtime. The focus is on attention, caches and kernels that also benefit other models. The work concerns support for the open gpt-oss model family.