bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Continuous batching explained from scratch

Summary

The article builds a scheduler that adds new requests to running batches between individual decoding steps. This keeps accelerators better utilised despite answers of different lengths. Continuous batching schedules requests at token level instead of per fully completed batch.

Ideas

  • Static batches waste capacity as soon as individual sequences finish earlier.
  • Scheduling affects throughput, tail latency and fairness at the same time.

Facts

  • Continuous batching schedules requests at token level instead of per fully completed batch.
  • Finished sequences free up cache and compute slots immediately.

Critique

  • Maximum throughput can starve individual long or low-priority requests.

Recommendations

  • Simulate short and long requests together and define explicit fairness and abort rules.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2026

Asynchrony in continuous batching

The article extends continuous batching with asynchronous preparation and output so that CPU work blocks the GPU scheduler less. Overlapping pipeline phases increase utilisation with many simultaneous requests. Tokenisation, scheduling and output can be overlapped with GPU computation.

HF · 2025

Lessons from gpt-oss for faster Transformers

The article transfers optimisations from OpenAI’s open gpt-oss models to the Transformers runtime. The focus is on attention, caches and kernels that also benefit other models. The work concerns support for the open gpt-oss model family.

HF · 2024

Faster output through self-speculative decoding

LayerSkip uses early layers of the same model as a fast draft and checks the proposals with the complete model. This makes a separate draft model unnecessary. Self-speculative decoding uses the same model for drafting and verification.