bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Asynchrony in continuous batching

Summary

The article extends continuous batching with asynchronous preparation and output so that CPU work blocks the GPU scheduler less. Overlapping pipeline phases increase utilisation with many simultaneous requests. Tokenisation, scheduling and output can be overlapped with GPU computation.

Ideas

  • Accelerators often wait for CPU and network phases rather than for additional model computation.
  • Asynchrony increases throughput but also the number of possible ordering and failure states.

Facts

  • Tokenisation, scheduling and output can be overlapped with GPU computation.
  • Continuous batching continuously adds requests to active batches.

Critique

  • More parallelism makes reproducibility, backpressure and clean abort handling harder.

Recommendations

  • Use tracing across queue, CPU, GPU and streaming to find hidden waiting times.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2025

Continuous batching explained from scratch

The article builds a scheduler that adds new requests to running batches between individual decoding steps. This keeps accelerators better utilised despite answers of different lengths. Continuous batching schedules requests at token level instead of per fully completed batch.

HF · 2025

Lessons from gpt-oss for faster Transformers

The article transfers optimisations from OpenAI’s open gpt-oss models to the Transformers runtime. The focus is on attention, caches and kernels that also benefit other models. The work concerns support for the open gpt-oss model family.

HF · 2024

Faster output through self-speculative decoding

LayerSkip uses early layers of the same model as a fast draft and checks the proposals with the complete model. This makes a separate draft model unnecessary. Self-speculative decoding uses the same model for drafting and verification.