bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Millisecond inference on modern CPUs

Summary

The case study optimises transformer inference on CPUs with ONNX Runtime, quantisation and dynamic batching. It shows when existing server hardware can be a practical alternative to GPUs. The platform runs optimised transformer models on modern CPUs.

Ideas

  • Small models often benefit more from low system latency than from maximum accelerator performance.
  • Quantisation, batching and runtime have to be tuned together for the service goal.

Facts

  • The case study measures latencies in the millisecond range for selected models.

Critique

  • The advertised latency only applies to the models, CPUs and load conditions shown.

Recommendations

  • Test CPU and GPU variants with your real request size, parallelism and power consumption.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2026

Asynchrony in continuous batching

The article extends continuous batching with asynchronous preparation and output so that CPU work blocks the GPU scheduler less. Overlapping pipeline phases increase utilisation with many simultaneous requests. Tokenisation, scheduling and output can be overlapped with GPU computation.

HF · 2025

Continuous batching explained from scratch

The article builds a scheduler that adds new requests to running batches between individual decoding steps. This keeps accelerators better utilised despite answers of different lengths. Continuous batching schedules requests at token level instead of per fully completed batch.

HF · 2025

Lessons from gpt-oss for faster Transformers

The article transfers optimisations from OpenAI’s open gpt-oss models to the Transformers runtime. The focus is on attention, caches and kernels that also benefit other models. The work concerns support for the open gpt-oss model family.