bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Accelerating transformer inference a hundredfold

Summary

The article describes an inference platform that combines batching, model optimisation and specialised hardware. What matters is not a single trick but tuning the entire processing chain. For certain workloads, Hugging Face reports up to a hundredfold acceleration.

Ideas

  • Dynamic batching turns many small requests into better utilised computation.
  • Optimisation has to consider tokenisation, model execution and data transfer together.

Facts

  • The platform uses ONNX Runtime and hardware accelerators for optimised execution.

Critique

  • Vendor figures cannot be generalised without identical models, hardware and load profiles.

Recommendations

  • Measure p50 and p99 latency together with throughput and cost under realistic parallelism.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2022

Accelerating transformers with OpenVINO

Optimum Intel exports transformers for OpenVINO and runs them optimised on Intel hardware. The article combines model conversion, quantisation and pipeline use. OpenVINO optimises neural networks for Intel processors and other Intel hardware.

HF · 2026

Asynchrony in continuous batching

The article extends continuous batching with asynchronous preparation and output so that CPU work blocks the GPU scheduler less. Overlapping pipeline phases increase utilisation with many simultaneous requests. Tokenisation, scheduling and output can be overlapped with GPU computation.

HF · 2025

Continuous batching explained from scratch

The article builds a scheduler that adds new requests to running batches between individual decoding steps. This keeps accelerators better utilised despite answers of different lengths. Continuous batching schedules requests at token level instead of per fully completed batch.