bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Longer generation through a quantised KV cache

Summary

The article quantises the key-value cache, which takes up a growing share of GPU memory with long contexts. This leaves more memory free for longer sequences or larger batches. The KV cache grows with sequence length, number of layers and batch size.

Ideas

  • In autoregressive generation, the state cache can become more important than the model weights.
  • Cache quantisation trades accuracy of memory for context capacity.

Facts

  • Transformers supports quantised cache implementations for selected backends.

Critique

  • Additional quantisation steps can slow down short requests and introduce subtle errors.

Recommendations

  • Test long contexts for quality loss and measure from which length quantisation actually helps.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2024

Faster output through self-speculative decoding

LayerSkip uses early layers of the same model as a fast draft and checks the proposals with the complete model. This makes a separate draft model unnecessary. Self-speculative decoding uses the same model for drafting and verification.

HF · 2026

Asynchrony in continuous batching

The article extends continuous batching with asynchronous preparation and output so that CPU work blocks the GPU scheduler less. Overlapping pipeline phases increase utilisation with many simultaneous requests. Tokenisation, scheduling and output can be overlapped with GPU computation.

HF · 2025

Continuous batching explained from scratch

The article builds a scheduler that adds new requests to running batches between individual decoding steps. This keeps accelerators better utilised despite answers of different lengths. Continuous batching schedules requests at token level instead of per fully completed batch.