bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Faster output through self-speculative decoding

Summary

LayerSkip uses early layers of the same model as a fast draft and checks the proposals with the complete model. This makes a separate draft model unnecessary. Self-speculative decoding uses the same model for drafting and verification.

Ideas

  • A model can use its own intermediate representations as cheap proposals.
  • The speed-up depends on how often the complete model accepts early tokens.

Facts

  • Training with layer dropout and early exit improves the usefulness of early layers.

Critique

  • The method requires suitably trained models and gains little at a low acceptance rate.

Recommendations

  • Measure the acceptance rate and output identity for different tasks and sequence lengths.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2025

Lessons from gpt-oss for faster Transformers

The article transfers optimisations from OpenAI’s open gpt-oss models to the Transformers runtime. The focus is on attention, caches and kernels that also benefit other models. The work concerns support for the open gpt-oss model family.

HF · 2024

Longer generation through a quantised KV cache

The article quantises the key-value cache, which takes up a growing share of GPU memory with long contexts. This leaves more memory free for longer sequences or larger batches. The KV cache grows with sequence length, number of layers and batch size.

HF · 2026

Asynchrony in continuous batching

The article extends continuous batching with asynchronous preparation and output so that CPU work blocks the GPU scheduler less. Overlapping pipeline phases increase utilisation with many simultaneous requests. Tokenisation, scheduling and output can be overlapped with GPU computation.