bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Lessons from gpt-oss for faster Transformers

Summary

The article transfers optimisations from OpenAI’s open gpt-oss models to the Transformers runtime. The focus is on attention, caches and kernels that also benefit other models. The work concerns support for the open gpt-oss model family.

Ideas

  • A specific model can trigger optimisations that then act as general infrastructure.
  • Architectural knowledge and runtime profiling have to come together to remove real bottlenecks.

Facts

  • The work concerns support for the open gpt-oss model family.
  • Optimisations are fed back into general Transformers components.

Critique

  • Model-specific tricks can make general code paths more complicated and harder to maintain.

Recommendations

  • After runtime updates, check speed and output quality again with pinned model revisions.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2025

Continuous batching explained from scratch

The article builds a scheduler that adds new requests to running batches between individual decoding steps. This keeps accelerators better utilised despite answers of different lengths. Continuous batching schedules requests at token level instead of per fully completed batch.

HF · 2024

Faster output through self-speculative decoding

LayerSkip uses early layers of the same model as a fast draft and checks the proposals with the complete model. This makes a separate draft model unnecessary. Self-speculative decoding uses the same model for drafting and verification.

HF · 2022

Accelerating transformers with OpenVINO

Optimum Intel exports transformers for OpenVINO and runs them optimised on Intel hardware. The article combines model conversion, quantisation and pipeline use. OpenVINO optimises neural networks for Intel processors and other Intel hardware.