bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Accelerating transformers with OpenVINO

Summary

Optimum Intel exports transformers for OpenVINO and runs them optimised on Intel hardware. The article combines model conversion, quantisation and pipeline use. OpenVINO optimises neural networks for Intel processors and other Intel hardware.

Ideas

  • Hardware acceleration can come from the compiler and runtime instead of model changes.
  • A standardised export path makes CPU optimisation more repeatable.

Facts

  • Optimum Intel provides suitable model classes and export tools.

Critique

  • Vendor-specific optimisation can limit portability and create new version dependencies.

Recommendations

  • Validate accuracy and latency on exactly the target CPU with representative inputs.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2025

Lessons from gpt-oss for faster Transformers

The article transfers optimisations from OpenAI’s open gpt-oss models to the Transformers runtime. The focus is on attention, caches and kernels that also benefit other models. The work concerns support for the open gpt-oss model family.

HF · 2024

Serving thirty LoRA models with one server

Text Generation Inference loads many LoRA adapters on top of a shared base model and serves them together. This avoids complete model copies and improves utilisation. The demonstration serves 30 LoRA models through one TGI instance.

HF · 2022

Loading very large models with Accelerate

Accelerate distributes model parts across GPU, CPU and disk instead of first building the whole model in memory. Empty initialisation and automatic device maps avoid unnecessary peaks. init_empty_weights first creates the model structure without parameter data.