bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Accelerated pipelines with Optimum

Summary

Optimum integrates optimised runtimes into familiar Transformers pipelines. This allows models to be accelerated with ONNX Runtime without rewriting the whole application. Optimum offers ORTModel classes for ONNX Runtime.

Ideas

  • Compatible APIs lower the cost of trying out an optimised runtime in practice.
  • Export and optimisation must remain reproducible as a build step of their own.

Facts

  • Optimised models can be used through the Transformers pipeline interface.

Critique

  • API compatibility guarantees neither identical results nor acceleration for every model.

Recommendations

  • Compare outputs and numerical tolerances before and after export and quantisation.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2024

Serving thirty LoRA models with one server

Text Generation Inference loads many LoRA adapters on top of a shared base model and serves them together. This avoids complete model copies and improves utilisation. The demonstration serves 30 LoRA models through one TGI instance.

HF · 2022

Accelerating transformers with OpenVINO

Optimum Intel exports transformers for OpenVINO and runs them optimised on Intel hardware. The article combines model conversion, quantisation and pipeline use. OpenVINO optimises neural networks for Intel processors and other Intel hardware.

HF · 2022

Loading very large models with Accelerate

Accelerate distributes model parts across GPU, CPU and disk instead of first building the whole model in memory. Empty initialisation and automatic device maps avoid unnecessary peaks. init_empty_weights first creates the model structure without parameter data.