bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Serving thirty LoRA models with one server

Summary

Text Generation Inference loads many LoRA adapters on top of a shared base model and serves them together. This avoids complete model copies and improves utilisation. The demonstration serves 30 LoRA models through one TGI instance.

Ideas

  • Adapter multi-tenancy shares expensive base weights between specialised services.
  • Schedulers have to batch different adapters together without unfair latency spikes.

Facts

  • All adapters use the same loaded base model.

Critique

  • The number of possible adapters depends on size, traffic pattern and the shared base model.

Recommendations

  • Set memory limits per adapter and test fairness and tail latency under mixed load.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2022

Loading very large models with Accelerate

Accelerate distributes model parts across GPU, CPU and disk instead of first building the whole model in memory. Empty initialisation and automatic device maps avoid unnecessary peaks. init_empty_weights first creates the model structure without parameter data.

HF · 2022

Accelerating transformers with OpenVINO

Optimum Intel exports transformers for OpenVINO and runs them optimised on Intel hardware. The article combines model conversion, quantisation and pipeline use. OpenVINO optimises neural networks for Intel processors and other Intel hardware.

HF · 2022

Accelerated pipelines with Optimum

Optimum integrates optimised runtimes into familiar Transformers pipelines. This allows models to be accelerated with ONNX Runtime without rewriting the whole application. Optimum offers ORTModel classes for ONNX Runtime.