bk99.de entertain the web since 1997

bk99.de
HF · InferenceCurated

Loading very large models with Accelerate

Summary

Accelerate distributes model parts across GPU, CPU and disk instead of first building the whole model in memory. Empty initialisation and automatic device maps avoid unnecessary peaks. init_empty_weights first creates the model structure without parameter data.

Ideas

  • Memory peaks during loading can be larger than the later inference requirement.
  • Hierarchical offloading trades capacity for transfer time and complexity.

Facts

  • device_map can distribute layers across GPU, CPU and disk.

Critique

  • A model that loads technically can become too slow for interactive use because of offloading.

Recommendations

  • Plan the memory budget and transfer paths per layer and measure the resulting token latency.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2024

Serving thirty LoRA models with one server

Text Generation Inference loads many LoRA adapters on top of a shared base model and serves them together. This avoids complete model copies and improves utilisation. The demonstration serves 30 LoRA models through one TGI instance.

HF · 2022

Accelerating transformers with OpenVINO

Optimum Intel exports transformers for OpenVINO and runs them optimised on Intel hardware. The article combines model conversion, quantisation and pipeline use. OpenVINO optimises neural networks for Intel processors and other Intel hardware.

HF · 2022

Accelerated pipelines with Optimum

Optimum integrates optimised runtimes into familiar Transformers pipelines. This allows models to be accelerated with ONNX Runtime without rewriting the whole application. Optimum offers ORTModel classes for ONNX Runtime.