bk99.de entertain the web since 1997

bk99.de
HF · QuantisationCurated

Quantisation backends for diffusion models

Summary

The article compares several quantisation routes for Diffusers and their effect on memory, speed and image quality. Because of their various components, diffusion pipelines have different requirements from pure LLMs. Diffusers supports several backends via quantised model components.

Ideas

  • Text encoder, transformer and VAE do not necessarily need the same quantisation strategy.
  • Memory savings can be paid for with dequantisation or slow kernels.

Insights

  • Lower precision is a system decision between quality, memory, kernels and target hardware.

Facts

  • The comparisons look at memory consumption, runtime and output quality.

Critique

  • Visual spot checks can easily miss rare quality losses or prompt dependencies.

Recommendations

  • Quantise pipeline components individually and check image artefacts with fixed seeds.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2024

Fine-tuning language models to 1.58 bits

The article applies ternary weights to existing language models and shows an accessible fine-tuning path. Extreme quantisation promises small models but requires careful quality measurement. 1.58 bits correspond to the information content of three possible weight values.

HF · 2024

Quanto as a PyTorch backend for quantisation

Quanto brings weight and activation quantisation into a PyTorch-centred workflow. Models therefore remain easier to examine and do not have to be exported to a foreign runtime straight away. Quanto is a quantisation backend for Hugging Face Optimum.

HF · 2023

An overview of quantisation methods in Transformers

The article compares integrated methods such as bitsandbytes, GPTQ and AWQ in terms of memory, hardware and deployment path. It helps to understand quantisation as a trade-off rather than a single switch. Transformers supports several quantisation backends through their own configurations.