bk99.de entertain the web since 1997

bk99.de
HF · QuantisationCurated

An overview of quantisation methods in Transformers

Summary

The article compares integrated methods such as bitsandbytes, GPTQ and AWQ in terms of memory, hardware and deployment path. It helps to understand quantisation as a trade-off rather than a single switch. Transformers supports several quantisation backends through their own configurations.

Ideas

  • Quantisation methods differ in calibration, bit width, kernel support and changeability.
  • The smallest model format is not automatically the fastest on a given hardware.

Insights

  • Lower precision is a system decision between quality, memory, kernels and target hardware.

Facts

  • Some methods target inference, others also allow adapter training.

Critique

  • Comparison figures age quickly because kernels and hardware support change constantly.

Recommendations

  • Create a decision matrix from quality, memory, latency, platform and maintainability.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2022

8-bit matrix multiplication for large transformers

The bitsandbytes integration loads large models with 8-bit weights and handles sensitive outliers separately. This lowers memory requirements while important numerical parts keep higher precision. LLM.int8 combines 8-bit matrix multiplication with higher-precision handling of outliers.

HF · 2025

Quantisation backends for diffusion models

The article compares several quantisation routes for Diffusers and their effect on memory, speed and image quality. Because of their various components, diffusion pipelines have different requirements from pure LLMs. Diffusers supports several backends via quantised model components.

HF · 2024

Fine-tuning language models to 1.58 bits

The article applies ternary weights to existing language models and shows an accessible fine-tuning path. Extreme quantisation promises small models but requires careful quality measurement. 1.58 bits correspond to the information content of three possible weight values.