bk99.de entertain the web since 1997

bk99.de
HF · QuantisationCurated

8-bit matrix multiplication for large transformers

Summary

The bitsandbytes integration loads large models with 8-bit weights and handles sensitive outliers separately. This lowers memory requirements while important numerical parts keep higher precision. LLM.int8 combines 8-bit matrix multiplication with higher-precision handling of outliers.

Ideas

  • Mixed precision works better when only the outliers, rather than all values, are processed expensively.
  • Quantisation shifts the limit of what can be used experimentally on existing hardware.

Insights

  • Lower precision is a system decision between quality, memory, kernels and target hardware.

Facts

  • Transformers and Accelerate integrate the loading of quantised models.

Critique

  • The benefits depend on supported GPUs, operators and the model’s distribution of outliers.

Recommendations

  • Compare memory, throughput and task quality against a higher-precision reference.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2023

An overview of quantisation methods in Transformers

The article compares integrated methods such as bitsandbytes, GPTQ and AWQ in terms of memory, hardware and deployment path. It helps to understand quantisation as a trade-off rather than a single switch. Transformers supports several quantisation backends through their own configurations.

HF · 2025

Quantisation backends for diffusion models

The article compares several quantisation routes for Diffusers and their effect on memory, speed and image quality. Because of their various components, diffusion pipelines have different requirements from pure LLMs. Diffusers supports several backends via quantised model components.

HF · 2024

Fine-tuning language models to 1.58 bits

The article applies ternary weights to existing language models and shows an accessible fine-tuning path. Extreme quantisation promises small models but requires careful quality measurement. 1.58 bits correspond to the information content of three possible weight values.