bk99.de entertain the web since 1997

bk99.de
HF · QuantisationCurated

Quanto as a PyTorch backend for quantisation

Summary

Quanto brings weight and activation quantisation into a PyTorch-centred workflow. Models therefore remain easier to examine and do not have to be exported to a foreign runtime straight away. Quanto is a quantisation backend for Hugging Face Optimum.

Ideas

  • Quantisation within PyTorch shortens the path between experiment, measurement and adjustment.
  • Weights and activations often need different precision strategies.

Insights

  • Lower precision is a system decision between quality, memory, kernels and target hardware.

Facts

  • It supports several number formats for weights and activations.

Critique

  • Backend support and real acceleration differ depending on device and operator.

Recommendations

  • Record the error and runtime per layer before you settle on a global quantisation configuration.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2025

Quantisation backends for diffusion models

The article compares several quantisation routes for Diffusers and their effect on memory, speed and image quality. Because of their various components, diffusion pipelines have different requirements from pure LLMs. Diffusers supports several backends via quantised model components.

HF · 2024

Fine-tuning language models to 1.58 bits

The article applies ternary weights to existing language models and shows an accessible fine-tuning path. Extreme quantisation promises small models but requires careful quality measurement. 1.58 bits correspond to the information content of three possible weight values.

HF · 2023

An overview of quantisation methods in Transformers

The article compares integrated methods such as bitsandbytes, GPTQ and AWQ in terms of memory, hardware and deployment path. It helps to understand quantisation as a trade-off rather than a single switch. Transformers supports several quantisation backends through their own configurations.