bk99.de entertain the web since 1997

bk99.de
HF · Code modelsCurated

Quantising and accelerating StarCoder on Xeon

Summary

The article combines 8-bit and 4-bit quantisation with speculative decoding for StarCoder on Xeon processors. It shows several independent routes to faster code generation. StarCoder is optimised for CPU inference with Optimum Intel.

Ideas

  • Speculative decoding saves time when a small model often proposes usable tokens.
  • Quantisation reduces memory bandwidth but can change the acceptance rate of speculative tokens.

Insights

  • For code generators, data hygiene and execution checks determine the security gain more than model size.

Facts

  • Q8, Q4 and speculative decoding variants are compared.

Critique

  • Acceleration depends heavily on processor, compiler, sequence length and draft model.

Recommendations

  • Measure acceptance rate, quality and end-to-end latency with your typical code prompts.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2021

Training CodeParrot from scratch

CodeParrot documents how a GPT-2-like code generator was built, from data collection to training. The article also shows how licence filtering and duplicates shape the data. The model was trained on publicly available Python code from GitHub.

HF · 2025

Quantisation backends for diffusion models

The article compares several quantisation routes for Diffusers and their effect on memory, speed and image quality. Because of their various components, diffusion pipelines have different requirements from pure LLMs. Diffusers supports several backends via quantised model components.

HF · 2025

An MCP agent in fifty lines of code

Tiny Agents shows a deliberately small agent that discovers and calls tools via the Model Context Protocol. The code makes the planning and execution loop directly comprehensible. The example implementation is about 50 lines of code.