bk99.de entertain the web since 1997

bk99.de
HF · Language modelsCurated

Training a language model from scratch

Summary

Using Esperanto as an example, Hugging Face shows how the tokenizer, data set and transformer are built up from scratch together. The experiment makes visible which decisions finished models usually hide. The example model has six transformer layers and around 84 million parameters.

Ideas

  • A tokenizer of your own prevents a target language from being processed with unsuitable word boundaries.
  • Training from scratch separates architectural knowledge from merely reusing finished weights.

Insights

  • A base model of your own is only worthwhile when data or language measurably overwhelm existing models.

Facts

  • The trained encoder is then fine-tuned for part-of-speech tagging.

Critique

  • The small Esperanto example does not yet prove superiority over strong multilingual base models.

Recommendations

  • First check whether an existing multilingual model serves the purpose before you train a new one.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2021

Training CodeParrot from scratch

CodeParrot documents how a GPT-2-like code generator was built, from data collection to training. The article also shows how licence filtering and duplicates shape the data. The model was trained on publicly available Python code from GitHub.

HF · 2025

Continuous batching explained from scratch

The article builds a scheduler that adds new requests to running batches between individual decoding steps. This keeps accelerators better utilised despite answers of different lengths. Continuous batching schedules requests at token level instead of per fully completed batch.

HF · 2024

Fine-tuning language models to 1.58 bits

The article applies ternary weights to existing language models and shows an accessible fine-tuning path. Extreme quantisation promises small models but requires careful quality measurement. 1.58 bits correspond to the information content of three possible weight values.