bk99.de entertain the web since 1997

bk99.de
HF · Open modelsCurated

BLOOM: an open multilingual large model

Summary

BLOOM was created as an international community project for open access to a very large language model. The article describes the languages, computing effort and the joint organisation behind the training. BLOOM has 176 billion parameters.

Ideas

  • Large models can be organised as scientific infrastructure rather than as a closed company product.
  • Open weights extend verifiability but shift responsibility for safe use to the users.

Insights

  • Open weights shift control to users and in return require more testing competence of their own.

Facts

  • More than 1,000 researchers from over 70 countries and 250 institutions took part.

Critique

  • Open weights remove neither the biases in the training corpus nor the high operating costs.

Recommendations

  • Read the model card and licence and test the languages you actually need before using it.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2023

Mixtral and mixture of experts

Mixtral activates only part of its experts per token and combines high model capacity with limited computation. The article explains the architecture, usage and open availability. Mixtral uses eight experts and activates two of them per token.

HF · 2023

The open ecosystem for text generation

The overview classifies open language models, training libraries, inference servers and evaluation tools around the Hub. It makes clear that open LLMs need a chain of compatible components. Models and their associated artefacts are versioned via the Hub.

HF · 2022

Loading very large models with Accelerate

Accelerate distributes model parts across GPU, CPU and disk instead of first building the whole model in memory. Empty initialisation and automatic device maps avoid unnecessary peaks. init_empty_weights first creates the model structure without parameter data.