bk99.de entertain the web since 1997

bk99.de
HF · Open modelsCurated

Mixtral and mixture of experts

Summary

Mixtral activates only part of its experts per token and combines high model capacity with limited computation. The article explains the architecture, usage and open availability. Mixtral uses eight experts and activates two of them per token.

Ideas

  • Mixture of experts separates the total number of parameters from the active computing effort.
  • Routing becomes an additional model component with its own load and quality questions.

Insights

  • Open weights shift control to users and in return require more testing competence of their own.

Facts

  • The model weights were made openly available on the Hub.

Critique

  • The inactive experts save computing time, but their weights still have to be stored and moved.

Recommendations

  • Measure memory requirements, active computing costs and routing behaviour instead of only comparing the number of parameters.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2023

The open ecosystem for text generation

The overview classifies open language models, training libraries, inference servers and evaluation tools around the Hub. It makes clear that open LLMs need a chain of compatible components. Models and their associated artefacts are versioned via the Hub.

HF · 2022

BLOOM: an open multilingual large model

BLOOM was created as an international community project for open access to a very large language model. The article describes the languages, computing effort and the joint organisation behind the training. BLOOM has 176 billion parameters.

HF · 2026

Why open AI matters for cybersecurity

The article argues that open models give defenders tools they can inspect and adapt. At the same time, it acknowledges that the same capabilities are available to attackers. Open weights allow local analysis without handing sensitive data to an API provider.