bk99.de entertain the web since 1997

bk99.de
HF · Text generationCurated

How decoding changes the text output

Summary

The article compares greedy search, beam search, sampling, top-k and nucleus sampling. It shows that the same model weights produce completely different texts depending on the decoding strategy. Beam search keeps several continuations in parallel and compares their overall probability.

Ideas

  • Decoding is a design parameter of its own and not just the last technical step.
  • The most probable token at each step does not automatically produce the most convincing overall text.

Insights

  • Model outputs arise from weights and decoding together; both belong in every quality assessment.

Facts

  • Top-p sampling dynamically restricts the choice to a share of the probability mass.

Critique

  • Examples can show differences in style, but do not replace a systematic assessment of truthfulness and safety.

Recommendations

  • Evaluate several decoding methods with real prompts and a task-specific quality metric.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2024

Faster output through self-speculative decoding

LayerSkip uses early layers of the same model as a fast draft and checks the proposals with the complete model. This makes a separate draft model unnecessary. Self-speculative decoding uses the same model for drafting and verification.

HF · 2023

The open ecosystem for text generation

The overview classifies open language models, training libraries, inference servers and evaluation tools around the Hub. It makes clear that open LLMs need a chain of compatible components. Models and their associated artefacts are versioned via the Hub.

HF · 2024

Longer generation through a quantised KV cache

The article quantises the key-value cache, which takes up a growing share of GPU memory with long contexts. This leaves more memory free for longer sequences or larger batches. The KV cache grows with sequence length, number of layers and batch size.