bk99.de entertain the web since 1997

bk99.de
HF · ArchitecturesCurated

Understanding transformers as encoder-decoders

Summary

The article takes apart encoder-decoder transformers and explains how they differ from pure encoder or decoder models. Cross-attention in particular connects the representation of the input with the step-by-step output. The encoder processes the input sequence bidirectionally.

Ideas

  • Encoder and decoder solve different parts of a sequence transformation.
  • Cross-attention makes the encoded input specifically usable at every output step.

Insights

  • Architecture decisions distribute computing costs and information paths; they are not mere implementation details.

Facts

  • The decoder masks future output tokens and takes the encoder states into account.

Critique

  • The conceptual explanation says little about the training costs and failure patterns of specific models.

Recommendations

  • Choose the architecture by task and data flow rather than by a model’s mere popularity.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2021

Transformers for long documents

The literature review classifies methods that avoid the quadratic cost of full attention. Local windows, sparsity and compressed representations trade a global view for scalability. Full self-attention grows quadratically with the sequence length.

HF · 2020

Reformer for long sequences

Reformer reduces the memory and computing requirements of classic transformers through locality-sensitive hashing and reversible layers. This makes much longer inputs possible on limited hardware. Reformer replaces full attention with locality-sensitive hashing.

HF · 2025

Transformers v5 simplifies model definitions

Transformers v5 aligns model definitions more closely with readable PyTorch code and reusable components. New architectures are meant to be integrated faster without making the library incomprehensible. Version 5 revises central patterns for defining models.