bk99.de entertain the web since 1997

bk99.de
HF · ArchitecturesCurated

Transformers for long documents

Summary

The literature review classifies methods that avoid the quadratic cost of full attention. Local windows, sparsity and compressed representations trade a global view for scalability. Full self-attention grows quadratically with the sequence length.

Ideas

  • Long contexts need a defined information route between distant tokens.
  • Every attention approximation determines which relationships a model can learn easily.

Insights

  • Architecture decisions distribute computing costs and information paths; they are not mere implementation details.

Facts

  • Longformer and BigBird use sparse attention patterns for longer inputs.

Critique

  • Maximum context length alone measures neither usable memory nor answer quality.

Recommendations

  • For long documents, check whether the information you are looking for can reach the chosen attention structure.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2020

Understanding transformers as encoder-decoders

The article takes apart encoder-decoder transformers and explains how they differ from pure encoder or decoder models. Cross-attention in particular connects the representation of the input with the step-by-step output. The encoder processes the input sequence bidirectionally.

HF · 2020

Reformer for long sequences

Reformer reduces the memory and computing requirements of classic transformers through locality-sensitive hashing and reversible layers. This makes much longer inputs possible on limited hardware. Reformer replaces full attention with locality-sensitive hashing.

HF · 2025

Transformers v5 simplifies model definitions

Transformers v5 aligns model definitions more closely with readable PyTorch code and reusable components. New architectures are meant to be integrated faster without making the library incomprehensible. Version 5 revises central patterns for defining models.