bk99.de entertain the web since 1997

bk99.de
HF · ArchitecturesCurated

Reformer for long sequences

Summary

Reformer reduces the memory and computing requirements of classic transformers through locality-sensitive hashing and reversible layers. This makes much longer inputs possible on limited hardware. Reformer replaces full attention with locality-sensitive hashing.

Ideas

  • Approximation can push quadratic attention down to manageable costs.
  • Reversible layers save activation memory by reconstructing intermediate states during the backward pass.

Insights

  • Architecture decisions distribute computing costs and information paths; they are not mere implementation details.

Facts

  • The architecture combines LSH attention with reversible residual layers.

Critique

  • Efficiency gains depend heavily on sequence length, implementation and hardware utilisation.

Recommendations

  • Measure quality loss and runtime with your actual sequence length rather than only with model size.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2021

Transformers for long documents

The literature review classifies methods that avoid the quadratic cost of full attention. Local windows, sparsity and compressed representations trade a global view for scalability. Full self-attention grows quadratically with the sequence length.

HF · 2020

Understanding transformers as encoder-decoders

The article takes apart encoder-decoder transformers and explains how they differ from pure encoder or decoder models. Cross-attention in particular connects the representation of the input with the step-by-step output. The encoder processes the input sequence bidirectionally.

HF · 2022

Transcribing long audio files with Wav2Vec2

The article splits long audio files into overlapping windows and then puts the recognition back together. This allows a model with a limited input window to process continuous recordings. With long input, Wav2Vec2 works with limited excerpts.