TRANSFORMERS & LLMs • 46 / 397
Follow text from characters to tokens, attention, transformers and generation.
Self-Attention
Self-attention is attention where tokens in one sequence compare with other tokens in that same sequence.
Think of it like
Think of Self-Attention as part of a very fast reader that repeatedly decides what information matters and what should come next.
Real life
Autocomplete, translation and chat assistants depend on concepts such as Self-Attention.
SRE lens
A model can relate an error at the end of a log summary to a service named near the beginning.
Remember thisSelf-attention is attention where tokens in one sequence compare with other tokens in that same sequence.