TRANSFORMERS & LLMs • 47 / 397
Follow text from characters to tokens, attention, transformers and generation.
Attention Head
An attention head is one learned attention mechanism inside a transformer layer.
Think of it like
Think of Attention Head as part of a very fast reader that repeatedly decides what information matters and what should come next.
Real life
Autocomplete, translation and chat assistants depend on concepts such as Attention Head.
SRE lens
Different heads can capture different useful relationships in logs or language.
Remember thisAn attention head is one learned attention mechanism inside a transformer layer.