TRANSFORMERS & LLMs • 44 / 397
Follow text from characters to tokens, attention, transformers and generation.
Positional Information
Positional information tells a transformer where tokens occur in a sequence so word order can matter.
Think of it like
Think of Positional Information as part of a very fast reader that repeatedly decides what information matters and what should come next.
Real life
Autocomplete, translation and chat assistants depend on concepts such as Positional Information.
SRE lens
“Restart after backup” and “backup after restart” contain similar words but different order.
Remember thisPositional information tells a transformer where tokens occur in a sequence so word order can matter.