TRANSFORMERS & LLMs • 38 / 397
Follow text from characters to tokens, attention, transformers and generation.
Token
A token is a unit of text processed by a language model; it may be a whole word, part of a word, punctuation or another text fragment.
Think of it like
Think of Token as part of a very fast reader that repeatedly decides what information matters and what should come next.
Real life
Autocomplete, translation and chat assistants depend on concepts such as Token.
SRE lens
Token counts affect context limits, latency and API cost.
Remember thisA token is a unit of text processed by a language model; it may be a whole word, part of a word, punctuation or another text fragment.