TRANSFORMERS & LLMs • 41 / 397
Follow text from characters to tokens, attention, transformers and generation.
Token ID
A token ID is the numeric identifier used internally for a token.
Think of it like
Think of Token ID as part of a very fast reader that repeatedly decides what information matters and what should come next.
Real life
Autocomplete, translation and chat assistants depend on concepts such as Token ID.
SRE lens
The model receives token IDs, not raw English words.
Remember thisA token ID is the numeric identifier used internally for a token.