TRANSFORMERS & LLMs • 39 / 397
Follow text from characters to tokens, attention, transformers and generation.
Tokenization
Tokenization is the process of splitting text into tokens before the model processes it.
Think of it like
Think of Tokenization as part of a very fast reader that repeatedly decides what information matters and what should come next.
Real life
Autocomplete, translation and chat assistants depend on concepts such as Tokenization.
SRE lens
A long stack trace may consume more tokens than its word count suggests.
Remember thisTokenization is the process of splitting text into tokens before the model processes it.