LLMOPS & AI INFRASTRUCTURE • 292 / 397
Operate AI like production: serving, GPUs, latency, capacity, routing and reliability.
Time per Output Token (TPOT)
TPOT is the average time required to generate each output token after generation starts.
Think of it like
Think of Time per Output Token (TPOT) like familiar infrastructure capacity and traffic engineering, except the scarce resources are often tokens, GPU memory and model latency.
Real life
High-scale AI services depend on Time per Output Token (TPOT) to keep cost and latency under control.
SRE lens
It helps separate startup delay from generation speed.
Remember thisTPOT is the average time required to generate each output token after generation starts.