LLMOPS & AI INFRASTRUCTURE • 290 / 397
Operate AI like production: serving, GPUs, latency, capacity, routing and reliability.
Latency
Latency is the elapsed time a request takes from start to finish or across a specific stage.
Think of it like
Think of Latency like familiar infrastructure capacity and traffic engineering, except the scarce resources are often tokens, GPU memory and model latency.
Real life
High-scale AI services depend on Latency to keep cost and latency under control.
SRE lens
Track end-to-end and per-component latency separately.
Remember thisLatency is the elapsed time a request takes from start to finish or across a specific stage.