LLMOPS & AI INFRASTRUCTURE • 294 / 397
Operate AI like production: serving, GPUs, latency, capacity, routing and reliability.
Requests per Second (RPS)
RPS measures how many requests the service handles each second.
Think of it like
Think of Requests per Second (RPS) like familiar infrastructure capacity and traffic engineering, except the scarce resources are often tokens, GPU memory and model latency.
Real life
High-scale AI services depend on Requests per Second (RPS) to keep cost and latency under control.
SRE lens
Use it with token throughput because LLM requests vary greatly in size.
Remember thisRPS measures how many requests the service handles each second.