LLM Performance Metrics for SREs: TTFT, TPOT, Tokens/sec & Throughput
Understand the production LLM performance metrics that matter to SRE teams, how they differ, and what each tells you about user experience and capacity.
Do not reduce LLM performance to one latency number
Streaming language generation has phases. A user first waits for output to begin, then experiences the speed at which new tokens arrive. Capacity adds another dimension: how many requests or tokens the system can sustain at once.
TTFT — Time to First Token
Time to First Token (TTFT) is the wait from request start until the first generated token becomes available. It includes queueing and prompt processing, so it is often the metric users notice first.
TPOT — Time per Output Token
Time per Output Token (TPOT) describes the time between generated tokens after decoding has started. Lower TPOT usually means output streams more quickly.
Tokens per second
Tokens per Second is an intuitive generation-rate view. Be clear whether you mean per-request output speed or aggregate system token throughput; those answer different questions.
Throughput and concurrency
Throughput measures work completed over time. Serving systems often improve throughput with Batching or Continuous Batching, but maximizing throughput can hurt individual latency if queueing grows.
A useful dashboard
- TTFT p50/p95/p99.
- TPOT or inter-token latency percentiles.
- Input and output token distributions.
- Requests/sec and tokens/sec.
- Active vs queued sequences.
- GPU utilization and memory.
- Errors, rate limits, OOMs and rejected requests.
397 concepts. One concept at a time.
No coding. No videos. Just simple explanations, diagrams, real-life examples and the SRE lens.
Start from AI →More practical guides