LLM Observability for SREs: What to Measure Beyond HTTP 200

A healthy endpoint can still be a broken AI product

Traditional service telemetry remains necessary, but AI introduces another dimension: the request can succeed technically while the answer is irrelevant, unsafe or expensive. Production visibility therefore has to combine system telemetry with model/application quality signals.

Golden signals still matter

• Latency — including Time to First Token (TTFT) and Time per Output Token (TPOT).
• Traffic — requests, tokens and concurrent sequences.
• Errors — model/provider errors, timeouts, tool failures and validation failures.
• Saturation — GPU memory, queue depth, rate limits and provider quotas.

Trace the AI application, not only the HTTP request

An agentic or RAG request can involve retrieval, an embedding service, one or more model calls and several tools. Use Trace and Span concepts to preserve the full path. A single user request should be explainable from entry to final answer.

Add quality and evaluation signals

Evaluation (Eval) gives you repeatable checks for task quality. Depending on the use case, track retrieval relevance, groundedness, answer correctness, refusal behavior, tool-call success and Hallucination Rate. Quality metrics should be interpreted alongside latency and cost, not in isolation.

Cost is an operational signal

Token usage, model choice and retry/agent-loop behavior can change spend quickly. Measure Token Usage, Cost per Task and provider/model attribution so cost regressions appear like any other production regression.