Large prompts primarily hurt prefill and TTFT, while long outputs primarily stress decode capacity.
LATENCY & PERFORMANCEAI SRE
Prefill vs Decode
Prefill processes the input context; decode generates output tokens one step at a time.
A 60k-token context causes high TTFT even though the generated answer is only 200 tokens.
Is the workload prompt-heavy, output-heavy or mixed, and does capacity planning reflect that?
REMEMBERInput size and output size stress different parts of inference.
No uploads · No company data · No account required