Batching can raise throughput but may also add waiting time and worsen latency if tuned badly.
LATENCY & PERFORMANCEAI SRE
Batching
Batching processes multiple inference requests together to improve hardware utilization.
Increasing batch size raises tokens/sec by 25% but adds 400ms to p95 TTFT for interactive users.
What throughput gain is worth the additional queueing latency for this workload?
REMEMBERBatching trades some immediacy for efficiency.
No uploads · No company data · No account required