Large model weights make cold starts much more expensive than ordinary application process starts.
LATENCY & PERFORMANCEAI SRE
Model Cold Starts
A cold start is the delay introduced when a model or inference worker must load, initialize or warm before serving traffic.
Autoscaling adds a GPU node, but the model takes 90 seconds to load, so the new capacity arrives after the traffic spike has passed.
Which models require warm pools, preloading or minimum replicas?
REMEMBERAI autoscaling must include model-load time.
No uploads · No company data · No account required