The baseline architecture
A single Kubernetes Deployment behind a ClusterIP Service, fronted by an Ingress. Works fine up to ~50 RPS for a small model. Falls apart when traffic spikes, when GPU pods take 3 minutes to schedule, or when the model server has a 2-second cold-start.
Autoscaling with KEDA
HPA (Horizontal Pod Autoscaler) scales on CPU and memory. LLM inference is GPU-bound and queue-depth-bound, and neither maps to CPU utilization well.
KEDA (Kubernetes Event-Driven Autoscaling) scales on arbitrary metrics: queue depth, Pub/Sub lag, Redis list length. We publish inference request counts to a Redis stream; KEDA scales the model server pods when the stream depth exceeds a threshold. Scaling-up latency drops from minutes (cluster autoscaler cold start) to seconds (replica scale-up from 1 to N).
GPU sharing with time-slicing
For models that fit in 4 to 8 GB VRAM, full GPU dedication is wasteful. NVIDIA's time-slicing MIG (Multi-Instance GPU) lets multiple pods share one A100, each getting a guaranteed slice.
Configure nvidia.com/gpu: 1 and set the time-slice profile to 1g.10gb. A single A100 80GB can serve 8 concurrent model instances at 10 GB each, 8 times the throughput per GPU.
Request hedging for tail latency
p50 latency is 12ms. p99 is 280ms. The tail is dominated by KV-cache misses and occasional GC pauses. Hedged requests: after 40ms, send a duplicate request to a second replica. Take whichever response arrives first; cancel the other. This cuts p99 from 280ms to ~45ms with only ~15% increase in total compute.