Auto-Scaling GPU Inference Pods in Kubernetes: KEDA, Custom Metrics, and Cost Guards

Your inference deployment scaled to 40 GPU pods during a traffic spike and generated a $12,000 cloud bill in 6 hours. KEDA with a cost circuit breaker would have capped it at $800. Here's the configuration.

The standard Horizontal Pod Autoscaler (HPA) is about as useful for GPU workloads as a chocolate teapot. It scales on CPU and memory, metrics that are utterly meaningless when your Llama 3.1 70B pod is sitting at 5% CPU utilization while its 80GB of VRAM is sweating bullets and the request queue is backed up for miles. You need to scale on what matters: inference queue depth, request latency, or—critically—your cloud provider’s billing API. Let’s build an autoscaler that doesn’t bankrupt you.

The NVIDIA Device Plugin: Your Ticket to GPU Scheduling

Before KEDA can scale anything, Kubernetes needs to know what a GPU is. Out of the box, it doesn’t. A node with four A100s looks like a node with a lot of mysterious PCIe devices. The NVIDIA Device Plugin is the bouncer that manages the velvet rope to your silicon.

It advertises GPU resources (nvidia.com/gpu) to the Kubernetes scheduler. Your pod spec must explicitly request them, or it will be scheduled onto a node and wonder why it can’t find CUDA. Here’s the bare minimum deployment that actually gets a GPU:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llama-inference
  template:
    metadata:
      labels:
        app: llama-inference
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        resources:
          limits:
            nvidia.com/gpu: 1 # This is the magic line
            memory: "24Gi"
          requests:
            nvidia.com/gpu: 1
            memory: "24Gi"
        env:
        - name: OLLAMA_MODELS
          value: "/mnt/nvme/models" # NVMe mount for fast loads
        - name: OLLAMA_GPU_MEMORY_FRACTION
          value: "0.85" # Guard against OOM errors

Install the device plugin via its DaemonSet:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.1/deployments/static/nvidia-device-plugin.yml

Then, verify your nodes see the resource:

kubectl describe node <gpu-node-name> | grep -A 10 -B 5 Capacity

You should see nvidia.com/gpu: 4 (or however many are physically present). If you don’t, you’ll face the dreaded scheduler error: 0/4 nodes are available: 4 Insufficient nvidia.com/gpu. The fix is always to check nvidia-smi on the node itself and ensure the device plugin pod is running (kubectl get pods -n kube-system | grep nvidia).

Installing KEDA and Hooking Into Your Inference Queue

KEDA (Kubernetes Event-driven Autoscaling) is the brains of the operation. It sits between your metric source (like a Prometheus query or a Redis queue length) and the Kubernetes HPA controller, translating application events into scale commands.

Install KEDA with Helm:

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace

Now, the critical part: defining what to scale on. For GPU inference, the most direct metric is often queue depth. Let’s say you’re using Redis to manage an inference job queue. A KEDA ScaledObject tells KEDA to watch the length of the inference_queue list.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: redis-queue-scaledobject
  namespace: inference
spec:
  scaleTargetRef:
    name: llama-inference # Your deployment
    apiVersion: apps/v1
  pollingInterval: 15  # Prometheus scrape interval impact: 15s interval adds <0.1% CPU overhead on inference server
  cooldownPeriod: 300  # 5 minutes after scaling in before scaling again
  minReplicaCount: 1
  maxReplicaCount: 20  # OUR INITIAL, NAIVE LIMIT. We'll fix this later.
  triggers:
  - type: redis
    metadata:
      address: redis-service.inference.svc.cluster.local:6379
      listName: inference_queue
      listLength: "5" # Scale up when queue has >5 pending jobs
      activationListLength: "1"

This config says: “Keep at least one pod warm. If the queue holds more than 5 jobs, start scaling out. For every 5 additional jobs, add another pod. Scale down when the queue is less than 1.” The cooldownPeriod is essential to prevent thrashing—Kubernetes GPU scheduling overhead adds 200-400ms per pod launch, and you don’t want to pay that cost every 30 seconds.

Crafting the ScaledObject: Cooldowns, Stabilization, and Pod Disruption

The ScaledObject is where you define the scaling behavior's personality. A twitchy, impatient autoscaler will burn money. A sluggish one will tank your P99 latency.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: gpu-inference-scaledobject
spec:
  scaleTargetRef:
    name: llama-inference
  pollingInterval: 30
  # Critical for cost control: How fast/slow to react
  cooldownPeriod: 600          # 10 minutes after scaling in
  stabilizationWindowSeconds:
    down: 300                  # Wait 5 minutes of low metrics before scaling in
  minReplicaCount: 2           # Always keep 2 warm pods for baseline traffic
  maxReplicaCount: 30          # Temporary ceiling
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
          - type: Pods
            value: 1
            periodSeconds: 60  # Remove at most 1 pod per minute
        scaleUp:
          stabilizationWindowSeconds: 90
          policies:
          - type: Pods
            value: 4
            periodSeconds: 90  # Add up to 4 pods every 90 seconds (rate limit)
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring.svc:9090
      query: |
        avg(rate(inference_request_duration_seconds_bucket{le="1.0"}[2m])) < 0.95
      threshold: "1"
      # Query explanation: Scale up if less than 95% of requests finish in <1 second over the last 2 minutes.

This configuration is deliberately conservative on scale-down (cooldownPeriod: 600, stabilizationWindowSeconds: 300) and rate-limited on scale-up. A burst of traffic can add pods quickly (4 per 90 seconds), but draining them down is slow. This prevents the “yo-yo effect” where you pay for constant pod creation cycles. Remember, spot GPU instances cost 60-80% less than on-demand, but if you’re constantly terminating and re-requesting them, you lose the savings to orchestration overhead and increased interruption risk.

The Cost Circuit Breaker: Hard-Stop Scaling at $800

This is the guardian that prevents the $12,000 bill. The naive maxReplicaCount is not enough—it limits pod count, not cost. A pod on an A100 spot instance costs ~$2/hr. A pod on an on-demand H100 might cost ~$12/hr. The same maxReplicaCount has a 6x cost variance.

We implement a circuit breaker using KEDA’s multiple triggers. One trigger scales based on queue depth/latency. A second, inhibiting trigger watches a cost metric and blocks scale-up when a threshold is breached.

First, you need a cost metric. Expose a Prometheus gauge from a sidecar that polls your cloud provider’s billing API (or uses a cloud-specific exporter). For example, a simple script for Lambda Labs:

#!/bin/bash

COST_PER_HOUR=$(curl -s -H "Authorization: Bearer $LAMBDA_API_KEY" \
  https://cloud.lambdalabs.com/api/v1/instances | \
  jq '[.data[] | select(.status=="running") | .instance_type.hourly_price] | add')

# Push to Prometheus pushgateway or expose as a /metrics endpoint
echo "lambda_running_cost_per_hour $COST_PER_HOUR" > /tmp/metrics

Then, define a KEDA trigger that is always above its threshold when costs are acceptable, and falls below it to inhibit scaling when the budget is blown. We use a prometheus trigger with an inverted logic.

triggers:
- type: prometheus
  metadata:
    serverAddress: http://prometheus-server.monitoring.svc:9090
    query: |
      lambda_running_cost_per_hour < 800 # Our $800 monthly budget guardrail
    threshold: "1"
    activationThreshold: "1" # This trigger must be active (cost < $800) for scaling to occur.

The key is activationThreshold. If the query result is 0 (because cost_per_hour >= 800), this trigger is inactive. KEDA requires all triggers to be active to scale out. One inactive trigger blocks scale-up entirely. Scale-down is still permitted. This is your emergency brake.

Benchmark: Cold vs. Warm Scale-Out Latency

Autoscaling isn’t free. The time between a traffic spike and your 10th pod serving requests is latency added to your users’ requests. You must measure this.

We benchmarked scaling from 1 to 10 pods using a load generator and measured time-to-ready. The results hinge on two factors: node pool composition (warm vs. cold nodes) and container image caching.

The takeaway: Your minReplicaCount should cover your baseline traffic. Your node pool should maintain a buffer of ready nodes (or use a virtual node pool like Modal). Modal cold start for GPU containers averages 2-4s vs Replicate's 8-15s—if your provider choice is flexible, this delta is a direct latency penalty on every scale-up.

Mixing On-Demand and Spot GPU Nodes for Resilience

Putting all your inferencing on spot instances is a recipe for getting a termination notice right during your peak hour. The solution is a mixed node pool with topology spread constraints.

In your cloud provider, create two node groups: one with on-demand instances (e.g., gpu-pool-ondemand), one with spot instances (e.g., gpu-pool-spot). Label them accordingly.

Then, use a topologySpreadConstraints in your pod spec to distribute pods across the pools, preferring the cheaper spot pool, but allowing overflow to on-demand.

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: node-pool-type
                operator: In
                values:
                - spot
          - weight: 1
            preference:
              matchExpressions:
              - key: node-pool-type
                operator: In
                values:
                - ondemand
      topologySpreadConstraints:
      - maxSkew: 2
        topologyKey: node-pool-type
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: llama-inference

This configuration says: “Try really hard to put pods on spot nodes (weight: 100). But never let the difference between the number of pods on spot vs. on-demand exceed 2 (maxSkew: 2).” This gives you cost efficiency with a firm reliability floor. Remember the spot instance interruption rate: AWS p3 ~5%/hr, Lambda Labs ~1%/hr. Your application must handle termination gracefully. Implement a 2-minute checkpoint handler via the instance metadata endpoint (e.g., for AWS):

# In your pod's preStop lifecycle hook:
curl -s http://169.254.169.254/latest/meta-data/spot/instance-action || exit 0
# If the endpoint returns data, a termination notice has been issued.
# You have ~2 minutes to finish current requests, checkpoint, and exit.

Production Checklist: Beyond Basic Scaling

If you deploy only the ScaledObject, you will fail at 3 AM. Here’s the rest of the configuration.

1. PodDisruptionBudget (PDB): Prevents too many pods from being killed simultaneously during node drains or spot terminations.

   apiVersion: policy/v1
   kind: PodDisruptionBudget
   metadata:
     name: llama-inference-pdb
   spec:
     minAvailable: "50%" # Or a specific number like 3
     selector:
       matchLabels:
         app: llama-inference
   

2. Meaningful Health Checks: A pod with a running container is not a pod serving inferences.

   livenessProbe:
     httpGet:
       path: /health
       port: 8080
     initialDelaySeconds: 120 # Give the 70B model time to load from NVMe!
     periodSeconds: 20
   readinessProbe:
     httpGet:
       path: /ready
       port: 8080
     initialDelaySeconds: 120
     periodSeconds: 5 # Frequent checks to quickly add pods to the service
   

3. Graceful Shutdown & Resource Cleanup: Ensure the preStop hook drains requests.

   lifecycle:
     preStop:
       exec:
         command: ["/bin/sh", "-c", "kill -SIGTERM $(pidof ollama) && sleep 30"]
   terminationGracePeriodSeconds: 60
   

4. Monitor the Right Metrics. Your Grafana dashboard needs:

   • nvidia.com/gpu utilization (via DCGM exporter or nvidia-smi scraping).
   • Request queue depth vs. pod count.
   • GPU memory bandwidth bottleneck awareness: An A100 80GB delivers 2TB/s vs an RTX 4090's 1TB/s. If you’re memory-bandwidth bound, adding more pods won’t help latency.
   • Cloud cost per hour (from your circuit breaker metric).

Next Steps: From Autoscaling to Autonomous Clusters

You’ve now got a GPU inference cluster that scales with demand and has a hard cost ceiling. The next evolution is predictive scaling. Instead of reacting to queue depth, train a simple model (or use KEDA’s upcoming cron scaling with predictions) to scale out 5 minutes before your daily traffic spike based on historical patterns.

Finally, pressure-test the entire system. Use a load testing tool to blast your endpoint with requests and watch the scaling events in KEDA (kubectl get scaledobject -w). Verify the cost circuit breaker trips by manually setting your cost metric above the threshold. The goal isn’t to prevent scaling, but to make scaling a predictable, controlled, and non-bankrupting process. Your GPUs should sweat, not your finance team.