How does auto-scaling actually decide when to add or remove instances — which metrics and policies matter?

#What auto-scaling is actually deciding

Auto-scaling exists to match capacity to demand without a human watching a dashboard: add instances before latency degrades, remove them when they're idle, and do both automatically within minutes. The interesting part isn't the mechanism of adding or removing a server — cloud providers and orchestrators handle that — it's choosing the right signal to react to and the right policy to react with, since both of those decide whether scaling actually protects users or just adds cost and instability.

#Metrics: not all of them measure the right thing

CPU utilization is the default because it's cheap to collect and universally available, but it's a lagging, indirect signal for a lot of real services. A service that spends most of its time waiting on a downstream database or a slow third-party API can look nearly idle on CPU while requests are queuing up and users are experiencing real latency — CPU never crosses the scaling threshold because the bottleneck isn't CPU at all.

Memory utilization matters for services with per-request memory cost or caches that grow, but memory pressure often shows up as OOM kills rather than a smooth scaling signal, so it's frequently used as a scale-out trigger but rarely as a scale-in one.

Request latency, and specifically a high percentile like p99, is a much more direct proxy for user pain: it captures queuing delay caused by any bottleneck, not just CPU. For services that process work from a queue rather than serving synchronous requests, queue depth (or backlog age — how old is the oldest unprocessed item) is often the most honest signal of all, since the relationship between backlog and required worker count is close to linear and easy to reason about directly.

#Policy types

Target tracking behaves like a thermostat: you declare a target value for a metric (say, 60% average CPU, or 1,000 requests per second per instance), and a controller continuously computes how much capacity is needed to hold the metric at that target, scaling up or down to match. This is the default mode for both AWS Auto Scaling target tracking policies and the Kubernetes? Horizontal Pod Autoscaler.

Step scaling reacts with different magnitudes depending on how far the metric is from a threshold — CPU between 70-85% might add 2 instances, CPU above 85% might add 5. This responds faster to a severe spike than target tracking's smoother adjustment, at the cost of more manual tuning of the steps themselves.

Scheduled scaling pre-scales ahead of known, predictable patterns — a retail site scaling up before a flash sale, or a B2B service scaling up before the 9am login rush — sidestepping the reaction lag of any metric-driven policy entirely for events you already know are coming.

Predictive scaling forecasts demand from historical patterns and scales proactively ahead of the trend, rather than reacting only once the metric has already moved. It's most valuable when instance boot time is long enough that purely reactive? scaling would arrive after the spike has already caused pain.

#The core formula behind target tracking

Both AWS target tracking and the Kubernetes HPA compute desired capacity with a version of the same formula:

desiredReplicas = ceil( currentReplicas × ( currentMetricValue / desiredMetricValue ) )

If you're running 4 instances at 90% CPU against a target of 60%, the controller computes ceil(4 × (90/60)) = ceil(6) = 6 instances. This is why the target value you choose matters enormously: a target set too close to 100% leaves no headroom to absorb the time it takes new capacity to actually come online, while a target set too low over-provisions and wastes cost.

#Damping: cooldowns, stabilization windows, and asymmetry

A naive controller that reacts to every metric sample oscillates: it scales up, the metric drops below target, it scales back down, load returns, it scales up again — flapping that wastes cost on rapid instance churn and adds instability from constantly-restarting capacity. Cooldown periods enforce a minimum gap between consecutive scaling actions. Stabilization windows go further: rather than reacting to the latest sample, the controller looks at a rolling window of recent recommendations and picks the most conservative one — the maximum recommendation over the window for scale-up decisions (so a brief spike doesn't get undone before it's absorbed), and the minimum over the window for scale-down decisions (so a brief dip doesn't trigger a cut in capacity that gets immediately reversed).

That asymmetry is deliberate and worth calling out explicitly in an interview: scale-out should be fast and aggressive, since under-provisioning directly hurts users, while scale-in should be slow and conservative, since over-provisioning only costs money — a much cheaper mistake than a latency spike.

#Kubernetes HPA v2 in practice

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-service-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service
  minReplicas: 4
  maxReplicas: 60
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Pods
      pods:
        metric:
          name: requests_per_second
        target:
          type: AverageValue
          averageValue: "150"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
        - type: Pods
          value: 8
          periodSeconds: 30
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
      selectPolicy: Min

This configuration tracks two metrics simultaneously (CPU and a custom requests_per_second metric sourced from a metrics adapter) — the HPA computes desired replicas for each and takes the larger result, so whichever signal says "scale more" wins. The behavior block is where the asymmetry lives: scale-up has no stabilization delay and can double capacity (or add 8 pods, whichever is larger) every 30 seconds, while scale-down waits a full 5-minute window and removes at most 10% of pods per minute — fast to react to load, slow and cautious to give it back.

#Callouts

🧠 Check your understanding
0/3 · 0/3 answered
1. A service is bottlenecked on a slow downstream database call, not CPU. What's the risk of scaling purely on CPU utilization?
CPU-based scaling always causes instance flappingCPU may stay low while requests queue and latency degrades, so the scaling policy never triggers when it shouldThe service will scale too aggressively and waste moneyThere is no risk — CPU is a universal proxy for load
2. In the HPA formula desiredReplicas = ceil(currentReplicas × (currentMetricValue / desiredMetricValue)), what happens if the target (desiredMetricValue) is set very close to the metric's practical maximum?
There's little headroom left, so a burst of load can breach capacity before new instances finish bootingThe formula becomes invalidScaling becomes perfectly smooth with no oscillationIt has no effect since maxReplicas is the real limit
3. Why do production HPA configs typically use a much longer stabilizationWindowSeconds for scaleDown than for scaleUp?
Removing capacity too eagerly risks cutting instances right before the next load increase, which is a more costly mistake than briefly over-provisioningScale-down operations are computationally slower to executeIt has no real effect on autoscaling behaviorKubernetes requires scaleDown windows to be longer by default