kubernetes hpa not scaling down after load decreases

Why HPA Gets Stuck Scaled Up

You dropped the load hours ago. Traffic graphs are flat. But kubectl get hpa still shows 12 replicas when it should show 3. This is one of the most common Kubernetes autoscaling complaints, and it almost never has a single root cause.

The Horizontal Pod Autoscaler is a controller that polls metrics on a loop, compares them against a target, and calculates a desired replica count. That sounds simple. In practice, four separate subsystems have to agree with each other before a scale-down actually happens: the metrics pipeline, the resource requests on your pods, the HPA controller's own stabilization logic, and the metrics-server's scrape interval. If any one of those is misconfigured, you get a controller that scales up fine but refuses to come back down.

Let's go through each failure mode the way you'd actually debug it in production, not the way the docs describe HPA in theory.

Failure Mode 1: Stale or Missing Metrics

HPA makes decisions based on whatever the metrics API reports at the moment it polls. If metrics-server (or your custom metrics adapter) is returning old data, cached data, or nothing at all, the HPA controller has no real signal to react to.

Check this first:

kubectl get hpa <name> -o yaml

Look at the status.currentMetrics field. If the values there don't match what you see in your actual monitoring dashboard, your metrics pipeline is lying to the controller. This is extremely common when metrics-server is under resource pressure itself, or when it's been restarted and hasn't finished re-scraping all nodes.

Also check metrics-server logs directly:

kubectl logs -n kube-system deployment/metrics-server

Errors like unable to fully scrape metrics or did not receive metrics for some nodes mean the HPA is working off partial data, and partial data almost always biases toward keeping replica counts high, since a missing metric often gets treated conservatively.

If you're running custom or external metrics from something like Prometheus Adapter, the lag is worse. Prometheus itself scrapes on an interval, the adapter queries Prometheus on its own interval, and then HPA polls the adapter on its own interval. You can stack three separate delays on top of each other. If you haven't already got solid visibility into what your metrics pipeline is actually reporting versus what's real, it's worth revisiting your Prometheus and Grafana setup to confirm metrics are flowing without gaps before you even touch HPA configuration.

Failure Mode 2: Missing or Wrong Resource Requests

This one catches people constantly. HPA's CPU/memory-based scaling doesn't work off raw usage numbers. It works off a percentage of the requested resources on the pod spec.

If your deployment doesn't set resources.requests.cpu, HPA has nothing to calculate a percentage against. In older Kubernetes versions this could silently produce broken behavior; in current versions the HPA will report a condition like FailedGetResourceMetric or show unknown targets in kubectl describe hpa. Either way, without requests defined, the whole autoscaling percentage math is undefined.

Even when requests exist, wrong ones cause a version of the same problem. Say you set cpu: 100m as a request but your app realistically idles around 90m and spikes to 300m under load. Every small blip pushes you way over 100 percent of request, so the HPA scales up aggressively, and it takes a long, sustained drop in usage before the average comes back down enough to justify removing pods. The fix isn't a magic setting, it's setting realistic requests based on actual observed usage, not a guess copied from a tutorial.

Run this to sanity check your current requests against real usage:

kubectl top pods -n <namespace>

Compare that against kubectl describe deployment <name> and look at the requests block. If usage is consistently far from requests in either direction, fix the requests before touching HPA thresholds.

Failure Mode 3: Stabilization Window Misconfiguration

This is the one that's actually intentional and most people don't realize it. HPA has separate scale-up and scale-down behaviors, and the scale-down side is deliberately conservative by design, specifically to prevent flapping.

By default, HPA looks at the highest recommended replica count across a stabilization window before scaling down, not the current one. If your load spikes and drops repeatedly, the controller keeps picking the peak value from that window, which means it can look like it's "stuck" for much longer than you'd expect. This isn't a bug, it's the stabilization window doing exactly what it's supposed to do.

You can inspect and tune this in behavior.scaleDown:

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
    - type: Percent
      value: 50
      periodSeconds: 60

If your workload has genuinely predictable traffic (steady drop at night, batch jobs that finish cleanly), a 300 second window is often too conservative and you're waiting five minutes past when you should have already scaled down. Dropping it to something like 60-120 seconds and adding a smaller percentage-based policy usually gets scale-down behavior matching real load much sooner, without reintroducing flapping.

On the flip side, if you're seeing rapid up-down-up-down replica churn, the fix goes the other direction: increase the stabilization window and reduce the scale-down policy percentage per period. Flapping and "stuck scaled up" are actually opposite symptoms of the same root misconfiguration — the window doesn't match your traffic pattern.

Failure Mode 4: Metrics-Server Scrape Interval Lag

Even with correct requests and correct behavior policies, metrics-server itself has a default scrape interval that isn't instantaneous. HPA also has its own sync period on top of that. Under load, these two intervals combine, and the effective delay before the controller "sees" a drop in usage can be longer than people expect, especially on larger clusters where scraping every node takes longer.

Check how metrics-server is deployed and whether it's resource-constrained:

kubectl top nodes
kubectl describe deployment metrics-server -n kube-system

If metrics-server itself is CPU throttled or getting OOMKilled under cluster load, its scrape cycles slow down or fail intermittently, and HPA ends up making decisions on metrics that are minutes old instead of seconds old. This is a real operational issue on clusters running hundreds of pods with default metrics-server resource limits untouched since install.

A Practical Debugging Order

When scale-down isn't happening, work through it in this order rather than guessing:

1. Confirm status.currentMetrics on the HPA object matches reality
2. Confirm resource requests exist and are realistic versus actual usage
3. Check behavior.scaleDown stabilization window and policies
4. Check metrics-server health and resource limits

Most teams jump straight to tweaking stabilizationWindowSeconds because it's the most visible knob, but if your metrics pipeline is stale or your requests are wrong, no amount of behavior tuning will fix it. Fix the data first, then tune the reaction to the data.

Preventing This Going Forward

Set resource requests based on load-tested numbers, not defaults copied from another service. Keep metrics-server on a recent version with adequate CPU/memory headroom for your cluster size. Set behavior.scaleDown explicitly instead of relying on the built-in default, because the default is tuned for general safety, not for your specific traffic shape. And actually watch kubectl describe hpa during a real load test, both scale-up and scale-down, before you trust it in production.

HPA isn't broken when it doesn't scale down. It's usually doing exactly what its inputs tell it to do. The job is figuring out which input is wrong.