The Symptom: One Pod Is On Fire, The Rest Are Idle
You scale a Deployment to six replicas behind a Service. CPU on one pod climbs to 90%. The other five sit at 5-10%. kubectl get endpoints shows all six pods listed and ready. Your readiness probes are green across the board. And yet traffic keeps landing on the same one or two pods, request after request, for minutes at a time.
This is one of the most common "it should just work" surprises in Kubernetes, and almost every guide on the internet tells you to check selectors and endpoint readiness. That checklist takes two minutes and almost never finds anything when this specific pattern shows up. The real cause lives one layer deeper, in how kube-proxy actually picks a backend and how your clients are talking to the Service in the first place.
Rule Out The Obvious First
Before chasing anything exotic, confirm the basics so you're not debugging the wrong problem:
kubectl get endpoints <service>— all pod IPs present, none missingkubectl get pods -l <selector> -o wide— labels actually match the Service selector- Readiness probes not flapping (
kubectl describe podfor recent probe failures) kubectl get endpointslices— confirm the slice isn't stale or split oddly across zones
If all of that looks clean and the imbalance is still there, you're not dealing with a broken Service. You're dealing with how connections get distributed, not how requests get distributed. Those are two very different things in Kubernetes, and conflating them is the root of almost every "uneven load" confusion thread you'll find.
The Actual Mechanism: Load Balancing Happens Per-Connection, Not Per-Request
kube-proxy, in both iptables and IPVS mode, makes its backend selection decision when a new connection is established — not for every packet, and not for every HTTP request inside that connection.
In iptables mode, kube-proxy writes a chain of DNAT rules using the statistic module to randomly pick a pod IP for a new connection. Once that connection's SYN picks a backend, the kernel's conntrack table remembers that mapping. Every subsequent packet belonging to that same TCP connection gets rewritten to the same pod automatically, with zero further randomness involved. That's by design — you absolutely want a TCP stream to stay pinned to one backend for the life of the connection.
The problem is what counts as "the life of the connection." If your clients use HTTP/1.1 keep-alive, or gRPC (which multiplexes many RPCs over a single long-lived HTTP/2 connection), that one TCP connection might live for minutes, hours, or until the process restarts. Every request on it goes to the same pod, no matter how many replicas you add behind the Service.
IPVS mode behaves similarly for this scenario. It supports real scheduling algorithms like round-robin, least-connection, and destination hashing, which gives you better distribution of new connections. But it doesn't rebalance existing ones either. If your traffic pattern is a small number of clients holding a small number of persistent connections, IPVS's smarter scheduler barely matters — you still only ever had a handful of connections to schedule in the first place.
Why This Hides Well: Few Clients, Many Pods
This gets worse the fewer upstream clients you have. If three API gateway pods each open one persistent gRPC connection to a backend Service with ten replicas, you have exactly three connections to distribute across ten pods. Even perfect randomness gives you a lopsided result — three pods get all the traffic, seven get none. This is structurally identical to the problem described in why adding more consumers doesn't fix Kafka consumer lag: adding more backend capacity doesn't help when the bottleneck is the number of active channels moving work, not the number of workers waiting for it.
It also shows up disguised as a scaling problem. Teams add an HPA rule, watch replica count go up, and traffic distribution doesn't change at all — because the new pods never get a connection routed to them. If you're also chasing HPA that won't scale down after load drops, check whether your "load" metric is even measuring the right pods in the first place; CPU on one hot pod can keep the HPA from scaling down a fleet that's mostly idle.
Hairpin NAT: The Quieter Variant
A related failure shows up when a pod calls a Service VIP that happens to resolve back to a pod on the same node — including itself. This is called hairpinning. Depending on kube-proxy's masquerade settings, traffic that hairpins can either get dropped (because the return path doesn't look like a valid conntrack entry anymore) or get locked onto the local endpoint disproportionately, since node-local routing short-circuits before the normal selection logic even runs. If you notice the imbalance correlates with which node the client pod is scheduled on rather than which pods are "up," this is likely what you're looking at. kube-proxy's masqueradeAll and hairpin-mode settings control this, and it's worth checking your CNI's documentation for known hairpin edge cases before assuming it's purely a Service config issue.
Fixing It
There's no single switch that fixes connection pinning — you have to attack it from the client side, the server side, or both.
Cap connection lifetime server-side. Force long-lived connections to periodically close and force the client to reconnect, which triggers a fresh backend selection. In practice:
- nginx:
keepalive_requestsandkeepalive_timeoutset to sane finite values instead of unlimited - Node.js HTTP servers:
server.keepAliveTimeoutandserver.maxRequestsPerSocket - gRPC servers:
MAX_CONNECTION_AGEandMAX_CONNECTION_AGE_GRACEon the server-side channel args, which forces clients to reconnect on a rolling basis
Move load balancing to L7, not L4. kube-proxy only ever does connection-level balancing because it operates below the HTTP layer. If you need per-request fairness, you need something that terminates and re-issues requests — a sidecar proxy (Envoy, Linkerd) or an ingress/mesh doing request-level load balancing. This is the standard fix for gRPC specifically, since gRPC's own client-side load balancing (via grpc.lb_policy=round_robin against a headless Service) solves the exact same problem without extra infrastructure.
Use a headless Service for client-aware balancing. A headless Service (clusterIP: None) returns all pod IPs via DNS instead of one VIP, letting the client library open and balance across multiple real connections itself. This is the right pattern for gRPC and for any client library that supports client-side LB — you give up kube-proxy's abstraction in exchange for actual per-request fairness.
Check Internal Traffic Policy if you're using Local. Setting Service internal traffic policy to Local restricts routing to node-local endpoints only, which is great for latency but will produce dramatic imbalance if your pods aren't spread evenly across nodes. If you've set this and see skew correlated with node pod counts, that's expected behavior, not a bug.
Switch kube-proxy to IPVS with least-conn scheduling if you're still on iptables mode and dealing with high connection churn. It won't fix long-lived connection pinning, but it gives you meaningfully better distribution for workloads that open and close connections frequently (typical REST clients without keep-alive).
Verifying The Fix
Don't just eyeball CPU graphs and declare victory. Confirm the mechanism directly:
# See which pod IPs currently hold active conntrack entries for your Service port
conntrack -L | grep <service-port>
# From inside a client pod, check how many connections are open and to whom
kubectl exec -it <client-pod> -- ss -tn
If you fixed it correctly, you should see connections rotating across multiple backend pod IPs over time instead of piling onto one. Pair that with per-pod request-count metrics (via your existing Prometheus setup, similar to how you'd verify fixes in a Prometheus and Grafana docker-compose setup) so you have before/after evidence instead of a guess.
The underlying lesson is worth keeping in mind for any system doing connection pooling, not just Kubernetes Services: whenever a layer reuses a connection to avoid the cost of reconnecting, it's trading efficiency for the risk of pinning. It's the exact same tension you see in database connection poolers like PgBouncer, where connection pooling strategy choices directly affect how evenly work gets spread across backend processes.