DNS latency spike when CoreDNS pod is deleted

The Symptom: DNS Timeouts That Show Up After the Pod Is Already Gone

You roll out a CoreDNS update, or a node drains and reschedules the CoreDNS pod, and a few minutes later your application logs fill up with i/o timeout or context deadline exceeded on hostname lookups. The weird part is the timing. It's not instant. It's not every request. It's a subset of pods, intermittently, for a window of anywhere from 30 seconds to a few minutes, and then it just stops on its own.

Most people reach for the obvious suspects first: CoreDNS CPU throttling, the CNI plugin being slow, or general "network flakiness." Those are reasonable guesses. They're also usually wrong here, and chasing them wastes the exact window where you could actually catch the real cause in the act.

What You'd Normally Check First (And Why They're Red Herrings)

Before blaming stale conntrack entries, rule out the usual suspects quickly, because they really do cause similar symptoms:

• CoreDNS resource throttling: check kubectl top pod -n kube-system and the container CPU throttling metrics for the CoreDNS pods. If CPU is pinned at the limit, that's a real bottleneck, but it causes sustained slowness, not a spike tied to a deletion event.
• CNI/network throughput: if your CNI is struggling (overlay overhead, MTU mismatches), you'd see latency across all traffic, not just DNS, and it wouldn't correlate with a pod delete event specifically.
• General conntrack table exhaustion: this is a real and distinct failure mode. When nf_conntrack_count approaches nf_conntrack_max, the kernel starts dropping packets that would create new entries, for every protocol, on that node, and you'll see nf_conntrack: table full, dropping packet in the kernel log. That's a capacity problem. What we're describing here is different: the table has plenty of room, but specific entries point to a pod IP that no longer exists.
• The conntrack insertion race: parallel A and AAAA lookups sent from the same socket can race when the entry is created, and one of the packets gets dropped, which produces the classic 5-second DNS delay. Check conntrack -S for a growing insert_failed counter. This shows up under steady load, not around pod deletions.

That distinction matters a lot, and it's the one most postmortems blur together. Exhaustion is "too many entries." What happens on CoreDNS pod deletion is "the right number of entries, but some of them are lying."

Reproducing It on Purpose

If you want to see this instead of just reading about it, do this on a test cluster. The key detail is that the client must keep reusing the same source port. A loop of nslookup calls won't show the problem, because every new process picks a new source port and therefore gets a fresh conntrack entry that is DNAT'd to a live backend. Long-lived sockets are what get stuck, which is exactly what real applications and resolvers keep around.

1. Note the CoreDNS pod IPs with kubectl get pods -n kube-system -o wide -l k8s-app=kube-dns, and the Service address with kubectl get svc kube-dns -n kube-system.
2. From a workload pod that has dig installed, query the Service IP from a fixed source port in a loop: while true; do dig -b 0.0.0.0#40000 @<kube-dns-clusterIP> kubernetes.default.svc.cluster.local +tries=1 +time=1 | grep -E 'status|timed out'; sleep 0.5; done
3. Delete the CoreDNS pod that the flow is currently pinned to (see the conntrack output below): kubectl delete pod <coredns-pod> -n kube-system.
4. Watch the loop. Depending on your kube-proxy version and timing you may see a stretch of timeouts. The race doesn't trigger every time, so you may need a few attempts.

While that's happening, go to the node where the client pod is running and look at the conntrack table directly (this needs conntrack-tools on the node):

conntrack -L -p udp --sport 40000

The entry's original direction shows the kube-dns ClusterIP as the destination, because that's what the client sent to. The DNAT result lives in the reply direction: look for src=<coredns-pod-ip> there. If that IP belongs to the pod you just deleted, and the entry is still there, that's the smoking gun.

The Actual Mechanism

DNS queries from pods go to the kube-dns Service's ClusterIP, not directly to a CoreDNS pod IP. kube-proxy, via iptables (or IPVS) rules, does destination NAT: it rewrites the packet's destination from the ClusterIP to one of the actual CoreDNS pod IPs behind it. That NAT decision gets cached as a conntrack entry so the kernel doesn't have to re-evaluate the iptables chain for every single packet in that flow.

For UDP, which is what DNS uses by default, that conntrack entry has a timeout. By default it's 30 seconds (nf_conntrack_udp_timeout) for an entry that hasn't seen reply traffic, and 120 seconds (nf_conntrack_udp_timeout_stream) once traffic has gone both ways. The timer is reset by every matching packet, so a client that keeps retrying from the same source port can keep a stale entry alive well past those numbers. For as long as the entry exists, every packet with the same protocol, source IP, source port, destination IP and destination port gets forwarded to the cached DNAT target, the dead pod IP, regardless of what the current Service endpoints list says.

When the CoreDNS pod gets deleted, the EndpointSlice controller updates the Service's endpoints almost immediately. kube-proxy updates its iptables/IPVS rules shortly after. But existing conntrack entries aren't automatically torn down just because the backend disappeared. kube-proxy does attempt to clean up UDP conntrack entries tied to removed endpoints, but this cleanup has historically been inconsistent, especially under rapid pod churn, rolling restarts, or when multiple CoreDNS replicas are cycled close together. The result: a live client socket keeps reusing a NAT'd path to a pod IP that no longer has anything listening on it, and every query on that flow blackholes until the entry goes quiet and the kernel ages it out.

This is also why the symptom looks random from the application's point of view. Only the pods whose client sockets happened to have an active conntrack entry pointing at the deleted CoreDNS pod are affected. Everyone else resolves fine, because their conntrack entry (if any existed) either expired already or pointed at a different, still-alive CoreDNS replica.

Confirming It's This and Not Something Else

Don't just trust the theory. Verify on the actual node:

• Get the old CoreDNS pod IP before or right when it's deleted, from kubectl get pods -n kube-system -o wide, your monitoring history, or your CNI logs. Pod events usually don't include the IP.
• Run conntrack -L -p udp --reply-src <old-pod-ip> on nodes hosting affected client pods (conntrack -L | grep <old-pod-ip> also works).
• If you see entries with that address still present after the pod is gone, that's confirmed stale state, not a CNI or throttling issue.
• Cross-check nf_conntrack_count against nf_conntrack_max at the same time. If you're nowhere near the max, it rules out plain exhaustion as the cause, and a flat insert_failed in conntrack -S rules out the insertion race.

If your platform uses IPVS mode for kube-proxy instead of iptables, the mechanics differ: IPVS keeps its own connection table, and stale connections to a removed backend are a known source of the same kind of blackhole. The relevant knobs (for example net.ipv4.vs.expire_nodest_conn) and their behavior depend on the kernel and kube-proxy version, so verify on your own versions.

The Fix

There isn't one silver-bullet patch, but there's a clear priority order:

1. Deploy NodeLocal DNSCache. This runs a DNS caching agent on each node that pods reach through a link-local address. Traffic from pods to the local cache skips connection tracking, so it never gets a DNAT entry pointing at a CoreDNS pod. By default the cache also talks to cluster DNS over TCP, where a backend that disappears produces a connection reset and a fast retry instead of a silent UDP blackhole. Cache hits never leave the node at all. Two caveats: if you configure the cache to use UDP toward its upstreams, you bring the original problem back on cache misses, and since the cache runs as a DaemonSet, restarting it causes a brief gap on that node.

2. Let CoreDNS drain gracefully before it exits. CoreDNS has its own mechanism for this: the lameduck option of the health plugin (many default Corefiles ship with lameduck 5s). On SIGTERM, CoreDNS keeps answering for that period so endpoint removal can propagate through kube-proxy before it stops. Make sure terminationGracePeriodSeconds is comfortably longer than the lameduck time. A generic preStop sleep is the usual alternative, but the CoreDNS image is distroless and has no shell or sleep binary, so an exec-based sleep won't work. On recent Kubernetes versions you can use the native lifecycle.preStop.sleep action instead (enabled by default since 1.30).

3. Set a PodDisruptionBudget for CoreDNS. This keeps rollouts and node drains from taking down multiple replicas at once, which is what turns a minor blip into the multi-minute outage pattern people actually notice. This overlaps with the same kube-proxy/iptables backend-selection behavior covered in why Kubernetes Services sometimes send traffic unevenly across pods, since both come back to how kube-proxy maintains and updates backend mappings.

4. Manually flush stale entries during an active incident. As a stopgap while you're bleeding, delete the entries that were DNAT'd to the dead pod on affected nodes: conntrack -D -p udp --orig-dst <kube-dns-clusterIP> --reply-src <deleted-pod-ip>. This forces those flows to be re-evaluated against the current rules on their next packet instead of waiting for the entry to expire. (Note that --orig-dst is the Service ClusterIP, not the pod IP, because that's the destination the client originally used.)

5. Tune nf_conntrack_udp_timeout_stream and nf_conntrack_udp_timeout lower if your environment churns pods frequently. Shorter timeouts mean stale entries age out faster, at the cost of slightly more conntrack table churn under normal load. This is a mitigation, not a fix: it only helps once the flow goes quiet, so a client that keeps retrying on the same source port can still hold on to a stale entry. It just shrinks the blackhole window.

Verifying the Fix Actually Worked

Repeat the reproduction steps from earlier, but this time with NodeLocal DNSCache, graceful draining and the PDB in place. Delete a CoreDNS pod again and watch the same query loop. You should see either no visible interruption, or a much shorter one.

Then check conntrack again during the deletion window. If UDP entries pointing to the dead pod IP either don't appear or clear out almost immediately instead of sitting there until timeout, you've actually fixed the mechanism, not just made the symptom quieter for a week until the next rollout. With NodeLocal DNSCache in place, expect the client-to-cache traffic not to show up in conntrack at all, so check the cache-to-CoreDNS path (TCP, by default) instead.

Keep an eye on nf_conntrack_count over time too, even after this fix. A system that's healthy today under this specific failure mode can still slide into plain conntrack exhaustion later as pod density on a node grows, and that's a different alarm you want tripped before DNS starts timing out for a completely separate reason.