The Secret Exists, the YAML Looks Fine, and It's Still Failing
You've already done the obvious checks. The image name is spelled correctly, the tag exists in the registry, and you created the imagePullSecrets object exactly the way the docs say. kubectl describe pod still shows ImagePullBackOff, and the event log gives you something vague like Failed to pull image: rpc error: code = Unknown desc = failed to authorize.
Most tutorials stop at "create a docker-registry secret and reference it in your pod spec." That's the easy 20% of the problem. The other 80% lives in places most people never check: service account bindings, token expiry windows, node-level credential stores, and registry-side throttling that masquerades as an auth failure. This guide walks through each of those, with the exact commands to confirm or rule out each one.
First, Confirm You're Actually Looking at an Auth Problem
Before chasing secrets, read the actual error text closely. Kubernetes bundles several different failure modes under the same ImagePullBackOff status, and the fix depends entirely on which one you're dealing with.
kubectl describe pod <pod-name> -n <namespace>
Look at the Events section at the bottom. The message text tells you which branch of this troubleshooting tree you're on.
| Error text contains | Likely cause |
|---|---|
| pull access denied, repository does not exist or may require authorization | Auth failure — secret missing, wrong, or not attached |
| unauthorized: authentication required | Token expired or credentials invalid on the registry side |
| toomanyrequests: You have reached your pull rate limit | Registry throttling (common on Docker Hub) |
| x509: certificate signed by unknown authority | TLS/cert issue, not credentials — different problem entirely |
| no basic auth credentials | The secret isn't reaching the kubelet at all |
If you're seeing no basic auth credentials after you've already added imagePullSecrets, that's your first strong signal the secret isn't actually attached where Kubernetes expects it.
Root Cause 1: The Service Account Isn't Referencing the Secret
This is the single most common reason imagePullSecrets "doesn't work" even when it's correctly defined. Kubernetes pulls images using the credentials attached to the pod spec or the credentials attached to the pod's service account — but if you patched the service account after pods were already scheduled, or created the pod in a namespace where the default service account has no pull secret, the pod will still fail.
Check what the pod's service account actually has:
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.serviceAccountName}'
kubectl get serviceaccount <sa-name> -n <namespace> -o yaml
If imagePullSecrets is empty under that service account, and your pod spec also doesn't declare it directly, the kubelet has nothing to authenticate with. Fix it at the service account level so every pod in the namespace inherits it:
kubectl patch serviceaccount default -n <namespace> \
-p '{"imagePullSecrets": [{"name": "regcred"}]}'
One detail people miss: patching the service account does not retroactively fix pods that are already stuck in ImagePullBackOff. You need to delete and recreate the pod (or let the deployment controller cycle it) for the new secret binding to take effect.
Root Cause 2: The Secret Exists, But the Token Inside It Has Expired
A docker-registry secret isn't a static password vault — it's a snapshot of a token at the time you ran kubectl create secret docker-registry. If that token came from a short-lived auth mechanism, the secret can go stale while looking completely valid in kubectl get secret.
This is especially common with AWS ECR, where the auth token returned by aws ecr get-login-password typically expires within hours, not days. If you built the secret once during initial setup and never automated a refresh, it will start failing silently weeks later.
Decode the secret and inspect the actual auth payload:
kubectl get secret regcred -n <namespace> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
You'll see something like:
{"auths":{"123456789.dkr.ecr.us-east-1.amazonaws.com":{"auth":"QVdT..."}}}
That auth field is a base64-encoded username:password pair. For ECR, decode it further and try authenticating manually against the registry with docker login or crane auth to confirm whether the registry itself rejects it right now. If it does, the token expired — the secret's existence tells you nothing about its validity.
For ECR specifically, the durable fix is a CronJob that regenerates the secret on a schedule shorter than the token's lifetime:
kubectl create secret docker-registry regcred \
--docker-server=<account>.dkr.ecr.<region>.amazonaws.com \
--docker-username=AWS \
--docker-password=$(aws ecr get-login-password --region <region>) \
--namespace=<namespace> \
--dry-run=client -o yaml | kubectl apply -f -
Run that on a kubectl CronJob or use a purpose-built controller like ecr-secret-refresher. Manually recreating the secret every few hours is not a real fix, it's a reminder to build automation you keep forgetting to build.
Root Cause 3: Node-Level Credential Mismatch
This one catches people off guard because it has nothing to do with the Kubernetes API objects at all. The kubelet delegates the actual pull operation to the node's container runtime — containerd or CRI-O — and in some cluster setups, that runtime has its own independent auth configuration that can conflict with, or completely override, what you defined in imagePullSecrets.
This shows up on self-managed clusters, bare-metal setups, or nodes provisioned with baked-in registry mirrors. If your node has a /etc/containerd/config.toml with a [plugins."io.containerd.grpc.v1.cri".registry] block pointing at a different mirror or set of credentials, the pod-level secret can be entirely ignored.
SSH into the affected node and test the pull directly, bypassing Kubernetes:
crictl pull <registry>/<image>:<tag>
If this fails with a different error than what you saw in kubectl describe pod, you've confirmed the node's runtime is using its own auth path. Check the runtime config:
cat /etc/containerd/config.toml | grep -A 10 registry
journalctl -u containerd -n 100 --no-pager
On managed Kubernetes (EKS, GKE, AKS), this is less common because the control plane manages runtime config for you, but it's a real problem on self-hosted clusters and homelab setups where nodes were provisioned manually or with drifted configuration.
Root Cause 4: You're Not Dealing With Auth at All — It's Rate Limiting
Docker Hub enforces pull rate limits for anonymous and free-tier authenticated pulls. If several nodes in your cluster are pulling the same public image without any registry credentials at all, you can hit toomanyrequests: You have reached your pull rate limit — and this gets misdiagnosed as a broken secret constantly, because the symptom category is identical: ImagePullBackOff.
Check for the specific string in the event message:
kubectl describe pod <pod-name> -n <namespace> | grep -i toomanyrequests
The fix isn't a new secret for a private registry — it's authenticating your pulls even for public images, which raises your limit tier, or routing through a pull-through cache. If you're on AWS, ECR's pull-through cache feature mirrors Docker Hub images through your own registry so nodes never hit Docker Hub's public limit directly. It's worth setting up before scaling node count, not after you get paged for it.
Root Cause 5: The Secret Is in the Wrong Namespace
Secrets are namespace-scoped objects. It's an easy mistake to create regcred in default, then deploy your workload into staging, and wonder why the exact same YAML that worked in one environment fails in another.
kubectl get secret regcred --all-namespaces
If it only shows up in one namespace and your pod lives in another, that's the entire bug. Recreate the secret in the correct namespace, or if you manage multiple namespaces, look at tools that replicate secrets across namespaces automatically rather than copy-pasting YAML by hand every time.
Verifying the Fix Actually Worked
Don't just wait and watch the pod status change. Confirm the mechanism directly:
kubectl delete pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | grep -i pull
You want to see Successfully pulled image in the events, not just the absence of an error. If the pod comes up but you're still unsure whether the fix was the secret or something else, revert one variable at a time — reattach the old secret, remove the service account patch, and see if the failure comes back. Isolating which layer actually fixed it saves you from repeating the same multi-hour debugging session next quarter.
What This Looks Like in Practice
Most ImagePullBackOff incidents that survive the first round of "check the image name" troubleshooting come down to one of two things: a token that expired silently, or a service account that never got the secret in the first place. Both fail identically from the pod's perspective but require completely different fixes. Building a small automated check — a CronJob that validates registry auth and alerts before pods start failing — costs less time than debugging this manually every time credentials rotate.