When kubectl logs Gives You Nothing
You run kubectl get pods, see CrashLoopBackOff, and do the obvious thing: kubectl logs <pod>. Nothing. You add --previous. Still nothing, or worse, unable to retrieve container logs for containerd://... or previous terminated container not found. kubectl describe pod shows you restart counts climbing and an event that says Back-off restarting failed container, which tells you what you already knew.
This is where most guides stop being useful. They tell you to run logs --previous, which works when the container prints an error before dying. It does nothing when the container dies before it prints anything, or when it prints somewhere other than stdout. That's the actual problem you're dealing with here, and it needs a different toolkit.
Confirm You're Actually in This Situation
Before going further, rule out the boring explanation: the container did log something, and you just missed the window.
kubectl get pod <pod-name> -o yaml | grep -A 10 lastState
Look at three fields specifically:
exitCode— 0, 1, 137, or 143 all mean very different thingsreason—Error,OOMKilled,Completed, orContainerCannotRunstartedAt/finishedAt— subtract one from the other; if the gap is under a second, the app almost certainly crashed before any logging library initialized
If exitCode is 0, don't treat this as a normal crash. A clean exit code with restartPolicy: Always still triggers CrashLoopBackOff, because Kubernetes expects the main process of a long-running Pod to keep running forever. If it exits successfully, Kubernetes still calls that a failure to stay up and restarts it anyway. This is a completely different bug class from a segfault, and the fix is usually in your entrypoint script, not your application code. There's a deeper breakdown of this exact exit-code-0 scenario in why containers exit cleanly but still get treated as failed if you want the full mechanics.
Why the Logs Are Actually Empty
There are five real mechanisms behind this, and they call for different fixes.
1. The crash happens before the logger initializes
Many runtimes buffer stdout, or don't attach a logging handler until several lines into the app's startup path. If the process dies from a missing shared library, a failed config parse, or a panic in a dependency-injection step that runs before your logger is configured, there's genuinely nothing to capture. The container never wrote anything, so kubectl logs isn't hiding data — there's no data.
2. kubelet already garbage-collected the previous container
Kubelet keeps a limited number of terminated containers around for inspection, and it prunes them based on count and size, not time. If your Pod is looping fast — every few seconds — you can burn through that retention window before you even finish reading the describe output. This is the actual reason --previous sometimes returns container not found: it's not a bug, it's garbage collection outrunning you.
3. Your app logs to a file, not stdout
This is more common than people admit. Legacy apps, Java services with a log4j file appender, or anything migrated from a VM without touching its logging config will happily write to /var/log/app.log inside the container and never touch stdout or stderr at all. Kubernetes only ever captures stdout/stderr from the container runtime. If your app never writes there, kubectl logs will be empty on every single run, crash or not.
4. A probe or hook is killing the container before it finishes starting
An aggressive livenessProbe with a short initialDelaySeconds can kill a slow-starting container mid-boot, before it's printed its startup banner. From the outside this looks identical to a crash — same status, same restart counter climbing — but the actual failure is your probe configuration, not your code. Check the probe section in the Pod spec and the describe events for Liveness probe failed before assuming the app itself is broken. A postStart hook failure produces the same symptom: the container is killed by the hook's failure, and nothing your app logged matters because the hook ran independently.
5. The image has no shell, so nothing debuggable happens at all
Distroless and scratch-based images are great for production but terrible for this specific failure mode. There's no shell to exec into, no cat, no ls. If the binary itself fails to start — missing env var, bad file permission on a mounted secret — you get a bare exit and nothing else.
The Actual Diagnostic Workflow
Once logs and describe have given you everything they're going to give you, move to override-and-inspect. The goal is to stop the container from running its normal entrypoint and instead give yourself a window to poke around.
Override the command
Patch a copy of the Pod (or the Deployment, in a non-prod namespace) to replace the entrypoint with a no-op that stays alive:
kubectl debug <pod-name> -it --copy-to=debug-pod --container=<container-name> -- sh
If that fails because there's no shell in the image, use --image to attach a different debugging image into the same Pod's namespaces instead:
kubectl debug <pod-name> -it --image=busybox --target=<container-name>
This gives you a process-namespace-sharing sidecar you can use to inspect /proc, check environment variables, and look at whatever files the real container mounted, without needing the broken container to cooperate.
Manually run the entrypoint
Once you have a shell, don't just poke around — actually run the binary or script the container would have run, by hand, and watch what it prints in real time:
sh /app/entrypoint.sh
This alone catches the majority of "empty logs" cases: a missing environment variable produces an error the moment you run it manually, even though it never made it into kubectl logs because the container died before the runtime finished attaching the log stream.
Check where logs are actually going
ls -la /var/log/ /app/logs/
cat /proc/1/environ | tr '\0' '
'
If you find a log file with the actual stack trace sitting right there, your fix isn't a Kubernetes fix at all — it's a logging config change to point the app at stdout, or a Fluent Bit sidecar to tail that file and ship it out. If you're already running a metrics stack and want a fuller picture before the next crash happens, this pairs well with getting Prometheus and Grafana actually reporting Pod-level metrics so you can correlate crash timing with memory or CPU pressure instead of guessing.
Check terminationGracePeriodSeconds and grace period races
If your app needs time to flush buffered logs before exiting, and something sends SIGTERM before that flush completes, you'll never see the final lines. Bump terminationGracePeriodSeconds temporarily and see if more of the picture appears.
Fixing the Actual Cause
- Buffered stdout: force unbuffered output at the runtime level (
PYTHONUNBUFFERED=1for Python,-Dorg.slf4j.simpleLogger.log...flush settings for Java loggers,--no-bufferflags where available). - File-based logging: redirect the app's log config to stdout/stderr explicitly, or add a sidecar container that tails the log file and streams it to the container's own stdout.
- Probe killing early: raise
initialDelaySecondsandfailureThresholdto match real startup time, not a guess. - No shell in image: keep a debug-variant image tagged alongside your distroless production image specifically for this scenario, or lean on
kubectl debug --imageevery time instead of trying to exec directly. - Fast GC of previous containers: increase kubelet's container log retention if you control node config, or catch the log with a cluster-wide log shipper so you're never depending on
--previoussucceeding.
Verify It's Actually Fixed
Don't just watch the restart count stop climbing — that can mean the app is stable, or it can mean it's stuck in a state where the probe stopped checking. Confirm the actual startup sequence completed:
kubectl get pod <pod-name> -w
kubectl logs <pod-name> -f
Watch for your app's real startup banner or health-check-passed line, not just an absence of crashes. A Pod that's quietly failing its liveness probe and not restarting due to a misconfigured probe is arguably worse than one that's honestly crash-looping, because now nothing is telling you it's broken.
The pattern across all five causes here is the same: kubectl logs only shows you what the container chose to write to stdout, in the narrow window kubelet kept it around. Once you stop treating that command as ground truth and start treating it as one input among several — exit codes, probe events, manual entrypoint runs, filesystem inspection — the "empty logs" mystery usually resolves in a few minutes instead of a few hours.