DEV Community

Yash Sonawane
Yash Sonawane

Posted on

Kubernetes Keeps Restarting My Pod… And I Have No Idea Why

#ai #devops #kubernetes #docker

The beginner-friendly guide to debugging CrashLoopBackOff without randomly deleting everything.

You deploy your application to Kubernetes.

kubectl apply -f deployment.yaml
Enter fullscreen mode Exit fullscreen mode

Everything looks good.

You check your Pods:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

And then you see this:

NAME                       READY   STATUS             RESTARTS
my-app-7d8f9c6b5d-abc12     0/1     CrashLoopBackOff   7
Enter fullscreen mode Exit fullscreen mode

Your first thought?

"Let me delete the Pod."

kubectl delete pod my-app-7d8f9c6b5d-abc12
Enter fullscreen mode Exit fullscreen mode

Kubernetes creates another Pod.

The new Pod crashes, too.

Congratulations! You have successfully restarted your problem. 😂

Here's the truth: Kubernetes is not necessarily broken. Your application or its configuration might be.

Let's learn how to troubleshoot this like a real DevOps engineer.

🧠 1. What Does CrashLoopBackOff Actually Mean?

Let's break down the name.

  • Crash: Your container's main process exits unexpectedly.
  • Loop: Kubernetes repeatedly attempts to restart the container.
  • BackOff: Kubernetes increases the delay between restart attempts.

The cycle looks like this:

Pod starts
    |
    v
Container starts
    |
    v
Application crashes
    |
    v
Kubernetes restarts it
    |
    v
Application crashes again
    |
    v
Restart delay increases
    |
    v
CrashLoopBackOff 💀
Enter fullscreen mode Exit fullscreen mode

Kubernetes is trying to maintain the desired state of your workload, but repeatedly restarting the container does not automatically fix the underlying problem.

🔍 2. Step One: Describe the Pod

Don't guess. Start by collecting evidence.

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Then inspect the failing Pod:

kubectl describe pod my-app-7d8f9c6b5d-abc12
Enter fullscreen mode Exit fullscreen mode

Look at the following sections:

  • State
  • Last State
  • Restart Count
  • Events

You might discover something like:

Last State:     Terminated
Reason:         Error
Exit Code:      1
Restart Count:  7
Enter fullscreen mode Exit fullscreen mode

An exit code of 1 generally indicates that the process reported an error, but it does not tell you the exact cause.

Scroll down to the events. Kubernetes may reveal issues involving configuration, image pulls, mounting volumes, or resource constraints.

Pro tip: kubectl describe pod is one of the first commands you should learn for Kubernetes troubleshooting.

📜 3. Step Two: Read the Previous Container Logs

Here's a mistake beginners often make: they check only the current logs.

But what if the container has already crashed?

Use:

kubectl logs my-app-7d8f9c6b5d-abc12
Enter fullscreen mode Exit fullscreen mode

For the previous terminated container instance, use:

kubectl logs my-app-7d8f9c6b5d-abc12 --previous
Enter fullscreen mode Exit fullscreen mode

If your Pod has multiple containers, specify the container:

kubectl logs my-app-7d8f9c6b5d-abc12 \
  -c app \
  --previous
Enter fullscreen mode Exit fullscreen mode

You might find:

ERROR: DATABASE_URL is not configured
Application startup failed
Enter fullscreen mode Exit fullscreen mode

Boom. 💥

You now have a useful clue.

The issue might be a missing environment variable, not Kubernetes itself.

⚙️ 4. Step Three: Check Your Environment Variables

Applications often depend on environment variables for database connections, API endpoints, and other configuration.

Inspect the Deployment:

kubectl describe deployment my-app
Enter fullscreen mode Exit fullscreen mode

Review the relevant environment-variable configuration in your manifest:

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: app-secrets
        key: database-url
Enter fullscreen mode Exit fullscreen mode

Check whether the Secret exists:

kubectl get secret app-secrets
Enter fullscreen mode Exit fullscreen mode

Check whether the expected key exists without printing its value:

kubectl get secret app-secrets \
  -o jsonpath='{.data.database-url}' \
  | wc -c
Enter fullscreen mode Exit fullscreen mode

If the Secret or key is missing, the container might fail to start, depending on how the configuration is referenced.

Security reminder: Never paste production credentials into logs, public repositories, or screenshots.

💾 5. Step Four: Investigate Memory Limits

Sometimes your application is being killed because it consumes too much memory.

Check the Pod's last termination state:

kubectl describe pod my-app-7d8f9c6b5d-abc12
Enter fullscreen mode Exit fullscreen mode

Look for:

Reason: OOMKilled
Enter fullscreen mode Exit fullscreen mode

OOMKilled means the container was killed because of an out-of-memory condition.

Inspect the resource configuration:

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "256Mi"
    cpu: "500m"
Enter fullscreen mode Exit fullscreen mode

These are example values, not universal recommendations. Choose resource requests and limits based on the application's measured requirements.

If the container repeatedly exceeds its memory limit, investigate the application's memory consumption before simply increasing the limit.

Useful commands:

kubectl top pods
kubectl top nodes
Enter fullscreen mode Exit fullscreen mode

These commands require the Kubernetes Metrics API to be available, commonly through Metrics Server.

🌐 6. Step Five: Check Dependencies

Imagine your application needs PostgreSQL to start.

Your application starts, attempts to connect to the database, fails, and exits.

Kubernetes restarts it.

The database is still unreachable.

The application crashes again.

The cycle continues.

Check the relevant Services:

kubectl get services
Enter fullscreen mode Exit fullscreen mode

Check the Endpoints or EndpointSlices:

kubectl get endpoints
kubectl get endpointslices
Enter fullscreen mode Exit fullscreen mode

Check whether the database Pod is running:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

A Service can exist while having no usable backend endpoints.

Also, remember that Running does not necessarily mean Ready.

Your application may need retry logic and a readiness probe to handle dependencies gracefully.

❤️ 7. Use Probes Correctly

Kubernetes supports three important probe types.

Probe What it checks
Startup Has the application finished starting?
Readiness Should the Pod receive traffic?
Liveness Should Kubernetes restart the container?

Example:

startupProbe:
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 30
  periodSeconds: 5

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 10

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 20
Enter fullscreen mode Exit fullscreen mode

Use endpoints that genuinely reflect your application's health. Configure the timing and thresholds for your workload.

A badly configured liveness probe can create a restart loop even when the application could recover on its own.

A startup probe gives slow-starting applications time to initialize before liveness checks begin.

Important: Probes are not interchangeable. Readiness controls traffic eligibility; liveness can trigger restarts.

🛠️ 8. My Kubernetes Troubleshooting Cheat Sheet

Keep these commands handy:

# List Pods
kubectl get pods -o wide

# Inspect a Pod
kubectl describe pod <pod-name>

# View current logs
kubectl logs <pod-name>

# View previous container logs
kubectl logs <pod-name> --previous

# View namespace events
kubectl get events --sort-by=.metadata.creationTimestamp

# Inspect a Deployment
kubectl describe deployment <deployment-name>

# Check resource usage
kubectl top pods

# Check Services
kubectl get svc

# Check recent ReplicaSet and Pod changes
kubectl get deployments,replicasets,pods
Enter fullscreen mode Exit fullscreen mode

Replace the placeholders with your actual resource names.

These commands won't solve every problem automatically, but they'll help you gather evidence before making changes.

🚀 9. Build This Project for Your DevOps Portfolio

Want to practice this in a safe environment?

Build a Kubernetes Failure Simulator and Troubleshooting Lab.

Use:

  • Docker to package a simple application.
  • Kubernetes or Minikube to run it.
  • YAML manifests to configure Deployments and Services.
  • Prometheus and Grafana for monitoring, if you want to extend it.
  • A README documenting each failure and its resolution.

Simulate failures in a local cluster:

  1. Deploy an application with an incorrect startup command.
  2. Configure a missing environment variable.
  3. Set an intentionally low memory limit.
  4. Configure an incorrect health-check endpoint.
  5. Break a test database connection.

For each failure, document:

Incident:
What users experienced:
Evidence collected:
Root cause:
Fix:
Prevention:
Enter fullscreen mode Exit fullscreen mode

This demonstrates more than your ability to run kubectl commands. It shows that you understand diagnosis, recovery, and prevention.

Never deliberately introduce these failures into a production cluster just to practice.

🎯 Final Thoughts

Kubernetes troubleshooting is not about memorizing hundreds of commands.

It's about asking the right questions:

  • What failed?
  • When did it fail?
  • What do the logs say?
  • What changed recently?
  • Is the problem in the application, configuration, dependencies, or infrastructure?
  • How can I verify the fix?

The next time you see CrashLoopBackOff, resist the urge to delete the Pod immediately.

Investigate first. Fix the root cause. Verify the recovery.

That's how you move from a Kubernetes beginner toward a confident DevOps engineer.

Your turn 👇

What's the most confusing Kubernetes error you've encountered?

Share it in the comments. Let's learn from each other's failures!

📚 Want to Learn Kubernetes Step by Step?

Explore my learning resources:

Build it. Break it safely. Debug it. Learn it.

Kubernetes #DevOps #CloudComputing #Docker #SRE #DevOpsLearning #CNCF

Top comments (0)