DEV Community

Cover image for DevOps Outage Scenario: Kubernetes OOMKilled (Exit Code 137) Cascades Under High Load
intervixa
intervixa

Posted on

DevOps Outage Scenario: Kubernetes OOMKilled (Exit Code 137) Cascades Under High Load

In technical interviews for Senior Cloud, DevOps, and SRE roles, trivial syntax questions are gone. Hiring managers want to see how you respond when production is down at 2 AM.

Today, let's dissect a real-world production incident: Kubernetes OOMKilled (Exit Code 137) Cascades Under High Load.


The Scenario

Incident Context: During peak hours, your primary microservice pods start restarting with ExitCode 137. Remaining pods absorb traffic and crash in cascade.

When facing this situation during a live interview or outage, interviewers evaluate structured triage:

  1. Blast radius identification
  2. Immediate traffic mitigation
  3. Root cause isolation
  4. Permanent architectural hardening

1. Immediate Incident Triage

Verify current telemetry and active states before applying code changes:

kubectl describe pod <pod-name> -n prod | grep -E '(OOMKilled|Exit Code)'
kubectl top pods -n prod --sort-by=memory
kubectl scale deployment/api --replicas=15 -n prod
Enter fullscreen mode Exit fullscreen mode

2. Root Cause Analysis (RCA)

Why did this happen?

Unset JVM MaxRAMPercentage caused container memory limits to be breached, while Horizontal Pod Autoscaler lacked memory metric rules.


3. Permanent Architectural Hardening

Configure JVM flags (-XX:MaxRAMPercentage=75.0), define memory requests/limits correctly, and enable memory-based HPA scaling.


💡 Practice Scenarios Aloud with Live AI

Knowing the fix is only half the battle. In a senior interview, how smoothly you articulate your debugging thought process under pressure determines your rating.

You can practice answering these exact incident scenarios aloud with real-time audio and video feedback on the Intervixa Live AI Mock Interviewer.

Helpful Resources:


What is your team's standard operating procedure for handling this type of incident? Let's discuss in the comments below!

Top comments (0)