Why explicit cgroup limits are non‑negotiable in production Kubernetes clusters

when a new tenant lands on a shared node, the first thing that goes wrong is usually a runaway container that steals memory or CPU, starving the rest of the workloads. in my last on‑call rotation the node went from 70 % to 100 % memory within minutes, the OOM killer started terminating unrelated pods, and the incident stretched beyond our SLA.

the missing guard: cgroup limits in pod specs

Kubernetes lets you describe resource requests and limits, but the defaults are often left at "0" (unbounded) because the UI hides the fields or the helm chart skips them for simplicity. without explicit memory and cpu limits, the container runs directly against the node’s cgroup hierarchy, which is exactly the noisy‑neighbor scenario we try to avoid.

apiVersion: v1
kind: Pod
metadata:
  name: data‑processor
spec:
  containers:
  - name: worker
    image: myorg/processor:latest
    resources:
      requests:
        cpu: "500m"
        memory: "256Mi"
      limits:
        cpu: "1"
        memory: "512Mi"
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "http://otel-collector:4317"

The limits line creates a hard cgroup cap; the kernel will throttle CPU and trigger OOM kill only for that container. note that the requests field also informs the scheduler, keeping the pod on a node with sufficient headroom.

trade‑offs you have to accept

• throughput vs safety – a tightly capped pod may hit back‑pressure sooner, requiring you to tune your application’s internal queueing or enable graceful degradation.
• cost impact – over‑provisioning limits forces you to allocate more nodes to meet aggregate demand, but the alternative is unpredictable eviction and higher incident toil.
• observability burden – you now need to surface cgroup metrics (e.g. container_cpu_cfs_throttled_seconds_total, container_memory_working_set_bytes) into Prometheus and set alerts for sustained throttling or OOM events.

continuous validation with chaos engineering

Static limits are only as good as the workload profiles you measured. embed a simple chaos test in your CI pipeline:

func TestMemoryThrottle(t *testing.T) {
    // launch pod with 128Mi limit, feed 200Mi of data
    // assert that the process respects context cancellation after 30s
    // verify that the node’s memory usage stays below 85% during the run
}

Running this test on every PR forces developers to keep the container footprint in check and surfaces regression before it hits production. combine it with a production‑grade chaos runner (e.g. Litmus or Gremlin) that randomly spikes CPU on a node to verify that other pods stay within their cgroup caps.

edge cases to watch

• burstable QoS – pods with requests < limits can be throttled under node pressure; if your SLAs depend on consistent latency, consider requests == limits to lock them in the Guaranteed QoS class.
• sidecar interference – a logging sidecar that writes massive buffers can bypass the main container’s limits; enforce limits at the pod level or run sidecars in separate pods.
• node‑level cgroup overrides – some managed K8s providers inject default limits that conflict with your spec; always inspect the effective cgroup hierarchy (/sys/fs/cgroup/...) on a test node.

by treating cgroup limits as a non‑negotiable contract rather than an optional knob, you shrink the failure surface dramatically and make chaos tests meaningful. how do you enforce and validate resource caps in your clusters, and what observability patterns have saved you from noisy‑neighbor incidents?