Effective CPU requests of Kubernetes pods: the number kubectl does not show

On paper, your Kubernetes cluster has headroom: the CPU requests declared by your application containers barely add up to half the nodes’ capacity. Yet some pods stay Pending with an Insufficient cpu message, and the node autoscaler adds machines sooner than expected. Before blaming the scheduler, take a closer look at what Kubernetes actually reserves for each pod: its effective CPU requests.

The figure we instinctively add up (the sum of the application containers’ requests) is not the one the scheduler uses. Kubernetes computes an effective value at pod level, which takes into account init containers, native sidecars and, more recently, resources declared directly on the pod. This value does not appear in any API field: you have to recompute it.

Dynatrace SaaS performs this calculation at ingest, for every pod, with no specific configuration as long as Kubernetes monitoring goes through an ActiveGate (Grail only). This article explains where the gap comes from, how to spot it with DQL in Grail and how to feed it into your capacity decisions.

Why the sum of containers sometimes lies

A pod goes through two phases. During the initialisation phase, init containers run one by one, each having to finish before the next. During the run phase, application containers run in parallel, alongside any native sidecars (init containers declared with restartPolicy: Always, which start early and stay active for the pod’s whole life).

The scheduler must guarantee that the pod can get through both phases on the same node. So it takes the maximum of:

• the sum of application containers and sidecars (run phase);
• the largest init container, plus the sidecars declared before it (initialisation phase).

The same logic applies to limits, which the kubelet then enforces on the pod’s cgroup. In most cases, init containers are lightweight (waiting for a dependency, fetching a secret) and the run phase dominates: the naive sum is correct. The trap appears when initialisation is hungrier than the application itself.

A concrete example

Take an inference service whose init container downloads and unpacks a large model, with a request of 2 cores, while the application container makes do with 250m once the model is in memory. kubectl describe pod shows 250m for the application container. The scheduler, however, reserved 2000m: eight times more. Multiply by twenty or so replicas and the gap amounts to entire nodes. Database schema migrations run in an init container have exactly the same effect.

Pod-level resources

Kubernetes 1.34 moved the PodLevelResources feature to beta, enabled by default. It lets you set requests and limits on the whole pod rather than container by container. Consequences to be aware of:

• a pod-level limit can be lower than the sum of the container limits: each container seems to have headroom, but the pod’s cgroup caps the whole. For memory, this means an OOM kill that the per-container view gave no hint of;
• a pod-level request can only raise the reservation: Kubernetes rejects a value lower than the sum of the containers;
• requests and limits, CPU and memory, are handled independently: the same pod can combine pod-level values and values computed from the containers.

How Dynatrace computes effective CPU requests at ingest

Resource metrics stay stored per container in Grail: dt.kubernetes.container.requests_cpu, dt.kubernetes.container.requests_memory and their equivalents for limits. Each application container and each sidecar keeps its own series. When a simple sum would give a wrong value, Dynatrace adds a compensation data point on the same metric, attached to the pod but with no container name or type. This delta is positive when initialisation dominates or when a pod-level request raises the reservation, and negative when a pod-level limit lowers the ceiling. Support depends on the ActiveGate version: native sidecars from 1.339, init container compensation from 1.343, pod-level resources and RuntimeClass overhead from 1.345.

The benefit of this approach: your existing aggregations (sum per pod, per workload, per namespace, per node) become correct without rewriting, and per-container values stay intact for detailed analysis.

In practice

Step 1 — Find the affected pods

In a Notebook, look for the compensation rows: they are the ones without a dt.smartscape.container dimension. Note that k8s.cluster.name, k8s.pod.name or k8s.workload.name are entity properties, not metric dimensions: you group by dt.smartscape.*, then fetch those properties through a lookup on Smartscape.

timeseries cpu = sum(dt.kubernetes.container.requests_cpu),
  by: { dt.smartscape.k8s_pod, dt.smartscape.container },
  from: now()-30m
| filter isNull(dt.smartscape.container)
| lookup [ smartscapeNodes "*" | filter type == "K8S_POD"
    | fields id, cluster = k8s.cluster.name, pod = k8s.pod.name ],
  sourceField: dt.smartscape.k8s_pod, lookupField: id
| fieldsAdd cluster = lookup.cluster, pod = lookup.pod, delta_millicores = arrayLast(cpu)
| filter cluster == "prod-eu"
| fields pod, delta_millicores
| sort delta_millicores desc

CPU values for this metric are expressed in millicores. The highest deltas point to pods whose initialisation costs far more than their run phase.

Step 2 — Compare the displayed value with the effective value

For a suspect workload, display the per-container breakdown and the total. The sum of all rows, including the one without a container, gives the effective reservation.

timeseries cpu = sum(dt.kubernetes.container.requests_cpu),
  by: { dt.smartscape.k8s_pod, dt.smartscape.container, k8s.container.type },
  from: now()-30m
| lookup [ smartscapeNodes "*" | filter type == "K8S_POD"
    | fields id, wl = k8s.workload.name, pod = k8s.pod.name ],
  sourceField: dt.smartscape.k8s_pod, lookupField: id
| fieldsAdd wl = lookup.wl, pod = lookup.pod, millicores = arrayLast(cpu)
| filter wl == "model-server"
| fields pod, dt.smartscape.container, k8s.container.type, millicores
| sort pod asc

Repeat the exercise with dt.kubernetes.container.limits_memory for pods using pod-level resources: a negative delta signals a memory ceiling stricter than the containers suggest.

Step 3 — Fix the cause, not the symptom

Once the pods are identified, you have several options:

• move the heavy work out of the init container into a dedicated Kubernetes Job, run once per deployment rather than on every replica start;
• review the init container’s request: many are oversized « to be safe » when a download is mostly limited by the network;
• for pods with a pod-level memory limit, check that the sum of the containers’ actual consumption stays below the shared ceiling.

Step 4 — Recalibrate capacity dashboards

Check your Dashboards and capacity queries: if they already sum the metric per pod or per node, they automatically include the deltas. If they filter on k8s.container.type or exclude rows without dt.smartscape.container, they underestimate the reservation. A tile showing the « gap between the application sum and the effective reservation » per namespace is a good indicator to track.

Best practices

• Base node pool sizing on the effective reservation, never on the sum of application containers alone.
• Bear in mind that HPA thresholds expressed as percentages rely on container requests: the effective reservation is what matters for placement and capacity.
• Group by the dt.smartscape.* dimensions and attach cluster, namespace or workload through a lookup: filtering directly on k8s.cluster.name in a timeseries does not work.
• Before adopting PodLevelResources at scale, test it on a few workloads and watch for negative deltas on memory limits.
• Document workloads with heavy initialisation in your capacity reviews: they are the ones that skew estimates.

Frequently asked questions

Why does kubectl not show the pod’s effective request?

Because that value does not exist in any API field: the scheduler computes it from the application containers, init containers, native sidecars and any pod-level resources. kubectl describe pod shows the values declared container by container.

Do you need to rewrite your capacity DQL queries?

Not if they already sum the metric per pod, workload, namespace or node: the compensation deltas are included. Yes if they filter on k8s.container.type or exclude rows without dt.smartscape.container, because they then underestimate the reservation.

What does a negative delta mean?

A limit set at pod level is lower than the sum of its containers’ limits. Each container seems to have headroom, but the pod’s cgroup caps the whole: for memory, that is an OOM kill risk the per-container view does not show.

Conclusion

The per-container figure is accurate, but it does not answer the question a platform team actually asks: how much has the cluster really set aside? By computing the effective value at ingest and exposing it as a delta in the existing metrics, Dynatrace SaaS makes that answer available through a simple DQL aggregation. As pod-level resources become widespread, this habit becomes essential to avoid scheduling surprises and unexplained OOM kills.

Further reading: Your pod may be requesting 25× more CPU than you think — and Kubernetes won’t tell you (Dynatrace blog).

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. SaaS features evolve quickly: check their availability in your environment and in the official documentation.

Originally published on span.observer. French version: https://span.observer/fr/requetes-cpu-effectives-pods-kubernetes/.