Kubernetes Memory on cgroup v2: Debugging OOM Kills and Evictions

A container is OOMKilled, but kubectl top never showed it reaching its limit. Or a Pod is evicted while your node dashboard still shows free memory. In both cases, the first question is which memory value the decision actually used.

This guide separates cgroup-limit OOM, node-side OOM and kubelet eviction, then shows how to inspect the evidence and review memory requests and limits. It assumes familiarity with Pod resources and basic PromQL.

TL;DR

• The working set is a heuristic, not a measurement of unreclaimable memory. It subtracts inactive_file from usage but still includes reclaimable active page cache and slab.
• Distinguish memcg OOM from global OOM using kernel logs. A container can be below its own limit when an ancestor cgroup reaches its limit or a node-side allocation fails. OOMKilled and SystemOOM alone cannot establish the cause.
• Eviction uses the kubelet's memory.available, not Linux MemAvailable. Check the kubelet's statistics and configured thresholds.
• Review requests and limits separately. Requests influence placement and victim selection. To address a cgroup limit, reduce usage or adjust the limit after checking node capacity.
• Check Memory QoS configuration as well as its feature gate. In v1.37, the gate defaults to enabled, but request-based protection and memory.high remain disabled by default.

This article targets cgroup v2, Kubernetes v1.37.1 and containerd. See Components and versions for exact versions. The kind examples use a v1.37.0 node image.

Use the investigation workflow for an incident, the metric map to understand a dashboard, or requests and limits for prevention. The kind examples explore reclaim and Memory QoS in more detail.

Identify the failure before choosing a metric

memcg OOM means an allocation cannot fit within a cgroup limit after reclaim. Global OOM means a node-side allocation cannot be satisfied after reclaim, potentially within NUMA or cpuset constraints. A container below its own limit can be affected by either an ancestor limit or a node-side failure.

Investigating OOM and eviction

Investigation workflow

Start with the observed termination or eviction, then compare metrics from the same time. Scrapes can miss the peak; kernel logs are stronger evidence of the OOM source than a sampled usage graph.

1. When a Pod was OOMKilled

• Check whether container_memory_usage_bytes is close to the limit. memcg OOM is evaluated against this value
• Compare with container_memory_rss to tell whether anonymous pages are the main cause. An OOM can occur even when RSS is low
• If the gap between the working set or usage and RSS is large, page cache, shmem or kernel memory may be pushing usage up
• If OOMKilled occurs despite headroom under the limit, suspect global OOM. Correlate SystemOOM with kernel logs and inspect ancestor limits. A missing event does not establish memcg OOM
• If you see no spike in the metrics, suspect a surge shorter than the scrape interval. See My working set is below the limit, but the container was OOMKilled for what to check

2. When a Pod was evicted

• If the message is The node was low on resource: memory., node-wide memory pressure is the cause
• Use the node-utilization query to check the kubelet-view node memory utilization. The node-exporter-based utilization alone is not enough to judge
• Check whether a Pod with a small (or unset) memory request relative to actual usage is on the same node. Compare each Pod's container_memory_working_set_bytes (cAdvisor) against kube_pod_resource_request{resource="memory"} (kube-scheduler). Note that Pods with large requests make the scheduler keep placement density low, so they are unlikely to be the cause of pressure

3. To prevent recurrence

• Set a memory request informed by workload history to avoid excessive placement density. PromQL for checking is collected in Align memory request with actual usage
• For containers where memcg OOM occurs, also review the limit
• Control placement with Pod anti-affinity or topology spread constraints so that Pods with large memory usage do not gather on the same node

Telling which path occurred

Using Kubernetes information

Start by narrowing down with what you can check using kubectl.

There is no OOM-specific event on the Pod side. You only see things like BackOff accompanying the restart.

Check it as follows. SystemOOM is an event attached to the Node, and since Node is not a namespaced object, the event itself is recorded in the default namespace.

# 1. Find which node the OOMKilled Pod is on
kubectl get pod -n <namespace> <Pod name> -o wide

Output:

NAME       READY   STATUS    RESTARTS      AGE   IP            NODE
myapp-...  1/1     Running   5 (2m ago)    1h    10.0.0.1      node-1

# 2. Check whether that node's events contain SystemOOM
kubectl describe node node-1

Output:

...
Events:
  Type     Reason     Age   From     Message
  ----     ------     ----  ----     -------
  Warning  SystemOOM  3m    kubelet  System OOM encountered, victim process: java, pid: 12345

# To list only events
kubectl get events -n default \
  --field-selector reason=SystemOOM,involvedObject.name=node-1

If SystemOOM exists, inspect kernel messages from the same time: constraint, oom_memcg, task_memcg and the killed PID. A parsing failure can also produce this event. Match the event and kernel log to the container's termination time (lastState.terminated.finishedAt).

Not finding SystemOOM is not evidence of memcg OOM. The same state results from event TTL expiry, a kubelet restart, or a missed event.

The event retention period is set by kube-apiserver's --event-ttl, and the default is 1 hour (options.go). So an old SystemOOM may no longer be available through kubectl. If you forward events to an external logging platform, check there.

On the node, inspect the kernel log:

sudo journalctl -k -b -o short-iso \
  | rg 'oom-kill:|Out of memory:|Memory cgroup out of memory:|Killed process'

CONSTRAINT_MEMCG and oom_memcg identify the limiting cgroup; task_memcg identifies the victim's cgroup. If these differ, inspect the Pod and ancestor limits. CONSTRAINT_CPUSET and CONSTRAINT_MEMORY_POLICY call for checking NUMA and cpuset restrictions. Align timestamps with the container's finishedAt.

If logs are unavailable, use the following as supporting evidence rather than proof:

• Whether container_memory_usage_bytes was close to the limit at that time (if so, memcg OOM is likely)
• Whether the kubelet-view node memory utilization was high at that time (if so, global OOM is likely)

Why OOMKilled cannot distinguish the paths

When a task exits with code 137, the container runtime (containerd) checks oom_kill in the cgroup's memory.events, and if it is 1 or more, it updates the termination reason to OOMKilled (events.go). The check is oomMetricsEventOccurred() in the same file.

    switch v := taskMetricsAny.(type) {
    case *cg1.Metrics:
        return v.GetMemoryOomControl().GetOomKill() > 0, nil
    case *cg2.Metrics:
        return v.GetMemoryEvents().GetOomKill() > 0, nil

The definition of oom_kill in memory.events is "The number of processes belonging to this cgroup killed by any kind of OOM killer". So a process chosen and killed by the kernel in a global OOM is counted too, and both paths produce OOMKilled.

        for event := range outStream {
            // Count every OOM kill per container to back the
            // container_oom_events_total metric.
            recordOOMKill(event.ContainerName)

            if event.VictimContainerName == recordEventContainerName {
                ...
                ow.recorder.WithLogger(logger).Eventf(ref, v1.EventTypeWarning, systemOOMEvent, "%s", eventMsg)
            }
        }

    if (memcg) {
        pr_cont(",oom_memcg=");
        pr_cont_cgroup_path(memcg->css.cgroup);
    } else
        pr_cont(",global_oom");

    containerRegexp = regexp.MustCompile(`oom-kill:constraint=(.*),nodemask=(.*),cpuset=(.*),mems_allowed=(.*),oom_memcg=(.*),task_memcg=(.*),task=(.*),pid=(.*),uid=(.*)`)

            oom := &OomInstance{
                ContainerName:       "/",
                VictimContainerName: "/",
                TimeOfDeath:         msg.Timestamp,
            }

How it appears in metrics

Metrics do not distinguish the path with a label. The kernel's __oom_kill_process() records the same event regardless of the path.

Only container_oom_events_total behaves differently because, as described earlier, the parser cannot extract a cgroup in a global OOM and treats it as the root (/). This is only a result of whether the log could be parsed, so do not use it to determine the path; confirm the source from constraint and oom_memcg in the kernel log.

Note: Although container_oom_events_total has container_ in its name, in v1.37 it is counted by the kubelet, not cAdvisor (recordOOMKill() in oom_counter.go). The meaning of the value does not change, but you have to look elsewhere when reading the code.

What to look at after narrowing down

• If you judge it to be a global OOM, inspect node-side memory pressure and any NUMA or cpuset constraints. Compare the memory requests and actual usage of the Pods placed on the same node, and check whether any Pod has too small a request
• If you judge it to be a memcg OOM, review that container's memory limit and its actual usage. If the container seems to have headroom under its limit, it may have reached the limit of the Pod as a whole or of /kubepods.slice
• If you cannot decide, keep both possibilities open and check both the container's limit and usage and the node's memory utilization

    nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,
                            gfp_mask, reclaim_options, NULL);
    psi_memstall_leave(&pflags);

    if (mem_cgroup_margin(mem_over_limit) >= nr_pages)
        goto retry;

// rankMemoryPressure orders the input pods for eviction in response to memory pressure.
// It ranks by whether or not the pod's usage exceeds its requests, then by priority, and
// finally by memory usage above requests.
func rankMemoryPressure(pods []*v1.Pod, stats statsFunc) {
    orderedBy(exceedMemoryRequests(stats), priority, memory(stats)).Sort(pods)
}

    if types.IsNodeCriticalPod(pod) {
        // Only node critical pod should be the last to get killed.
        return guaranteedOOMScoreAdj
    }

Checking with PSI (Pressure Stall Information)

This is an auxiliary indicator for the case where things are slow although usage has not reached the limit. It shows how long processes were actually stalled waiting for memory reclaim.

# Per Pod, the share of time (%) all processes were stalled waiting for memory
100 * max by (namespace, pod) (
  rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m])
)

# Per Pod, the share of time (%) some processes were waiting for memory
100 * max by (namespace, pod) (
  rate(container_pressure_memory_waiting_seconds_total{container="", image="", pod!=""}[5m])
)

# Share of time (%) the whole node was completely stalled waiting for memory
100 * max by (node) (
  rate(container_pressure_memory_stalled_seconds_total{id="/"}[5m])
  * on(instance) group_left(node) kubelet_node_name
)

# Containers whose usage is under 80 % of the limit but that are stalled waiting for memory
(
  100 * rate(container_pressure_memory_stalled_seconds_total{container!="", image!=""}[5m]) > 1
)
and
(
  container_memory_usage_bytes{container!="", image!=""}
  / container_spec_memory_limit_bytes{container!="", image!=""} < 0.8
)

- alert: ContainerMemoryStalled
  # Fires when the 5-minute average of fully stalled time is 5 % or more for 10 minutes
  expr: |
    100 * max by (namespace, pod) (
      rate(container_pressure_memory_stalled_seconds_total{container="", image="", pod!=""}[5m])
    ) > 5
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "A Pod is stalled waiting for memory allocation (add the namespace and pod labels with your own template)"
    description: "It may be made to wait by reclaim even though it has not reached its memory limit. Check the gap between usage and RSS and the amount of page cache."

Quick reference: contradictions explained and a metric map

The working set, cgroup usage, process RSS and node availability answer different questions. Use this map to identify the value behind the dashboard before comparing it with a limit or threshold. For a non-root cgroup, working set is max(memory.current - inactive_file, 0). It still includes some reclaimable memory.

Looking at the same node, utilization computed from node_memory_MemAvailable_bytes does not match the utilization the kubelet uses for eviction decisions. With workloads that do a lot of file I/O or logging, the difference can reach several GiB.

Different emitters also mean different labels on the metrics. cAdvisor metrics carry no label identifying the node, so when you compute per-node utilization you need to align labels with kubelet_node_name, as shown later.

Kubernetes memory usage and cgroup v2

Why the numbers differ

A process RSS measurement includes its resident mappings, including shared and file-backed pages. Cgroup accounting charges memory to a group, while working set subtracts one category of file cache from usage. An application's heap metric covers a different subset again.

Reclaimability is a separate question: clean file cache can often be reclaimed, dirty pages may need writeback, and anonymous memory normally needs swap or explicit release by the process. Calling free does not necessarily return allocator-held memory to the OS.

These differences explain why heap size, process RSS, cgroup RSS and working set need not match. The following sections connect those values to cgroup v2 accounting.

Memory breakdown in cgroup v2

Container memory usage is managed in a per-container cgroup. In cgroup v2, memory.current is the current usage and memory.stat is its breakdown.

# Example values in a container's cgroup (checked on the node)
cat /sys/fs/cgroup/<container cgroup>/memory.current

Output:

5368709120

cat /sys/fs/cgroup/<container cgroup>/memory.stat

Output:

anon 2147483648           # anonymous pages (heap, stack: memory not backed by a file)
file 3087007744           # page cache (including tmpfs and shared memory)
shmem 268435456           # the part of file that is tmpfs or shared memory
...
inactive_anon 134217728    # anon LRU: not accessed for a while
active_anon 2013265920     # anon LRU: recently accessed
inactive_file 1073741824   # file LRU: not accessed for a while
active_file 1744830464     # file LRU: recently accessed
unevictable 268435456      # pages that cannot be reclaimed (here, the tmpfs part)
...

The following relationships hold in this example. Note that inactive_file + active_file does not equal file; it equals file - shmem. The next section explains why.

anon + file + kernel memory = 5368709120 (memory.current)
inactive_file + active_file = 2818572288 = file - shmem
inactive_anon + active_anon = 2147483648 = anon        (shmem is not in here)
unevictable                 =  268435456 = shmem       (tmpfs without swap)

The main fields of memory.stat are:

anon (anonymous pages)

• Memory not backed by a file. Regions the application allocated with malloc and similar belong here.
• Unlike page cache, the kernel cannot reclaim it independently of the process. With swap it can write pages out and free up space, but on nodes that do not use swap it is memory the kernel cannot reclaim under pressure. When swap is unavailable, the kernel sets scan_balance = SCAN_FILE and excludes the anon LRU from scanning (get_scan_count() in mm/vmscan.c).
• On the other hand, this value drops if the process itself returns memory to the OS with munmap or madvise(MADV_DONTNEED). Note that calling free does not reduce it unless the allocator or language runtime returns the memory to the OS.

file (page cache)

• Contents of files held in memory by the kernel during reads and writes. The application did not allocate it explicitly.
• When memory runs short the kernel discards (reclaims) it automatically, so it is basically reclaimable memory.
• Cache of files on disk is split into active_file (recently accessed) and inactive_file (not accessed for a while). Reclaim takes inactive_file first.
• active_file + inactive_file does not equal file. These are totals of the LRU lists, and as described below shmem is on the anon LRU rather than the file LRU. Besides shmem, pages that fall off the LRU, such as mlocked pages, also show up as a difference.
• It includes not only the cache of files on disk but also shmem (tmpfs and shared memory), because Linux treats these as page cache too. shmem is written out to swap rather than to a file, so on nodes without swap it cannot be reclaimed. The breakdown is visible as shmem in memory.stat.

Warning: shmem is counted in file, but it is not on the file LRU. The kernel classifies swap-backed pages as anon, and tmpfs pages are among them. folio_is_file_lru() in mm_inline.h is defined as "0 if @folio is a normal anonymous folio, a tmpfs folio or otherwise ram or swap backed folio".

The counters they are charged to are different, though. A tmpfs page is added to both NR_FILE_PAGES (= file) and NR_SHMEM (= shmem) (mm/shmem.c):

  __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr);
  __lruvec_stat_mod_folio(folio, NR_SHMEM, nr);

Furthermore, for a Kubernetes emptyDir with medium: Memory, the kubelet mounts tmpfs with the noswap option when the node supports it (generateTmpfsMountOptions() in empty_dir.go). The kernel makes the inode of a noswap tmpfs unevictable (shmem_get_inode() in mm/shmem.c), so these pages sit on the unevictable LRU, not the anon LRU.

  if (sbinfo->noswap)
      mapping_set_unevictable(inode->i_mapping);

So shmem is accounted as follows:

mount	LRU	Where it shows up in memory.stat
emptyDir with medium: Memory (noswap)	unevictable	unevictable
Any other tmpfs / shared memory	anon LRU	inactive_anon / active_anon

In neither case does it enter inactive_file / active_file. That matters in two ways later:

• inactive_file + active_file generally corresponds to file - shmem; unevictable pages and asynchronous statistics can cause differences
• Only inactive_file is subtracted from the working set, so shmem stays in the working set in full

With workloads that write a lot of logs or read and write large files, the page cache can grow to several GiB even though the application did not intend that.

Note: memory.current includes memory the kernel itself uses (slab, socket buffers and so on) in addition to anon and file. So adding anon and file alone does not give memory.current.

How cgroup v2 memory controls map to Pod settings

cgroup v2 memory control interface

cgroup v2 offers four levels of control over memory usage. memory.min and memory.low are protections ("do not reclaim up to here"), and memory.high and memory.max are limits ("do not allow use beyond here").

Other files that report state:

• memory.current: current usage (what container_memory_usage_bytes is backed by)
• memory.stat: usage breakdown
• memory.events: counts of low, high, max, oom, oom_kill and oom_group_kill
• memory.swap.max: swap limit
• memory.pressure: PSI for waiting on memory allocation

For exact definitions of each file see the kernel's Memory Interface Files, and for how Kubernetes treats cgroup v2 see About cgroup v2.

How Pod settings are reflected

Pod settings are reflected into cgroup v2 as follows.

cgroup v2 is a directory tree. Its top is the root cgroup (/sys/fs/cgroup), where any process that is in no child cgroup belongs, so it contains not only Pods but every process on the node, such as the kubelet and containerd. Kubernetes creates the hierarchy below it: "all Pods on the node → Pod → container".

# With the systemd cgroup driver (the kubeadm default; names differ with the cgroupfs driver, see below)
/sys/fs/cgroup                                  ← root cgroup (id="/")
│                                                 contains every process on the node
│                                                 the value used for eviction decisions
│
├── system.slice/                               kubelet, containerd, sshd, etc.
│   ├── kubelet.service                           (processes that are not Pods)
│   └── containerd.service
│
└── kubepods.slice/                             ← all Pods on the node
    │                                             (id="/kubepods.slice")
    ├── kubepods-pod<UID>.slice/                ← Pod (Guaranteed)
    │   ├── cri-containerd-<ID>.scope           ← container
    │   └── cri-containerd-<pause ID>.scope     ← pause container
    │
    ├── kubepods-burstable.slice/                 (intermediate slice per QoS class)
    │   └── kubepods-burstable-pod<UID>.slice/  ← Pod (Burstable)
    │       └── cri-containerd-<ID>.scope       ← container
    │
    └── kubepods-besteffort.slice/
        └── kubepods-besteffort-pod<UID>.slice/ ← Pod (BestEffort)
            └── cri-containerd-<ID>.scope       ← container

When filtering metrics, the root cgroup is id="/", all Pods on the node are id="/kubepods.slice", a Pod is container="", image="", pod!="", and a container is container!="", image!="". What the kubelet uses for eviction decisions is the working set of the topmost root cgroup (see below). That is a different level from memcg OOM caused by a container or Pod exceeding its limit.

Memory limits are set not only on containers but also on higher levels.

The examples here use container-level resources and omit Pod-level resources. Without a Pod-level limit, the required container limits must be present for a finite Pod limit to be set. A Pod's effective limit is not simply the sum of its regular containers' limits.memory. If there are init containers, their maximum is also considered, and spec.overhead is added if set. For the calculation see Resource sharing within containers and Pod Overhead.

So even if a container has not reached its own limit, an OOM kill can occur when the Pod as a whole or /kubepods.slice hits its limit.

Note: The paths above assume the kubelet uses the systemd cgroup driver. With the cgroupfs driver, paths look like /kubepods/burstable/pod<UID>/<container ID>, with no .slice or .scope and no cri-containerd- prefix on the container.

The kubelet's own default is cgroupfs, but kubeadm sets systemd when the value is empty (defaults.go, kubelet.go). In v1.37, a supported runtime's CRI RuntimeConfig response takes precedence through driver auto-detection. Check the effective driver with:

$ kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz" \
  | jq '.kubeletconfig.cgroupDriver'
"systemd"

The id label of cAdvisor metrics also changes with the cgroup driver, so when filtering in PromQL, check the value in your own cluster with this query:

group by (id) (container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"})

A Pod's cgroup path depends on its QoS class. As the tree above shows, Burstable and BestEffort go through an intermediate slice per QoS class.

In the <UID> part, systemd escaping replaces the hyphens of the Pod UID with underscores (example: 6983351e-4ede-... → pod6983351e_4ede_...). The UID of a static Pod (mirror Pod) is a hash with no hyphens, so it looks like pod1ff70b81885169da4e9e32c9530b77c9.slice.

The QoS class and UID of a Pod can be read from the Pod resource, and once you have those two you can assemble the cgroup path.

$ kubectl get pod -n <namespace> <Pod name> -o jsonpath='{.status.qosClass}{"\n"}{.metadata.uid}{"\n"}'
Burstable
6983351e-4ede-4611-8bab-b986614af18d

# To list them
$ kubectl get pods -n <namespace> -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'

Container metrics

The cAdvisor built into the kubelet exposes the cgroup values above as Prometheus metrics. They are served from the kubelet's /metrics/cadvisor endpoint, and you do not need to run a separate cAdvisor Pod or DaemonSet.

Note: The kubelet has endpoints such as /metrics, /metrics/cadvisor and /metrics/resource (server.go). The last two relate to container and node usage. /metrics/cadvisor exposes the container_* metrics above, while /metrics/resource exposes only CPU, memory and swap usage (resource_metrics.go). The latter has container_memory_working_set_bytes and also per-Pod and per-node series (pod_memory_working_set_bytes, node_memory_working_set_bytes) (see podMemoryUsageDesc and nodeMemoryUsageDesc in resource_metrics.go). When to use which for per-Pod usage is covered in Align memory request with actual usage.

The important thing here is the definition of the working set.

container_memory_working_set_bytes = max(
    container_memory_usage_bytes - container_memory_total_inactive_file_bytes,
    0,
)

In other words, only inactive_file is subtracted from usage, with the result clamped to zero; active page cache remains in the working set. For the implementation see cAdvisor's handler.go.

    workingSet := ret.Memory.Usage
    if v, ok := s.MemoryStats.Stats[inactiveFileKeyName]; ok {
        ret.Memory.TotalInactiveFile = v
        if workingSet < v {
            workingSet = 0
        } else {
            workingSet -= v
        }
    }
    ret.Memory.WorkingSet = workingSet

As a breakdown, it is roughly the sum of the following. RSS (container_memory_rss) is exactly anon in memory.stat under cgroup v2, that is, the amount of anonymous pages, so it is included in the working set.

container_memory_working_set_bytes
  ≒ container_memory_rss                        (anonymous pages; shmem not included)
  + shmem                                       (tmpfs and shared memory; stays in full)
  + container_memory_total_active_file_bytes    (active page cache)
  + unreclaimable pages, kernel memory, etc.    (unevictable, slab, socket buffers, ...)

shmem is listed as a separate term because neither container_memory_rss (= anon = NR_ANON_MAPPED) nor active_file includes shmem. As described above, shmem is not on the file LRU, so it never appears in active_file / inactive_file, and since only inactive_file is subtracted from the working set, it remains in full. No container_* metric is exposed for shmem, so this term can only be checked from memory.stat on the node.

The approximation reflects overlapping accounting categories and LRU classifications in memory.stat. cAdvisor also exposes container_memory_kernel_usage, but this alone cannot reconstruct the full breakdown. Read memory.stat directly for more detail.

Caution: RSS in container_memory_rss stands for Resident Set Size, but its scope differs from the real meaning. True RSS means "the pages a process holds in physical memory" and includes mmaped file pages.

But memory.stat in cgroup v2 has no rss field, and cAdvisor exposes anon as container_memory_rss. The kernel documentation defines anon as "Amount of memory used in anonymous mappings such as brk(), sbrk(), and mmap(MAP_ANONYMOUS)", which does not include file mappings. So it does not match the RSS column of ps or VmRSS in /proc/[pid]/status.

Warning: A large container_memory_working_set_bytes does not necessarily mean the application needs that much memory. It may just be accumulated page cache. To see how much memory the application has actually allocated, compare against container_memory_rss.

Node metrics

There are two families of metrics representing whole-node memory usage, with different emitters.

From cAdvisor (the kubelet's view)

cAdvisor exposes memory usage not only for containers but also for the root cgroup (id="/", the topmost cgroup that contains every process on the node, as described above) and for the cgroup of all Pods. The id of the all-Pods cgroup depends on the cgroup driver, so the queries below match both values with =~.

• container_memory_working_set_bytes{id="/"}: working set of the whole node (Pods + other processes). But the root cgroup has no memory.current, so the backing value differs from containers (see below)
• container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"}: working set of all Pods on the node
• container_memory_total_active_file_bytes{id="/"}: active page cache of the whole node

Both the MEMORY(bytes) column of kubectl top node and the kubelet's eviction decision described below use this working set.

From node-exporter (/proc/meminfo as is)

node-exporter turns the contents of /proc/meminfo directly into metrics.

• node_memory_MemTotal_bytes: total physical memory on the node
• node_memory_MemAvailable_bytes: the kernel's estimate of "how much can be allocated when new memory is requested"
• node_memory_Cached_bytes: page cache size. As the kernel documentation says, "In-memory cache for files read from the disk (the pagecache) as well as tmpfs & shmem", it includes tmpfs and shared memory. The breakdown is in node_memory_Shmem_bytes

node_memory_MemAvailable_bytes counts reclaimable page cache as free. Computing node memory utilization as 1 - MemAvailable / MemTotal is also in node-exporter's official mixin as the NodeMemoryHighUtilization alert (alerts.libsonnet). Its expr uses variables for the selector and threshold; stripped of those it is 100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100) > <threshold>. However, as shown below, this value does not match the utilization the kubelet uses for eviction decisions.

Values the kubelet uses for eviction decisions

The kubelet normally evaluates memory availability every 10 seconds. Crossing a threshold contributes to MemoryPressure; eviction also depends on soft-threshold grace periods and reclaim handling. This memory.available is not /proc/meminfo MemAvailable: it is computed from the root cgroup's working set.

memory.available = node_memory_MemTotal_bytes - container_memory_working_set_bytes{id="/"}

For the implementation see Kubernetes' helper.go and helpers_others.go. The 10-second interval is defined as evictionMonitoringPeriod.

Warning: The root cgroup working set used for eviction decisions (and by kubectl top node) is not the same thing as a container's working set. In cgroup v2, memory.current has the CFTYPE_NOT_ON_ROOT flag, so the root cgroup has no memory.current (memory_files[] in mm/memcontrol.c).

  {
      .name = "current",
      .flags = CFTYPE_NOT_ON_ROOT,
      .read_u64 = memory_current_read,
  },

memory.stat, on the other hand, has no such flag, so it can be read even on the root. The cgroup library cAdvisor uses therefore substitutes anon + file from memory.stat as the usage when reading memory.current fails with ENOENT, for the root only (rootStatsFromMeminfo() in opencontainers/cgroups).

  // sum `anon` + `file` to report the same value as `usage_in_bytes` in v1.
  stats.MemoryStats.Usage.Usage = stats.MemoryStats.Stats["anon"] + stats.MemoryStats.Stats["file"]

Kubernetes v1.37.1 selects opencontainers/cgroups v0.0.7. For a real host root cgroup, the id="/" series is therefore:

container_memory_usage_bytes{id="/"}       = anon + file
container_memory_working_set_bytes{id="/"} = anon + file - inactive_file

Unlike memory.current, this includes no kernel memory at all, such as slab, pagetables, kernel_stack, percpu and sock. This is mainly why the node working set tends to come out smaller than usage derived from /proc/meminfo. For container and Pod cgroups memory.current can be read directly, so this substitution does not occur.

Because the working set includes active page cache, the kubelet treats active page cache as "memory in use".

For memory on Linux, the default is the hard-eviction threshold memory.available<100Mi, as in DefaultEvictionHard in defaults_linux.go; no soft eviction is configured by default.

Choose thresholds appropriate to the node size and workload. In practice, though, I think it is better to override this value explicitly. In the v1.37 cluster I used for verification I also pass the following flags to the kubelet to set both hard and soft thresholds.

--eviction-hard=memory.available<5%,nodefs.available<5%,pid.available<5%
--eviction-soft=memory.available<10%,nodefs.available<10%,pid.available<10%
--eviction-soft-grace-period=memory.available=2m,nodefs.available=5s,pid.available=1m

The default 100Mi is a fixed value, so it becomes relatively thinner on nodes with more memory. If you specify percentages, the threshold follows the node size. Placing soft eviction ahead of hard (10 %) and giving it a grace period with --eviction-soft-grace-period avoids evicting Pods on momentary spikes while still evicting them when the pressure persists.

You can check your cluster's settings with the following query.

Setting only some evictionHard signals does not inherit the unspecified defaults unless mergeDefaultEvictionSettings is enabled. Include all required disk and memory signals or check that setting.

kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz" \
  | jq '.kubeletconfig | {evictionHard, evictionSoft, evictionSoftGracePeriod}'

Output:

{
  "evictionHard": {
    "memory.available": "5%",
    "nodefs.available": "5%",
    "pid.available": "5%"
  },
  "evictionSoft": {
    "memory.available": "10%",
    "nodefs.available": "10%",
    "pid.available": "10%"
  },
  "evictionSoftGracePeriod": {
    "memory.available": "2m",
    "nodefs.available": "5s",
    "pid.available": "1m"
  }
}

Note: The denominator is Capacity (node.status.capacity.memory, kube_node_status_capacity{resource="memory"}), not Allocatable. These normally match node_memory_MemTotal_bytes; verify that the metrics describe the same host and cgroup namespace. The kubelet uses cAdvisor's value as the machine's memory capacity, and cAdvisor reads it from MemTotal in /proc/meminfo (raw/handler.go). A threshold like memory.available<5% is also evaluated as a percentage of this Capacity.

Allocatable is Capacity minus the system reservation and the hard eviction threshold, and is the upper bound the scheduler uses when assigning Pod requests. It is not used to decide whether eviction happens. See Reserve Compute Resources for System Daemons for details.

Why node-exporter and kubelet usage disagree

Putting the above together, there are two views of the same node.

MemAvailable is the kernel's estimate of "how much can be newly allocated", and it adds part of the page cache and the reclaimable slab to free memory. It does not count the whole page cache as free. The implementation is si_mem_available() in mm/show_mem.c.

    /*
     * Estimate the amount of memory available for userspace allocations,
     * without causing swapping or OOM.
     */
    available = global_zone_page_state(NR_FREE_PAGES) - totalreserve_pages;

    /*
     * Not all the page cache can be freed, otherwise the system will
     * start swapping or thrashing. Assume at least half of the page
     * cache, or the low watermark worth of cache, needs to stay.
     */
    pagecache = global_node_page_state(NR_ACTIVE_FILE) +
        global_node_page_state(NR_INACTIVE_FILE);
    pagecache -= min(pagecache / 2, wmark_low);
    available += pagecache;

    /*
     * Part of the reclaimable slab and other kernel memory consists of
     * items that are in use, and cannot be freed. Cap this estimate at the
     * low watermark.
     */
    reclaimable = global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B) +
        global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE);
    reclaimable -= min(reclaimable / 2, wmark_low);
    available += reclaimable;

Three things can be read from this.

• What is subtracted is min(half, watermark_low), so at least half of the page cache and the reclaimable slab is counted as free
• From the free pages, totalreserve_pages is subtracted, not watermark_low
• The page cache term is the file LRU (NR_ACTIVE_FILE + NR_INACTIVE_FILE), so shmem, which sits on the anon LRU, is never counted as free. On nodes that use tmpfs heavily, MemAvailable also comes out small

The important point is that the definitions differ, and the size and direction of the difference depend on the environment. There are four factors behind the divergence, and they push in opposite directions.

These rows are not independent additive terms; some describe the same cache treatment from different angles. Compare statistics from the same time rather than counting how many factors favor either side.

The kubelet side is higher on nodes running workloads where active_file builds up easily, for example:

• Batch jobs and ETL that read and write large files
• Services that emit a lot of logs (stdout is written to files on the node by the container runtime, so it becomes page cache)
• Middleware designed around the page cache (Kafka, Elasticsearch, etc.)
• Log collection agents that keep reading log files on the node

What matters is the amount of file access, not the language or runtime. For example, a JVM heap is anonymous pages, so it raises RSS, not active_file.

On such a node it can happen that utilization computed from MemAvailable looks like 33 % while the kubelet's side has reached 53 %. If you look at the MemAvailable-based utilization and conclude there is plenty of room, the kubelet may in fact be approaching its threshold and evictions will occur.

On the other hand, on a node where page cache has not built up much:

MemTotal          3.820 GiB
MemAvailable      2.692 GiB   →  1 - 2.692/3.820 = 29.6 %  (node-exporter side)
usage(/)          2.557 GiB   (= anon + file of the root)
inactive_file(/)  1.622 GiB
active_file(/)    0.227 GiB
Working Set(/)    0.935 GiB   →  0.935/3.820     = 24.5 %  (kubelet side)

active_file is only 0.23 GiB while inactive_file is 1.6 GiB, so the working set removes it all as "free", and the kubelet side comes out about 5 points lower. The kubelet side is not always higher.

Either way, the conclusion is that you cannot judge whether there is headroom from only one of the values. active_file is an auxiliary indicator for confirming the cause of the divergence, and its value is not itself the difference. When investigating node memory pressure, also check the kubelet-view utilization with the following PromQL.

# Node memory utilization from the kubelet's view (the value used in eviction decisions)
max by (node) (
  container_memory_working_set_bytes{id="/"}
  * on(instance) group_left(node) kubelet_node_name
)
/ max by (node) (kube_node_status_capacity{resource="memory"})

# Amount of active page cache, the main cause of the divergence
max by (node) (
  container_memory_total_active_file_bytes{id="/"}
  * on(instance) group_left(node) kubelet_node_name
)

Note: The label alignment is there because no cAdvisor container_* metric carries any label identifying the node. Looking directly at the kubelet's /metrics/cadvisor, the root cgroup series looks like this:

container_memory_working_set_bytes{container="",id="/",image="",name="",namespace="",pod=""} 8.92522496e+08

The only thing that tells you the node is the instance label Prometheus adds at scrape time, and labels such as kubernetes_io_hostname are added by your Prometheus relabel configuration. Because the name depends on the relabel configuration (or may not exist at all), referring to it with label_replace is environment specific.

So the queries above use kubelet_node_name, which the kubelet exposes on /metrics.

kubelet_node_name{node="memqos-control-plane"} 1

This is a gauge whose value is always 1, and the node name is in the node label. The value itself is meaningless; it is the info metric pattern, which carries the information in labels. The kubelet's help text says the same.

  // NodeName is a Gauge that tracks the node's name. The count is always 1.
  NodeName = metrics.NewGaugeVec(
      &metrics.GaugeOpts{
          Subsystem: KubeletSubsystem,
          Name:      NodeNameKey,
          Help:      "The node's name. The count is always 1.",

In OpenMetrics, Info is defined as an independent metric type, with a mandatory _info suffix and a sample value that is always 1 (OpenMetrics spec - Suffixes).

Info metrics are used to expose textual information which SHOULD NOT change during process lifetime.

kubelet_node_name has no _info suffix and is exposed as a gauge, so strictly speaking it is not an OpenMetrics Info type. But "value always 1, information in labels" is the same usage, so this article treats it as an info-metric pattern. Similar examples are kube-state-metrics' kube_node_info and node-exporter's node_uname_info. When both endpoints are scraped with matching target labels, instance matches, and * on(instance) group_left(node) brings in the node label. And the resulting node label has the same name as kube-state-metrics' node label, so they join directly.

If your environment adds its own node label by relabeling, you can use label_replace as follows (replace kubernetes_io_hostname with the value in your environment).

max by (node) (
  label_replace(
    container_memory_working_set_bytes{id="/"},
    "node", "$1", "kubernetes_io_hostname", "(.*)"
  )
)

Note: Eviction candidates are not limited to Pods in particular namespaces. When node memory gets tight, Pods in every namespace on that node are candidates. Which Pod is chosen is decided in this order: (1) whether usage exceeds the memory request, (2) Pod priority, (3) how far above the request it is. See Pod selection for kubelet eviction for details.

However, critical Pods are exempt from node-pressure eviction by the kubelet (evictPod() in eviction_manager.go). "Critical pod" is a term in the official Kubernetes documentation and refers to a Pod whose PriorityClass is system-cluster-critical or system-node-critical (Guaranteed Scheduling For Critical Add-On Pods says "To mark a Pod as critical, set priorityClassName for that Pod to system-cluster-critical or system-node-critical.").

  if kubelettypes.IsCriticalPod(pod) {
      logger.Error(nil, "Eviction manager: cannot evict a critical pod", "pod", klog.KObj(pod))
      return false
  }

The kubelet's check is broader than this definition: IsCriticalPod() is also true for static Pods and mirror Pods. The remaining condition is that pod.Spec.Priority is at least SystemCriticalPriority (= 2 × 1000000000), and since the upper limit of user-definable priority is HighestUserDefinablePriority (= 1000000000), the only PriorityClasses that match are the two above. These priority classes are exempt from node-pressure eviction, but that exemption does not itself determine their protection during global OOM. See Which process gets killed for details.

What to do about it

Once the cause is clear, review requests and placement for node pressure, and usage and limits for memcg OOM.

Start with the limiting cgroup

For memcg OOM, identify the cgroup named by oom_memcg before changing resources. It may be the container, its Pod or an ancestor such as /kubepods.slice.

Compare memory.current with that cgroup's memory.max, then inspect memory.stat to see whether anonymous memory, shmem, file cache or kernel memory dominates. Reduce the required memory or adjust the relevant limit after checking node capacity. Raising a request alone does not raise that limit.

Source: cgroup v2 memory interfaces.

Align memory request with actual usage

Aligning memory requests with actual usage helps prevent excessive placement density on a node.

With the default v1.37 kubelet configuration, resources.requests.memory is not applied as cgroup memory protection, so it does not directly limit how much memory a container can use. It nevertheless matters because the request determines how likely OOM kills and evictions are, in these three ways:

• Placement (how likely both global OOM and eviction are): the standard resource-fit check uses requests and Allocatable rather than actual usage. Affinity, taints and other rules also affect placement. A Pod with no request, or one smaller than actual usage, is placed while overestimating the node's free memory. As a result, Pods with large memory usage concentrate on the same node and the whole node becomes tight, inviting both global OOM and eviction
• Eviction candidate selection: the kubelet preferentially evicts Pods whose actual usage exceeds their memory request. If you set the request smaller than actual usage, that Pod is evicted more easily
• oom_score_adj (the order of kills in a global OOM): as described earlier, the Burstable oom_score_adj is 1000 - 1000 × memory request / node memory capacity. The smaller the request, the larger the oom_score_adj, and the more likely the Pod is chosen as a kill target in a global OOM. BestEffort Pods receive 1000. An unset memory request alone does not imply BestEffort: CPU resources can make a Pod Burstable, and a memory limit can default its request

"Whether the event happens" and "which Pod is chosen" are separate matters, so organized by effect:

Only placement affects whether an event occurs, and it affects both global OOM and eviction, because both start from node pressure. The effect is indirect, though: the request does not appear in the computation of memory.available (the kubelet's memory-availability calculation), so raising a request does not relax the threshold. The node just becomes less likely to get tight as a result of lower placement density.

The other two only affect whether your Pod is chosen after the pressure has built up. With the default settings, increasing a request does not raise memory.max or resolve a memcg limit violation.

Raising requests can reduce placement density and alter victim selection. It can also leave Pods Pending or trigger node expansion, so account for capacity and cost.

Warning: Increasing a request alone does not resolve memcg OOM. Reduce usage or review the limiting cgroup's limit after checking node capacity. If Memory QoS protection or throttling is configured, requests also affect reclaim behavior. This section focuses on reducing global OOM and eviction.

PromQL for comparing requests and actual usage

You can check whether a request matches actual usage with PromQL. I included queries for the per-Pod ratio, for finding Pods with no request, for per-node comparison, and for deriving a recommended value from past usage. I also explain the pitfalls that come from where the metrics originate (such as double counting).

        podUID := types.UID(podStats.PodRef.UID)
        // Lookup the pod-level cgroup's CPU and memory stats
        podInfo := getCadvisorPodInfoFromPodUID(podUID, allInfos)
        if podInfo != nil {
            cpu, memory := cadvisorInfoToCPUandMemoryStats(podInfo)
            podStats.CPU = cpu
            podStats.Memory = memory

# Actual per-Pod usage (when scraping the kubelet's /metrics/resource)
pod_memory_working_set_bytes

# Actual per-Pod usage (looking at the Pod's cgroup series directly)
max by (namespace, pod) (
  container_memory_working_set_bytes{container="", image="", pod!=""}
)

Pod cgroup                666,116,096   ← correct value
Sum of real containers     665,882,624
pause container                225,280
sum{container="",pod!=""}  666,341,376   ← Pod cgroup + pause, double counted

# Actual per-Pod usage (when summing containers)
sum by (namespace, pod) (
  container_memory_working_set_bytes{container!="", image!=""}
)

# Ratio of observed working set to memory request (above 1 means usage exceeds the request)
max by (namespace, pod) (
  container_memory_working_set_bytes{container="", image="", pod!=""}
)
/
max by (namespace, pod) (
  kube_pod_resource_request{resource="memory", unit="bytes"}
)

# Amount over the request (bytes)
max by (namespace, pod) (
  container_memory_working_set_bytes{container="", image="", pod!=""}
)
-
max by (namespace, pod) (
  kube_pod_resource_request{resource="memory", unit="bytes"}
)

# Pods with no memory request, and their actual usage
max by (namespace, pod) (
  container_memory_working_set_bytes{container="", image="", pod!=""}
)
unless
max by (namespace, pod) (
  kube_pod_resource_request{resource="memory", unit="bytes"}
)

# Per node: sum of memory requests / Allocatable
sum by (node) (
  max by (node, namespace, pod) (
    kube_pod_resource_request{resource="memory", unit="bytes"}
  )
)
/
max by (node) (
  kube_node_status_allocatable{resource="memory"}
)

# Actual usage of all Pods on the node / Allocatable
max by (node) (
  container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"}
  * on(instance) group_left(node) kubelet_node_name
)
/
max by (node) (
  kube_node_status_allocatable{resource="memory"}
)

# Maximum memory usage per Pod over the past 7 days (a guide for the request)
max_over_time(
  max by (namespace, pod) (
    container_memory_working_set_bytes{container="", image="", pod!=""}
  )[7d:5m]
)

# 95th percentile over the past 7 days
quantile_over_time(0.95,
  max by (namespace, pod) (
    container_memory_working_set_bytes{container="", image="", pod!=""}
  )[7d:5m]
)

Do not base request design on kubectl top

kubectl top pod is convenient, but it is not suited to designing requests. The reason lies in the design of Metrics Server itself.

• What Metrics Server gets from the kubelet is the value of the /metrics/resource endpoint, and for memory it is the working set ("Memory is reported as the working set at the instant the metric was collected"). It is the same value as container_memory_working_set_bytes seen through Prometheus, so it includes page cache in the same way
• Metrics Server keeps only the latest value. The binary default for --metric-resolution is 60 seconds, while the v0.8.1 deployment manifest sets it to 15 seconds. Metrics Server keeps no long-term history, so it cannot show past peaks
• Metrics Server's own README states in its use cases "Don't use Metrics Server when you need: ... An accurate source of resource usage metrics" (README.md)

Use kubectl top to get a feel for "what is using a lot right now", and when you decide a request, specify a period and check with PromQL for comparing requests and actual usage.

Adjust requests automatically with VPA

Instead of running the PromQL in the previous section by hand, you can have the Vertical Pod Autoscaler (VPA) compute recommendations. The idea is the same (derive from quantiles of past usage), and VPA runs it continuously, combined with OOM detection.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: sample-vpa
  namespace: default
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample
  updatePolicy:
    updateMode: "Off"          # only compute recommendations. Pods are not changed
  resourcePolicy:
    containerPolicies:
    - containerName: app
      controlledResources: ["memory"]   # default is ["cpu", "memory"]
      controlledValues: RequestsOnly    # default is RequestsAndLimits
      minAllowed:
        memory: 128Mi
      maxAllowed:
        memory: 4Gi

kubectl describe vpa sample-vpa

...
Status:
  Recommendation:
    Container Recommendations:
      Container Name:  app
      Lower Bound:
        Memory:  262144k
      Target:
        Memory:  367001600
      Uncapped Target:
        Memory:  367001600
      Upper Bound:
        Memory:  524288k

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: sample-vpa
  namespace: default
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample
  updatePolicy:
    updateMode: "Recreate"
    minReplicas: 2             # do not evict when replicas is below this (the global default is also 2)
  resourcePolicy:
    containerPolicies:
    - containerName: app
      controlledResources: ["memory"]
      controlledValues: RequestsOnly
      minAllowed:
        memory: 128Mi
      maxAllowed:
        memory: 4Gi

    containerPolicies:
    - containerName: app
      controlledResources: ["memory"]
      oomBumpUpRatio: "1.5"                    # bump up more strongly on OOM
      oomMinBumpUp: 256Mi
      memoryAggregationIntervalSeconds: 3600   # record a peak every hour
      memoryAggregationIntervalCount: 24       # compute from 24 intervals = 1 day of history

FAQ

My working set is below the limit, but the container was OOMKilled

What memcg OOM compares against the limit is memory.current (container_memory_usage_bytes), not the working set. Furthermore, memory.current reaching the limit does not cause an OOM by itself; the kill happens when reclaim fails to bring usage back under the limit. Also, even if your container is under its limit, it can be killed in a node-wide memory exhaustion (global OOM). First use The two OOM kill paths to tell the path.

If none of the values seems to have reached the limit, suspect a spike shorter than the scrape interval. A spike in memory usage shorter than Prometheus' scrape interval (the global scrape_interval defaults to 1 minute; see Configuration) does not show up in metrics. Batch processing right after startup, or suddenly accepting a very large request, are examples.

In that case, check values like these:

Besides metrics, also check Last State in kubectl describe pod (exit code 137 means termination by SIGKILL).

Node usage computed from MemAvailable is low, but Pods are evicted

MemAvailable counts part of the page cache as "free", but the kubelet is working-set based and treats all of active_file as "in use". With workloads that do a lot of file I/O or logging, active_file can grow to several GiB, so even if the MemAvailable-based utilization looks low, the kubelet side may be approaching its threshold.

Conversely, on nodes where page cache has not built up much, node-exporter's side comes out higher. Which is higher depends on the environment, so you cannot judge from just one of them. For the breakdown of factors and the PromQL to get the kubelet-side utilization, see Why node-exporter and kubelet usage disagree.

kubectl top pod and the Prometheus working set differ

kubectl top pod shows the sum of the containers' working sets obtained from the kubelet (via Metrics Server). Prometheus' container_memory_working_set_bytes comes from the same source, but it is as stale as the scrape interval, so instantaneous values differ.

Memory working set keeps growing

First check whether container_memory_rss or container_memory_cache is the one growing.

• container_memory_rss is growing: suspect a memory leak in the application
• container_memory_cache is growing: page cache is accumulating. But it includes not only the cache of files on disk but also tmpfs and shared memory. To tell which is growing, check shmem in memory.stat on the node. tmpfs (such as emptyDir with medium: Memory) is not reclaimed on nodes without swap, so accumulation leads to OOM kills. shmem sits on the anon or unevictable LRU, so it does not appear in inactive_file and is not subtracted from the working set. That means the increase pushes the working set up as is

For containers with no memory limit, page cache accumulates with the whole node's memory as the ceiling. Set a memory limit on containers that handle large files.

The RSS my application reports differs from the RSS metric

They measure different things. The value called "RSS" in Go, Java, Node.js and so on is per-process RSS, whereas container_memory_rss is anonymous pages charged to the container's cgroup.

There are two main differences:

• Whether file mappings are included: a process's RSS includes the pages of the executable, shared libraries and mmaped files that are in physical memory (RssFile). container_memory_rss does not. So process RSS also includes RssFile and RssShmem. Shared-page charging and sampling mean the difference is not necessarily an exact subtraction
• Unit of aggregation: a process's RSS is for one process. container_memory_rss is per container cgroup, so it is the total of all processes in the container. Also, a process's RSS counts shared pages in each process, whereas in a cgroup a page is charged to only one cgroup

Note that JVM Runtime.totalMemory() and JMX heap usage are not RSS to begin with; they are the heap size. Metaspace, code cache, thread stacks and GC bookkeeping areas are not included, so you cannot compare these values directly with the memory limit.

Neither value is used in the OOM kill decision. What memcg OOM compares against the limit is memory.current (container_memory_usage_bytes). Global OOM follows a node-side allocation failure after reclaim, potentially within allocation constraints; it has no per-container usage threshold.

Which metric should I use for node memory utilization

Use them according to purpose.

Note: What emits kube_pod_resource_request is kube-scheduler, not kube-state-metrics (resources.go). kube-state-metrics also has a similarly named kube_pod_container_resource_requests, but kube-state-metrics itself says in its help text that it recommends using the more accurate kube-scheduler metrics. Unless Prometheus is configured to scrape kube-scheduler's /metrics/resources, kube_pod_resource_request is not collected.

Version and configuration differences

Statistics collection and Memory QoS behavior depend on the Kubernetes version and kubelet configuration.

cgroup v2 prerequisites

Kubernetes support for cgroup v2 became stable in v1.25. The documented requirements include Linux 5.8 or later, a compatible runtime and matching systemd cgroup drivers. Confirm that the following command prints cgroup2fs:

stat -fc %T /sys/fs/cgroup/

See About cgroup v2. The official blog, The Shift to cgroup v2 in Kubernetes, also explains active page cache and group OOM handling. Its Memory QoS discussion uses v1.36 settings; the v1.37 defaults below differ.

KEP-2371: cAdvisor-less, CRI-full Container and Pod Stats

KEP-2371 moves the source of container and node metrics from the kubelet's embedded cAdvisor to the container runtime (CRI). Its goal is to remove the situation where the kubelet collects duplicate statistics from both cAdvisor and CRI.

• The /metrics/cadvisor endpoint and the container_* metric names are kept, and the plan is to change only where the values come from to CRI
• The feature gate PodAndContainerStatsFromCRI was Alpha in v1.23 and becomes Beta in v1.37, but even at Beta it is disabled by default

So the definition of the working set and the eviction decision method described in this article will not change for the time being. If this migration becomes the default in the future, the source of values becomes the runtime side, so small differences may arise from differences in how cgroups are read.

KEP-2570: Support Memory QoS with cgroups v2

KEP-2570 reflects requests.memory into cgroup v2 memory protection settings. When enabled, the memory for the request is protected from reclaim, and throttling kicks in before the limit is reached.

The feature gate MemoryQoS has been Alpha (disabled by default) since v1.22 and became Beta and enabled by default in v1.37. However, just enabling the feature gate does not make requests.memory reflected in the cgroup. To reflect it, the kubelet needs the following settings, both disabled by default as of v1.37.

• memoryReservationPolicy: TieredReservation: sets requests.memory into memory.min for Guaranteed and memory.low for the rest. A setting added to KubeletConfiguration in v1.36, with a default of None. With None, memory.min is not set
• memoryThrottlingFactor: sets memory.high to requests.memory + factor × (limits.memory - requests.memory). Up to v1.36 the default was 0.9, but the feature gate itself was disabled by default then, so it had no actual effect. In v1.37 the default is nil, and unless you set it explicitly memory.high is not set

Both can be confirmed in the KubeletConfiguration type definition.

    // MemoryThrottlingFactor specifies the factor multiplied by the memory limit or node allocatable memory
    // ...
    // Default: nil
    MemoryThrottlingFactor *float64 `json:"memoryThrottlingFactor,omitempty"`
    // MemoryReservationPolicy controls how the kubelet applies cgroup v2 memory protection.
    // "None" (default): The kubelet does not set memory.min for containers and pods,
    // ...
    MemoryReservationPolicy MemoryReservationPolicy `json:"memoryReservationPolicy,omitempty"`

So even on a v1.37 cluster, requests.memory is not reflected in the cgroup, as in the table above. "Not reflected" is only about the default configuration, and it is reflected if you set the two above.

What is actually written with the default configuration is as in the implementation. When memoryReservationPolicy is None, an explicit 0 is written instead of requests.memory (equivalent to no protection).

        if memoryRequest != 0 && m.memoryReservationPolicy == kubeletconfiginternal.TieredReservationMemoryReservationPolicy {
            ...
        } else {
            unified[cm.Cgroup2MemoryMin] = "0"
            unified[cm.Cgroup2MemoryLow] = "0"
        }

So with the v1.37 default configuration, a container's cgroup has memory.min = 0, memory.low = 0 and memory.high unset (max), and only memory.max (= limits.memory) is in effect. If you use a managed service, check the provider's release notes to see whether these settings get enabled.

What the cgroup v2 values become if you enable it

This section works out what gets written to each cgroup v2 file when you explicitly set memoryThrottlingFactor: 0.9, the default up to v1.36, and also enable memoryReservationPolicy: TieredReservation.

# /var/lib/kubelet/config.yaml (KubeletConfiguration)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  MemoryQoS: true            # true by default in v1.37
memoryReservationPolicy: TieredReservation   # default is None
memoryThrottlingFactor: 0.9                  # default in v1.37 is nil (= memory.high is not set)

apiVersion: v1
kind: Pod
metadata:
  name: sample
spec:
  containers:
  - name: app
    image: nginx
    resources:
      requests:
        memory: 256Mi
      limits:
        memory: 1Gi

            memoryHigh = int64(math.Floor(
                float64(memoryRequest)+
                    (float64(memoryLimitVal)-float64(memoryRequest))*float64(*m.memoryThrottlingFactor))/float64(defaultPageSize)) * defaultPageSize

Hands-on verification with kind

The following examples compare protection settings, page cache and anonymous memory in a local kind cluster.

Verifying it with kind

# kind-memqos.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: memqos
nodes:
- role: control-plane
  # the kind default image changes with the version, so pin it explicitly
  image: kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
  kubeadmConfigPatches:
  - |
    kind: KubeletConfiguration
    featureGates:
      MemoryQoS: true
    memoryReservationPolicy: TieredReservation
    memoryThrottlingFactor: 0.9

kind create cluster --config kind-memqos.yaml

# Check that the settings reached the kubelet
kubectl get --raw "/api/v1/nodes/memqos-control-plane/proxy/configz" \
  | jq '.kubeletconfig | {memoryThrottlingFactor, memoryReservationPolicy, featureGates}'

{
  "memoryThrottlingFactor": 0.9,
  "memoryReservationPolicy": "TieredReservation",
  "featureGates": { "MemoryQoS": true }
}

# Inside a kind node (root of the cgroup namespace = actually a non-root cgroup)
docker exec memqos-control-plane ls /sys/fs/cgroup/memory.current

/sys/fs/cgroup/memory.current

# On the real host (the root cgroup of cgroup v2)
colima ssh -- ls /sys/fs/cgroup/memory.current

ls: /sys/fs/cgroup/memory.current: No such file or directory

colima ssh -- ls /sys/fs/cgroup/memory.stat

/sys/fs/cgroup/memory.stat

apiVersion: v1
kind: Pod
metadata:
  name: sample-burstable
spec:
  containers:
    - name: app
      image: nginx:1.27
      resources:
        requests:
          memory: 256Mi
        limits:
          memory: 1Gi

kubectl apply -f sample-burstable.yaml
kubectl wait --for=condition=Ready pod/sample-burstable --timeout=120s

# Identify the cgroup by the real container ID (a glob can pick up the pause side, as explained below)
CID=$(kubectl get pod sample-burstable \
    -o jsonpath='{.status.containerStatuses[0].containerID}' | sed 's|containerd://||')
docker exec memqos-control-plane sh -c "
    d=\$(find /sys/fs/cgroup/kubelet.slice -type d -name 'cri-containerd-${CID}.scope')
    grep . \$d/memory.max \$d/memory.min \$d/memory.low \$d/memory.high"

/sys/fs/cgroup/.../cri-containerd-50fc89e1....scope/memory.max:1073741824
/sys/fs/cgroup/.../cri-containerd-50fc89e1....scope/memory.min:0
/sys/fs/cgroup/.../cri-containerd-50fc89e1....scope/memory.low:268435456
/sys/fs/cgroup/.../cri-containerd-50fc89e1....scope/memory.high:993210368

                                    memory.min   memory.low   memory.high
Burstable with only memory matched           0    536870912           max
Guaranteed with cpu and memory matched  536870912          0           max

kubelet-kubepods.slice            memory.min 1243611136   memory.low 706740224
kubelet-kubepods-burstable.slice  memory.min 0            memory.low 706740224
kubelet-kubepods-besteffort.slice memory.min 0            memory.low 0

1243611136 - 706740224 = 536870912 = 512Mi  ← matches the Guaranteed Pod's request

kubectl get --raw "/api/v1/nodes/memqos-control-plane/proxy/metrics" \
  | grep kubelet_memory_qos

kubelet_memory_qos_node_memory_low_bytes 7.06740224e+08
kubelet_memory_qos_node_memory_min_bytes 5.36870912e+08

Comparing page cache with anonymous memory

apiVersion: v1
kind: Pod
metadata:
  name: memtest
spec:
  containers:
    - name: app
      image: busybox:1.37
      command: ["sh", "-c", "sleep 3600"]
      resources:
        requests:
          memory: 64Mi
        limits:
          memory: 256Mi
      volumeMounts:
        - name: disk
          mountPath: /disk
  volumes:
    - name: disk
      emptyDir: {}

kubectl apply -f memtest.yaml
kubectl wait --for=condition=Ready pod/memtest --timeout=120s

kubectl exec memtest -- sh -c 'dd if=/dev/zero of=/disk/big bs=1M count=1536'

1610612736 bytes (1.5GB) copied, 1.501464 seconds, 1023.0MB/s

memory.max      268435456   (256Mi)
memory.high     248299520   ← floor((64Mi + (256Mi - 64Mi) * 0.9) / 4096) * 4096
memory.current  246484992
memory.peak     249049088

--- memory.events ---
low 0
high 2666        ← number of times it was throttled for exceeding memory.high
max 0            ← memory.max was not reached
oom 0
oom_kill 0       ← no OOM kill occurred
oom_group_kill 0

--- memory.stat (excerpt) ---
file           235982848
inactive_file  235937792   ← nearly all of it is reclaimable inactive page cache
active_file        45056

kubectl run oomtest --image=busybox:1.37 --restart=Never \
    --overrides='{"spec":{"containers":[{"name":"app","image":"busybox:1.37",
      "command":["sh","-c","tail /dev/zero"],
      "resources":{"limits":{"memory":"64Mi"}}}]}}'

kubectl get pod oomtest -o jsonpath='{.status.containerStatuses[0].state.terminated}'

{"exitCode":137,"reason":"OOMKilled",...}

oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-9be8306a....scope,
mems_allowed=0,oom_memcg=/docker/.../kubelet-kubepods-burstable-pod....slice,
task_memcg=/docker/.../cri-containerd-9be8306a....scope,task=tail,pid=4867,uid=0

Memory cgroup out of memory: Killed process 4867 (tail) total-vm:68880kB,
anon-rss:64768kB, file-rss:1280kB, shmem-rss:0kB, UID:0 pgtables:176kB oom_score_adj:984

kubectl get events -A --field-selector reason=SystemOOM

No resources found

When the dedicated test cluster is no longer needed, remove it:

kind delete cluster --name memqos

Terminology and kernel references

Closing thoughts

Separate the values used to display usage from those used to enforce limits and trigger eviction. That distinction narrows an OOM investigation to the relevant container, ancestor cgroup or node-side constraint. Then review requests against workload history and placement density, and review limits against the memory the workload actually needs.

References

Components and versions

This article assumes cgroup v2. With cgroup v1, the mapping between metrics and memory.stat fields is different.

The statements in this article target the following versions.

Note: I assume containerd as the container runtime. With other runtimes such as CRI-O the meaning of cgroup v2 values is the same, but the OOMKilled detection logic and cgroup path naming differ.

I also assume the cluster has Prometheus, node-exporter and kube-state-metrics installed and that you can run PromQL. These are not part of Kubernetes itself, so if you do not have them, substitute kubectl top or read memory.stat directly on the node.

Links

Listed in the order they appear in the article.

• Memory Resource Controller (cgroup v2) (definitions of memory.current / memory.max / memory.stat and other files)
• About cgroup v2 (cgroup v2 from Kubernetes' point of view)
• Resource Management for Pods and Containers (basics of requests and limits)
• Reserve Compute Resources for System Daemons (calculation of Allocatable)
• Node-pressure Eviction (overview of eviction: signals, thresholds, selection of target Pods)
• Concepts overview - OOM killer (conditions and role of the OOM killer invocation)
• Node out of memory behavior (behavior when eviction is not in time)
• Pod Quality of Service Classes (how QoS classes are determined)
• proc.rst - oom_score / oom_score_adj (the score used to select the process to kill)
• sysctl/vm.rst (OOM-related sysctls such as panic_on_oom)
• Understand Pressure Stall Information (PSI) Metrics (detecting waits for memory allocation)
• Metrics Server FAQ (definition and limits of the values kubectl top shows)
• Vertical Pod Autoscaler (recommending requests and limits from usage)

Metrics covered in this article

Grouped by emitting component. The takeaway of this article is that even metrics representing the same "memory usage" have different definitions if they come from different emitters, so when investigating, first check which component emits the value.

cAdvisor (kubelet /metrics/cadvisor)

cAdvisor embedded in the kubelet reads cgroups and exposes them. You do not need to run a separate cAdvisor.

The whole node can be selected with id="/", all Pods on the node with id=~"/kubepods|/kubepods.slice", and a single Pod with container="", image="", pod!="". Only id="/" has no memory.current, so its usage is substituted with anon + file from memory.stat (kernel memory is not included). For details see Values the kubelet uses for eviction decisions.

kubelet (/metrics/resource)

An endpoint that exposes only CPU, memory and swap usage per container, per Pod and per node. It is a different path from /metrics/cadvisor, so to scrape it with Prometheus you need to add configuration.

Metrics Server gets values from here, and the official documentation also says "The metrics-server fetches resource metrics from the kubelets" (Resource metrics pipeline). It is not exclusive to Metrics Server, though, and Metrics Server's own README directs you to scrape this endpoint directly for monitoring purposes ("In such cases please collect metrics from Kubelet /metrics/resource endpoint directly.").

kubelet (/metrics)

node-exporter

Turns the contents of /proc directly into metrics. It knows nothing of Kubernetes concepts.

kube-state-metrics

Converts Kubernetes API objects (spec / status) into metrics. It does not handle actual resource usage.

kube-scheduler (/metrics/resources)

Exposed on the secure port (default 10259) at /metrics/resources. It is a different endpoint from /metrics.