Kubernetes v1.36: Tiered Memory Protection with Memory QoS (3 minute read)
Kubernetes v1.36 refines its Memory QoS feature to provide tiered memory protection that separates hard guarantees for critical pods from soft protection for burstable workloads.
What: An update to Kubernetes v1.36's alpha Memory QoS feature that introduces opt-in memory reservation with three-tier protection: Guaranteed Pods get hard memory protection (memory.min), Burstable Pods get soft protection (memory.low), and BestEffort Pods get no protection, preventing over-reservation of node memory.
Why it matters: Previous versions locked all requested memory as hard reservations, which could exhaust available memory on nodes with many Burstable pods; the new tiered approach allows the kernel to reclaim memory from lower-priority workloads under pressure while still protecting critical Guaranteed pods.
Takeaway: If you're running Kubernetes v1.36+, you can enable tiered memory protection by setting memoryReservationPolicy: TieredReservation in your kubelet configuration, and use the new kubelet_memory_qos_node_memory_min_bytes and kubelet_memory_qos_node_memory_low_bytes metrics to monitor memory reservation levels.
Deep dive
- Kubernetes v1.36 updates the alpha Memory QoS feature to separate memory throttling from memory reservation, giving operators more granular control
- The new memoryReservationPolicy field allows choosing between None (default, throttling only) or TieredReservation (adds memory protection)
- With TieredReservation, Guaranteed Pods receive hard protection via cgroup v2's memory.min, which the kernel will never reclaim even under memory pressure
- Burstable Pods get soft protection via memory.low, which the kernel tries to preserve but can reclaim under extreme pressure to avoid system-wide OOM
- BestEffort Pods receive neither protection, making their memory fully reclaimable
- This fixes a major issue from v1.27 where enabling MemoryQoS would lock all requested memory as memory.min, potentially exhausting node capacity
- Two new alpha metrics track total hard and soft reservations: kubelet_memory_qos_node_memory_min_bytes and kubelet_memory_qos_node_memory_low_bytes
- The kubelet now checks kernel versions at startup and warns if running on kernels older than 5.9, which have a memory.high livelock bug
- The feature requires cgroup v2, Kubernetes v1.36+, and a compatible container runtime (containerd 1.6+ or CRI-O 1.22+)
- Operators can now enable throttling first to observe behavior, then opt into memory reservation when confident the node has sufficient headroom
Decoder
- Memory QoS: Quality of Service for memory, using Linux cgroup v2 controls to guide kernel memory reclamation decisions
- cgroup v2: Second version of Linux control groups, providing hierarchical resource management for processes
- memory.min: cgroup v2 hard memory guarantee that the kernel will never reclaim, triggering OOM killer on other processes if needed
- memory.low: cgroup v2 soft memory protection that the kernel avoids reclaiming under normal pressure but can reclaim under extreme pressure
- memory.high: cgroup v2 throttling threshold; when exceeded, the kernel slows down the process to reduce memory consumption
- Guaranteed Pods: Pods where all containers have equal memory requests and limits
- Burstable Pods: Pods with at least one container having a memory request lower than its limit, or no limit specified
- BestEffort Pods: Pods with no memory or CPU requests or limits specified
- OOM killer: Linux out-of-memory killer that terminates processes when the system runs out of memory
Original article
Kubernetes v1.36 introduced significant updates to its alpha Memory QoS feature, adding opt-in memory reservation with tiered protection that separates Guaranteed Pods (hard protection via memory.min), Burstable Pods (soft protection via memory.low), and BestEffort Pods (no protection).