Introducing the GKE standby buffer: Improve node startup times without blowing your budget

Application owners and platform engineers have long faced a difficult choice: spend excessively by over-provisioning to guarantee quick startups, or minimize costs but endure slow cold starts.

We are excited to announce a solution to this compromise: Google Kubernetes Engine standby buffers. This builds on the launch of GKE active buffers earlier this year, a native version of the Kubernetes CapacityBuffers API that makes it easy to provision readily available capacity to handle traffic spikes, delivering near-zero startup latency for new pods. However, active buffers still impose a trade-off between performance and cost. New GKE standby buffers help by maintaining a low-cost, suspended capacity buffer for your GKE clusters. With a cost overhead in the low single-digit percent, GKE standby buffers help you achieve near-immediate scheduling for your workloads with negligible cost overhead. This is useful for all kinds of workloads — general-purpose, agentic, and everything in between.

Under identical traffic loads, the cluster without standby buffers suffered severe latency spikes, with P50, P95, and P99 metrics trapped between 4 and 6 minutes. Conversely, the cluster with standby buffers maintained a P50 latency of just single-digit seconds, while its P95 and P99 metrics briefly peaked at one minute before quickly normalizing to single-digit seconds. Both setups exhibited a similar allocatable core cost, making the buffered approach far more efficient.

The problem: High costs and latency

Traditionally, autoscaling with standard Kubernetes has been effective but slow. Traffic surges or batch jobs require cluster autoscalers to provision fresh nodes, leaving Pods in a pending state. To circumvent delays, you have to resort to clunky workarounds like lowering your Horizontal Pod Autoscaler (HPA) thresholds or managing so-called balloon pods. These workarounds are expensive:

• Managing balloon pods is operationally complex, requiring manual configuration and ongoing maintenance of priority classes and resource requests to ensure they function correctly.
• Lowering the HPA threshold adds empty (wasted) space that linearly scales with the size of the node pool.

Both GKE active and standby buffers allow capacity to be defined declaratively, removing the need for clunky and operationally heavy workarounds.

In addition, GKE standby buffers lower infrastructure costs by storing the node’s state to disk, releasing compute and memory costs and keeping only persistent disk and IP address costs. Then, combined with an active buffer, you can achieve near-instant pod scheduling that has similar performance to over-provisioning, but at a very affordable price.

Active and standby buffers working together

All GKE capacity buffers operate on a principle similar to video streaming on platforms like YouTube. By proactively attempting to provision and manage available capacity ahead of impending demand (much like pre-downloading video content) GKE helps to ensure that resources are readily available when they’re needed.

With today’s launch, the two types of capacity buffers can work in harmony:

• Active buffer: Cluster Autoscaler works to reserve enough capacity for a predefined amount of pods on existing cluster nodes, and, if needed, provisions extra nodes. Select this ready-to-use buffer to provide capacity to your most latency-sensitive workloads.
• Standby buffers: Nodes are pre-provisioned and fully initialized with necessary components like Kubernetes DaemonSets, and given time to preload images, but are then suspended, while the underlying compute capacity is released to save costs. When demand spikes, these nodes resume 2-3x faster than creating a fresh node, bridging the gap between cold starts and always-on capacity.

The active buffer covers the initial spike until standby buffers resume. The system prioritizes refilling the active buffer from the standby buffer. The standby buffer handles an extended load and protects against slower node cold starts. As standby buffers refill, they initially kick into an active state for a configurable amount of time before they are suspended, providing a boost of active capacity during sustained traffic loads.

Early benchmarks

In our tests, using standby buffers enabled us to deliver sub-second Agent Sandbox scheduling latency for up to 90% lower cost compared to complete overprovisioning.

Optimized for business needs

Businesses are under constant pressure to optimize resource consumption while streamlining operations. Recognizing that organizations need smarter tools to manage sporadic and spikey workloads, we worked hard to deliver standby buffers quickly. Now, whether you’re running agents, batch jobs, CI/CD pipelines, game servers, or spiky workloads, GKE capacity buffers allow you to dynamically balance performance and cost. You can finally define your "insurance policy" against traffic spikes without paying a high premium for it. With GKE standby buffers you can:

1. Circumvent cold starts: Nodes suspended by standby buffers resume 2-3x faster than provisioning fresh nodes, reducing pod scheduling latency during traffic spikes and sustained traffic load.
2. Enjoy lower costs: A standby buffer incurs a fraction of the cost of active capacity because the underlying VM is suspended. You pay for storage and an IP address, rather than for full compute-hours.
3. Gain declarative control: Replace complex balloon pod workarounds with the simple, native declarative CapacityBuffers API, explicitly stating how much headroom you need, and letting GKE handle the rest.

“Using GKE standby capacity buffers has lowered our time-to-ready from several minutes to 30 seconds at a very affordable price.”
- Pedro Spagiari, Chief Architect at Unico

Get started

Ready to improve your performance and save on costs?

• Start by defining a CapacityBuffer resource in your cluster to specify your target buffer size.
• Try balancing between standby buffers to reduce pod scheduling latency for sustained loads, and active buffers to address immediate unpredictable capacity needs.

Let’s look at an example of how to configure buffers for a Deployment while also using custom ComputeClasses.

Basic setup

Beginning with some basic setup, create a namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace

Then, create a custom ComputeClass (optional):

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: my-ccc
  namespace: my-namespace
spec:
  # Buffers will also be created according to these priorities
  priorities:
  - machineFamily: n4
  - machineFamily: n4d
  - machineFamily: c4
  - machineFamily: c4d
  nodePoolAutoCreation:
    enabled: true

Define the buffer unit size

You can use a PodTemplate as a reference for the buffer unit size. You can also create a buffer for a specific deployment or any object that defines scale subResource.

# Defines the resource requirements for one unit of buffer.
apiVersion: v1
kind: PodTemplate
metadata:
  name: my-buffer-unit-template
  namespace: my-namespace
template:
  spec:
    terminationGracePeriodSeconds: 0
    tolerations:
    # Optional: Ensures buffer pods can land on any node.
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"
    containers:
    - name: buffer-container
      image: registry.k8s.io/pause:3.9
      resources:
        requests:
          cpu: "1"
          memory: "1Gi"
        limits:
          cpu: "1"
          memory: "1Gi"
      # Optional: Using buffers with a custom ComputeClass /
      # controls the properties of the nodes GKE provisions.
      nodeSelector:
        cloud.google.com/compute-class: my-ccc

Create buffers

Lastly, create a CapacityBuffer object by referring to our PodTemplate. Here, you create a standby buffer of 50 CPUs and 50 GB of RAM:

apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityBuffer
metadata:
  name: my-standby-buffer-resource-limits
  namespace: my-namespace
  annotations:
    # Optional: Time after which buffer nodes are suspended.
    # Default is 5 minutes.
    buffer.gke.io/standby-capacity-init-time: "5m"
    # Optional: Time after which standby buffers are recreated.
    # Default is 1 day, "never" avoids refreshing.
    buffer.gke.io/standby-capacity-refresh-frequency: "1d"
spec:
  podTemplateRef:
    name: my-buffer-unit-template
  # The desired state is 20 standby buffer units.
  # When a standby buffer gets used, a new one gets created.
  limits:
    cpu: "50"
    memory: "50Gi"
  provisioningStrategy: "buffer.gke.io/standby-capacity"

And an active buffer of seven 5 CPUs and 5 GB of RAM (optional):

apiVersion: autoscaling.x-k8s.io/v1beta1
kind: CapacityBuffer
metadata:
  name: my-active-buffer-resource-limits
  namespace: my-namespace
spec:
  podTemplateRef:
    name: my-buffer-unit-template
  # The desired state is 2 active buffer units.
  # When an active buffer gets used, a new one gets created.
  limits:
    cpu: "5"
    memory: "5Gi"
  provisioningStrategy: "buffer.x-k8s.io/active-capacity"

Finally, apply the above objects to your cluster. That’s it!

Now, any existing and future deployments that can schedule on the space reserved by the buffers will benefit from faster pod scheduling latencies.

Test the buffers

You can check on the status of your buffers. In Kubernetes, suspended nodes can be identified by condition Suspended.

kubectl get nodes -o custom-columns='NAME:.metadata.name,SUSPENDED:.status.conditions[?(@.type=="Suspended")].status'

Expect the following kind of output, and wait for the standby buffers to get suspended.

NAME                                            SUSPENDED
gke-my-cluster-nap-n4-standard-8-k960-...-ffbx  False # Node has been resumed.
gke-my-cluster-nap-n4-standard-4-k960-...-h2x4  <none> # Node was never suspended.
gke-my-cluster-nap-n4d-standard-8-1cip-...-74jf  True # Node is suspended.

To test the buffers, create a deployment and scale it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  namespace: my-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-deployment
  template:
    metadata:
      labels:
        app: my-deployment
    spec:
      containers:
      - name: busybox
        image: busybox
        command: ["sleep", "inf"]
        resources:
          requests:
            cpu: "500m"
            memory: "500Mi"
      # Optional: Using buffers with a custom ComputeClass /
      # controls the properties of the nodes GKE provisions.
      nodeSelector:
        cloud.google.com/compute-class: my-ccc

Scaling this deployment to two replicas allows them to be assigned to the active buffer for immediate scheduling. The active buffer is then immediately refilled from the standby buffer. Simultaneously, the standby buffer initiates the provisioning of new nodes.

If you further scale the deployment to 50 replicas, scheduling all of them on the standby buffer occurs once the nodes resume. New nodes provisioned to refill the standby buffer briefly function as active buffers providing a temporary active standby boost. Therefore, when further scaling the deployment to 100 replicas during this time, you may notice that new replicas benefit from immediate scheduling.

GKE standby buffer best practices

1. Define standby buffers that are sufficient to cover the extended load you expect to encounter, so that buffers can refill in the background from a cold start. A sufficiently sized standby buffer can drop your max pod scheduling latency to the time it takes to resume a node — around 30 seconds.
2. When the buffer starts to get used and is refilled, new buffer nodes initially swing into an active state prior to suspending. This helps to boost active capacity during a prolonged load.
3. If your application requires the lowest possible pod scheduling latency, define an active buffer size that is sufficient to cover any initial spikes you expect to encounter until standby buffer nodes are able to resume. The system prioritizes refilling the active buffer by consuming the standby buffer. A sufficiently sized active buffer and a sufficiently sized standby buffer can help you achieve one-second pod scheduling latency for a fraction of the cost of overprovisioning.
4. Experiment with different buffer sizes to get the best result for your workload.

To help, we created a simulator to help with sizing the buffers to achieve your performance targets, available at https://github.com/gke-labs/buffers-simulator.

Try it yourself!

Active and standby buffers in GKE provide a native solution for low-latency and cost-effective workload scaling by maintaining warm and standby capacity buffers. By circumventing slow node cold starts, buffers help performance-critical applications handle sudden traffic spikes. This feature replaces complex manual workarounds like balloon pods with a simple, declarative API, and allows for fixed, percentage-based, or resource-limited buffering strategies to help maintain strict service-level objectives cost-effectively and without over-provisioning for peak.

Standby buffers are available for GKE clusters running version 1.36.0-gke.2253000 or later. To get started with buffers, check out the documentation.