When the Kubernetes API went silent: A three-layer networking mystery
Google engineers traced a GKE API timeout issue to a rare 'perfect storm' of packet reordering, Linux kernel bugs, and Envoy half-close behavior.
Deep dive
- The Networking Path: Pods connect to the Kubernetes API through a virtual endpoint managed by Private Service Connect, which NATs traffic and injects proxy protocol headers.
- The Bug Cascade: Packet reordering at the virtual NIC caused clients' FIN packets to arrive out of order.
- Socket Persistence: Envoy's graceful 'half-close' (SHUT_WR) prevented the kernel from considering the sockets 'orphaned,' meaning tcp_fin_timeout never triggered.
- The Kernel Fix: Linux kernel commit 795a7dfbc3d9 was authored by Eric Dumazet to properly handle 'old but acceptable' ACKs in the closing sequence, fixing the silent FIN discard issue.
- Lesson: When troubleshooting complex network timeouts, simultaneous packet captures at multiple points (NIC, container, etc.) are essential to detect reordering.
Decoder
- FIN_WAIT_2: A TCP state where the local side has sent a FIN and received an ACK, but is waiting for the remote side to send its own FIN.
- 5-tuple: A set of five values (source IP, source port, destination IP, destination port, and protocol) that uniquely identifies a TCP connection.
- RFC 5961: A set of security standards for TCP that define how stacks should respond to unexpected packets (like challenge ACKs) to prevent data injection attacks.
Original article
When the Kubernetes API went silent: A three-layer networking mystery
Every support engineer has a case that stays with them. The one that comes up at team lunches. The one where, even after it's resolved, you find yourself staring at a packet capture at midnight just to make sure you really understand what happened. This is that case.
It started in January 2026 with a support ticket that read deceptively simply: pods are intermittently failing to reach the Kubernetes API server. The error was a familiar-looking timeout. The customer had already ruled out their own operators. What none of us knew yet was that we were looking at three separate bugs — in three different layers of the stack — all conspiring to produce the same symptom at the same time.
By the time it was over, around 800 developers had been blocked from deploying to production for weeks on end, and GKE clusters across europe-west had ground to a halt. In a race to fix the problem, we involved engineers across Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, kernel, and the Linux upstream community, who together tracked down not one but three distinct root causes. And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself.
This is the story of how we got there.
Part one: The alarm bell that wouldn't stop
The customer ran a large multi-cluster GKE environment in Google Cloud's europe-west region. Their clusters used Private Service Connect — Google Cloud's mechanism for exposing the GKE control plane privately — and had a rich ecosystem of system-level workloads: Twistlock security scanners, External Secrets Operator for secrets management, Cilium for network policy, as well as several internal Kubernetes operators.
The first errors appeared in one of its user acceptance testing (UAT) clusters in europe-west, and a few days later in a second cluster in the same region. They looked like this:
TimeoutError: Connect call failed ('1.2.3.4', 443) dial tcp 1.2.3.4:443: i/o timeout
Initially the errors were intermittent — spikes in the logs, not a hard outage. The customer suspected their own code. Their team had recently deployed a custom operator built with the kopf framework, and it was the first thing they looked at. But the logs showed the errors had started eight days before the operator was ever deployed. When they shut it down completely, the timeouts kept coming.
Part two: Tracing the ghost in the wire
To understand what the packet captures revealed, you first need to understand the architecture that a GKE pod traverses when it makes an API call. In a Private Service Connect-based GKE cluster, the path is deceptively long for something that looks like a local connection.
The key component here is Private Service Connect. When a pod dials 1.2.3.4:443, it is not actually going directly to the control plane; that IP is a virtual endpoint. The connection is intercepted by Google's networking fabric, NAT'd through a Private Service Connect endpoint, passed through an internal passthrough Network Load Balancer, and ultimately delivered to the KCP VM, where Envoy acts as a local proxy in front of the actual kube-apiserver process.
Private Service Connect adds a proxy protocol header to each new connection, so the kube-apiserver can see the real client IP. This will turn out to be important later.
What the packets revealed
When we captured simultaneously at the KCP VM's NIC and from inside the kube-apiserver container, we saw something striking. SYN packets from the pods were arriving at the KCP VM's NIC. But instead of receiving a SYN-ACK in response — the expected second step of a TCP handshake — the client was getting back an out-of-place ACK. Not an RST. Not silence. An ACK that logically belonged to a different, earlier connection.
The "Invalid ACK" spike
While one thread of the investigation was analyzing packet captures, another team was digging into internal metrics for the Private Service Connect infrastructure — specifically the user space packet switch (USPS), Google's software switching layer that handles NAT and proxy protocol header injection for Private Service Connect connections. They found that a counter called "Proxy protocol drop: invalid ACK packet" had spiked sharply upward in February 2026, coinciding with a USPS version rollout.
Challenge ACKs: The RFC 5961 mechanism
RFC 5961 defines a security mechanism called the challenge ACK. When a server receives a SYN that matches an existing or recently-closed connection's remembered state, it sends a challenge ACK — an acknowledgment carrying the last known sequence number — to verify if the new client is legitimate. If the client doesn't recognize the sequence, it sends a RST to clear the state, allowing the server to accept the next SYN normally.
The culprit: A well-intentioned fix
The change in USPS had been designed to drop packets classified as "invalid ACKs" during the connection handshake phase. Unfortunately, challenge ACKs arrive from the server before the new handshake has completed; the new USPS logic identified these as invalid and dropped them silently. Because Private Service Connect uses NAT to share a single public endpoint, the high rate of port collisions made these challenge ACKs a frequent necessity for clearing stale states, which were now being blocked.
Part three: Three bugs, one crime scene
The mitigation (rolling back the USPS change) bought us time, but it left a question: Why did the KCP VM keep connections in FIN_WAIT_2 for so long? The answer involved three distinct failure modes.
Bug #1: Packet reordering at the PIGv
Due to how Private Service Connect's miss-thread processing works, packets occasionally appeared at the KCP VM's virtual NIC (PIGv) in a different order than they arrived at the host NIC. This resulted in a client's FIN arriving after the server had already sent its own FIN, causing the server's kernel to ignore the client's packet.
Bug #2: Envoy's half-close and the frozen timer
Envoy, the proxy in front of the kube-apiserver, was performing a half-close (SHUT_WR). Because the socket was not yet "orphaned," the Linux tcp_fin_timeout parameter never fired. The socket sat in FIN_WAIT_2 indefinitely, waiting for a client FIN it would never be able to process correctly due to the reordering.
Bug #3: The Linux kernel's silent failure
Finally, we discovered a bug in the Linux kernel 6.1 implementation of tcp_ack(). When an ACK was received that was "old but acceptable" under RFC 5961, the function returned zero, causing the caller to silently discard the incoming FIN instead of moving the connection to TIME_WAIT. This prevented the kernel from cleaning up the connection naturally.
The resolution: A three-pronged fix
- Immediate mitigation: Rollback of the USPS change in February 2026 restored the pass-through of challenge ACKs.
- Correctness fix: A replacement change to handle the challenge ACK scenario properly in the Private Service Connect proxy protocol layer.
- Root cause fix: An upstream Linux kernel patch (authored by Eric Dumazet, committed as 795a7dfbc3d9) was backported to the 6.1 and 6.6 LTS branches to correctly handle old-but-acceptable ACKs during connection close.
Most networking cases resolve at one layer. This one was different because the symptom was the same whether you were looking at the Private Service Connect layer, the kernel's TCP state machine, or Envoy's connection management. What finally cracked it was the discipline of simultaneous packet capture at multiple points in the path and the collaboration of engineers across many different specialties.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi