DEVOURED
ingress-nginx to Envoy Gateway migration on CNCF internal services cluster
CNCF documents their migration from the deprecated ingress-nginx to Envoy Gateway using Gateway API, sharing real-world solutions to certificate management, load balancing, and TLS configuration challenges.
What: A detailed migration guide documenting how CNCF moved their internal Kubernetes services cluster from ingress-nginx (which is retiring) to Envoy Gateway with Gateway API, including configuration decisions, troubleshooting steps, and cert-manager integration for ongoing certificate management.
Why it matters: With ingress-nginx being deprecated, many Kubernetes operators will need to migrate, and this real-world case study reveals non-obvious gotchas like certificate ownership cascading deletes, externalTrafficPolicy causing health check failures, and cross-namespace access patterns that could save teams hours of debugging.
Takeaway: Review the migration patterns for certificate ownership removal, ReferenceGrant configuration for cross-namespace access, and externalTrafficPolicy settings if planning a Gateway API migration from ingress-nginx.
Deep dive
- Gateway API uses a multi-layer architecture (GatewayClass, Gateway, HTTPRoute) compared to ingress-nginx's single LoadBalancer controller, providing better separation of concerns but requiring more resources to understand
- CNCF chose a shared Gateway approach rather than per-route Gateways to minimize costs (one load balancer vs many), simplify operations (single TLS config point), and maintain a single stable IP address
- Critical issue: externalTrafficPolicy defaults to Local in Envoy, which makes NodePorts only listen on nodes with pods, causing Oracle Cloud Load Balancer health checks to fail on empty nodes and mark all backends unhealthy
- Certificate migration required removing ownerReferences from Certificate objects to prevent cascade deletion when Ingress objects are removed, accomplished via a kubectl/jq one-liner
- Cross-namespace certificate access requires ReferenceGrant resources to explicitly allow the Gateway in one namespace to reference Secret certificates in other namespaces
- Backend HTTPS connections (previously handled by nginx annotations like backend-protocol and proxy-ssl-verify) now require BackendTLSPolicy resources with validation configuration
- Migration strategy avoided round-robin DNS (safer but complex) in favor of reserving the existing IP address in EnvoyProxy configuration and moving all traffic at once
- Day 2 operations require enabling Gateway API support in cert-manager, updating ClusterIssuer to use gatewayHTTPRoute solver instead of ingress, and annotating the Gateway for automatic certificate management
- Separate HTTP and HTTPS listeners are required per hostname for cert-manager HTTP01 challenges to work properly (wildcard listener approach doesn't work without DNS solver)
- After moving certificates to the Gateway namespace, ReferenceGrants become redundant and can be removed, simplifying the access control model
Decoder
- ingress-nginx: A Kubernetes Ingress controller (now being retired) that uses a single load balancer to route traffic based on Ingress resource configuration
- Gateway API: A Kubernetes networking API that replaces Ingress with a more flexible multi-layer model (GatewayClass, Gateway, and route resources)
- Envoy Gateway: A CNCF project that implements Gateway API using Envoy Proxy as the underlying data plane
- HTTPRoute: Gateway API resource that defines HTTP traffic routing rules, similar to Ingress but more expressive
- ReferenceGrant: Gateway API resource that explicitly allows cross-namespace references, enabling a Gateway to use certificates stored in different namespaces
- BackendTLSPolicy: Gateway API resource that configures TLS settings for upstream/backend connections from the gateway to services
- externalTrafficPolicy: Kubernetes service setting controlling whether traffic can route to any node (Cluster) or only nodes with pods (Local)
- GatewayClass: Gateway API resource defining the controller implementation to use (like Envoy Gateway)
- cert-manager: Kubernetes controller that automates TLS certificate management and renewal from providers like Let's Encrypt
- LoadBalancer service: Kubernetes service type that provisions a cloud load balancer with a public IP address
Original article
CNCF migrated its Kubernetes services from ingress-nginx to Gateway API using Envoy Gateway, improving flexibility and architecture while addressing challenges like certificate management, load balancing, and TLS configuration. The shift reflects a move toward scalable, multi-layer ingress alternatives after ingress-nginx retirement.
Explore all 55 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi