Federating clusters for zero-downtime Kubernetes
Linkerd's multicluster extension now allows developers to mix federation, flat mirroring, and gateway mirroring modes on the same network links.
Deep dive
- Federation: A mode where same-name services are unioned across clusters, providing automatic load balancing and failover.
- Flat Mirroring: A mode where traffic is routed directly to pod IPs, requiring a flat network between clusters.
- Gateway Mirroring: A mode where traffic is routed through a cluster gateway, useful when direct pod-to-pod connectivity isn't available.
- Full-Mesh Topology: A network setup where every node or cluster is connected to every other cluster.
Decoder
- VPC Peering: Connecting separate Virtual Private Clouds to allow internal traffic routing.
- mTLS (mutual TLS): Security protocol where both client and server verify each other's identity using certificates.
- TrafficSplit: A Kubernetes resource that allows splitting traffic between different service versions or clusters.
Original article
Every multi-region setup eventually meets the same awkward moment: a whole cluster goes away, and the identical copy of your service running two regions over might as well not exist, because nothing is wired to treat them as one thing. Failover becomes a runbook: restore, repoint DNS, and wait for an outage that, on paper, you’d already paid to survive.
Linkerd’s multicluster extension closes that gap by letting several clusters present a service as a single, load-balanced endpoint. The part that the official tasks gloss over is that a real platform almost never picks one multicluster mode. Some services want federation (same service everywhere, one endpoint, automatic failover). While others want mirroring (reach a specific remote service by name). And you frequently want both patterns living on the same set of links. The docs walk through each mode on its own. This post wires all three together across three GKE clusters, with a full-mesh link topology, a chaos test that takes out an entire cluster, and scripts you can clone and run on a fresh GCP project.
Companion repo: Every script referenced here lives in this repository. Feel free to clone it, set your project ID, and run it.
Linkerd multicluster modes: Gateway, flat, and federated
Linkerd’s multicluster extension supports three modes. The nice thing is they’re not mutually exclusive: on the same set of linked clusters, the mode is chosen per service via a label.
| Mode | Label | What happens | Network Requirement |
|---|---|---|---|
| Hierarchical (gateway) | mirror.linkerd.io/exported=true |
Service mirrored as <svc>-<cluster>, traffic routed through a gateway |
Gateway IP reachable |
| Flat (pod-to-pod) | mirror.linkerd.io/exported=remote-discovery |
Service mirrored as <svc>-<cluster>, traffic goes directly to remote pods |
Flat network (pod IPs routable) |
| Federated | mirror.linkerd.io/federated=member |
All same-name services unioned into <svc>-federated, load balanced across all clusters |
Flat network (pod IPs routable) |
The distinction that matters operationally is that hierarchical mirroring works on any network. Only the gateway IP needs to be reachable, while flat and federated modes need real pod-to-pod connectivity. On GCP, VPC-native GKE clusters on peered VPCs give you that flat network for free. So, you can run federated services for your core workloads over a flat network and still mirror a specialized service through a gateway from a cluster that isn’t on that network. Most platform teams I’ve seen end up with exactly this kind of mix.
Multi-region architecture: GKE cluster setup
We have three GKE clusters across three regions, fully linked to each other (six directional links total). Three demo services, each using a different multicluster mode:
frontend is federated and runs in all three clusters. A single federated frontend service in each cluster load-balances across all nine pods (3 replicas × 3 clusters). When a cluster goes down, the remaining six pods absorb the traffic with no application changes.
api is flat-mirrored and runs in west and east. The north cluster consumes it as api-west and api-east, which are explicit remote service names with traffic sent straight to the remote pods. This is what you reach for when the client needs to decide which backend it talks to, for example, to keep a request in-region for data locality.
analytics is gateway-mirrored and runs only in east. Exported through the Linkerd gateway so west and north reach it as analytics-east-gw without needing flat-network connectivity to east’s pods. It’s here mainly to prove that gateway mode coexists with flat and federated modes on the same links.
Deployment prerequisites: GKE, Linkerd, and CLI tools
- A GCP account (free-tier credits cover this. Use three standard clusters with small node pools)
- `gcloud` CLI, authenticated (`gcloud auth login`)
- `kubectl` v1.28+
- `step` CLI, `brew install step` (for certificate generation)
- `helm` v3
- ~30 minutes for the full setup
Step 0: Configure
Clone the repo, create a local .env file from the example file, and customize it for your GCP project.
git clone <your-repo-url>
cd blog-linkerd-federation
cp env.example .env
Load the variables into your current shell so the scripts can read them:
source .env
Step 1: Provision three GKE clusters with VPC peering
Run the infrastructure script to create the networks and clusters.
./scripts/01-infra.sh
This script sets up three VPCs with full-mesh peering, creates three GKE Standard clusters, and renames the kubectl contexts.
Step 2: Install Linkerd with a shared trust anchor
Install Linkerd into all three clusters using a shared trust anchor.
./scripts/02-linkerd-install.sh
Step 3: Install multicluster and create a full-mesh link topology
Set up the multicluster components and create a full-mesh topology between the clusters.
./scripts/03-multicluster-setup.sh
Step 4: Deploy the demo services
Deploy the demo workloads and apply the necessary labels for federation, flat mirroring, and gateway mirroring.
./scripts/04-deploy-app.sh
Step 5: Verify all three modes
Generate traffic against all three service patterns and verify that each resolves the way you expect.
./scripts/05-verify.sh
Step 6: The chaos test, kill a cluster
Simulate a full cluster failure and watch how each service type reacts.
./scripts/06-chaos-test.sh
Step 7: Teardown
When you’re finished with the demo, run the teardown script to remove all the infrastructure.
./scripts/99-teardown.sh
Selecting your Linkerd multicluster architecture strategy
| Question | → Mode |
|---|---|
| Should the client be cluster-agnostic? | Federated |
| Does the client need to pick a specific cluster? | Flat mirror |
| Is there no flat network between clusters? | Gateway mirror |
| Do you need automatic failover with no app changes? | Federated |
Key takeaways: Mastering multi-region Linkerd deployments
The docs show each multicluster mode in isolation; real platforms need all three at once. Federation covers the common case: The same service everywhere, automatic failover, and nothing to change in the app. Flat mirrors give you explicit, cluster-aware routing when data locality matters. Gateway mirrors get you cross-cluster reach when a flat network isn’t on the table.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi