Nginx vs Caddy vs Traefik in 2026 — I Ran All Three Under Real Production Load

Same Kubernetes cluster, same microservices, same TLS requirements. One proxy broke during ingress-nginx migration week. Guess which.

I’ve terminated more TLS than I care to admit. Reverse proxies are the part of the stack everyone forgets until certificates expire, upstreams go silent, or — in March 2026 — your ingress controller hits end-of-life and compliance asks what your migration plan is.

A logistics client ran ingress-nginx on EKS for four years. Fine until it wasn’t supported. They asked me to compare Nginx (F5 NIC), Caddy, and Traefik on the same cluster before picking a path forward. Same twelve microservices. Same cert issuer. Same load tests. I ran each as the edge for six weeks.

RKE2 v1.36 making Traefik the default ingress is not abstract news if you operate Rancher clusters. It is a migration ticket with a deadline.

The Test Bed: Production Shape, Controlled Variables

Cluster: EKS, 12 services, ~35,000 requests/minute average, ~61,000 peak during dispatch windows. Mix of REST JSON, WebSocket tracking, and one large CSV export (15–40 MB responses).

Each proxy setup:

• Automatic TLS (Let’s Encrypt / ACME)
• HTTP/2 and HTTP/3 where supported
• Rate limiting on public endpoints
• Prometheus metrics scraped the same way
• Same pod resource requests: 2 CPU / 2Gi per ingress controller replica

I tracked: p50/p99 proxy latency, memory at idle and at 10k connections, config reload behavior, and operator time per new service route.

Throughput vs Operator Time — Pick Your Poison

Nginx (F5 NIC ingress on Kubernetes):

• Proxy throughput: ~46,000 req/s in our wrk suite (2 KB JSON upstream) — highest of the three
• p99 proxy overhead: ~0.9ms at median load
• Idle memory: ~35 MB per controller pod; ~120 MB at 10k connections
• New service route: manual Ingress YAML + annotation review — ~25 minutes including PR
• Strength: raw performance, massive community knowledge, every senior SRE has scars here

Caddy 2.11 (standalone ingress on VMs + DaemonSet edge):

• Proxy throughput: ~41,000 req/s — ~11% behind Nginx on pure JSON proxying
• p99 overhead: ~1.2ms — TLS termination gap narrowed; automatic HTTPS is not a gimmick
• Idle memory: ~28 MB; ~95 MB at 10k connections
• New route: 12-line Caddyfile block or JSON API push — ~8 minutes for a standard service
• HTTP/3 on by default. ECH support mattered for a client with aggressive TLS inspection policies.

Traefik 3.7 (Kubernetes CRD + IngressRoute):

• Proxy throughput: ~36,000 req/s — lowest raw number, still 3x headroom over peak traffic
• p99 overhead: ~1.6ms under burst; token-bucket rate limiting smoother than Nginx leaky bucket for our spike pattern
• Idle memory: ~48 MB; ~135 MB at 10k connections
• New route: label on Deployment — often zero manual steps after conventions exist
• ingress-nginx annotation compatibility layer saved two weeks on migration — rewrite-target, proxy-body-size, most of what they already had

TLS, Reloads, and the Config Formats You’ll Maintain at 2 AM

Nginx: Cert rotation via cert-manager worked. Config reload via nginx -s reload is battle-tested. When we fat-fingered a proxy_pass typo, Nginx refused reload and kept the old config — annoying in CI, lifesaving in prod.

Caddy: Automatic HTTPS is the real product. Two lines, cert appears, HTTP/3 works. JSON API reload with zero downtime felt like cheating on a homelab. In Kubernetes we still needed to wire cert storage — not hard, but not literally zero config at scale.

Traefik: Watching Docker/K8s events and building routes dynamically is magic until it isn’t — a mislabeled pod got public traffic for eleven minutes before we caught it in access logs. Convention documentation became mandatory: label schema, entrypoint names, middleware chains.

The Migration Week Incident

Cutover Tuesday, 09:40 local — dispatch managers start their morning export ritual. Traefik at 60% traffic, ingress-nginx still handling legacy annotations on two services. A deployment rolled out with a malformed Ingress annotation — nginx.ingress.kubernetes.io/proxy-read-timeout set to "30" (seconds) on an export service that needed 300.

Nginx ingress honored the low timeout. Traefik’s translated middleware did too — correctly, per config. Large CSV exports died at 30 seconds flat. Users saw 502 Bad Gateway. Error rate: 0% → 8% in three minutes.

We extended the timeout annotation, confirmed upstream p99 was 42 seconds for huge exports, set 300s with a async export fallback on the roadmap.

For the node-level debugging — journalctl, connection tracking, file descriptor limits — turned out nf_conntrack table was 92% full on two nodes from the export spike. Not a Traefik bug. A connection tracking budget we had never sized for long-lived downloads.

When I Would Choose Nginx

• You terminate massive traffic on dedicated edge nodes and need every req/s
• Large-file streaming (GB-scale) where Nginx’s buffer tuning still wins
• Team has a decade of Nginx config and F5 support contract
• You are standardizing on F5 NIC post ingress-nginx EOL and want vendor continuity

When I Would Choose Caddy

• Small cluster or VM fleet — TLS just works with minimal config
• Developer velocity over raw throughput; HTTP/3 and ECH matter
• You hate maintaining Certbot sidecars and annotation archaeology
• <10 services, mostly stable routes, homelab-to-production pipeline

When I Would Choose Traefik

• Kubernetes-native stack with frequent service churn — labels beat YAML surgery
• Migrating off ingress-nginx in 2026 — annotation compatibility is real
• RKE2 / Rancher shops where Traefik is already the default path
• You want dynamic upstreams without reloading the world

Verdict

There is no “best” proxy — only the one that matches your operational model.

For this client’s EKS fleet with weekly deploys and a migration deadline, Traefik won — not because it was fastest, but because operator time and ingress-nginx compatibility beat 11% throughput we weren’t using. We kept Nginx on a dedicated edge VM for the CSV export CDN path where buffer tuning and raw streaming matter. Caddy runs my personal staging cluster because I refuse to maintain Certbot on a Saturday.

What You Should Do This Week

1. If you still run ingress-nginx, inventory annotations and test Traefik’s compatibility layer on a non-prod namespace. March 2026 EOL is not a surprise anymore — it’s a calendar event.
2. Log X-Forwarded-For and upstream status on every 502. Half of "proxy broke" is timeout misconfig.
3. Check nf_conntrack usage before blaming the ingress controller for connection drops on large downloads.
4. Write a rate-limit test that hits the same endpoint from two IP paths — corporate LB misconfig shows up before customers complain.
5. Document your label/schema conventions before Traefik auto-discovery becomes auto-misrouting.

Proxies do not create outages. They reveal the config you did not test. Make sure you know what’s on the other side.