DEVOURED
When failover isn't safe: Building high-availability PostgreSQL on Kubernetes
Datadog rearchitected their PostgreSQL-on-Kubernetes clusters to use synchronous replication for failover candidates, trading slight write latency for guaranteed data consistency during network failures.
What: During a simulated zonal failure, Datadog discovered that replication lag prevented safe failover. They implemented a hybrid model using Patroni, where write-handling leaders wait for acknowledgement from synchronous standbys.
Why it matters: This highlights the trade-off between strict durability and availability; moving to synchronous replication prevents data loss but requires careful tuning to avoid bottlenecking performance.
Takeaway: If your database cluster can't fail over safely during network latency spikes, monitor 'SyncRep' wait events and consider implementing synchronous replication for designated standby nodes.
Deep dive
- Used Patroni for leader election and coordination via ZooKeeper.
- Adopted a hybrid replication model: synchronous for failover candidates, asynchronous for read replicas.
- Tuned 'synchronous_commit' to 'remote_apply' for the strongest consistency guarantee.
- Implemented 'strict' synchronous mode to block writes rather than risking data divergence when no synchronous replica is available.
- Benchmarked latency: 'remote_apply' increased write latency by 53% and reduced throughput by 34%.
- Added monitoring for 'SyncRep' wait events to detect unhealthy replication connections.
Decoder
- Synchronous Replication: A replication mode where the primary database waits for a confirmation that a replica has received (and potentially applied) a transaction before notifying the client.
- WAL (Write-Ahead Log): A standard method for ensuring data integrity by recording modifications to a log file before applying them to the actual data files.
- Patroni: An open-source template used to manage high-availability PostgreSQL clusters, typically orchestrating failover through a distributed configuration store.
Original article
Full article content is not available for inline reading.
Explore all 55 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi