How We Moved Discord Voice to the Edge
Discord migrated 80% of its voice traffic to Cloudflare's edge network, achieving significant latency drops through custom hardware-software optimization.
Deep dive
- Migration involved moving voice processing closer to users, reducing pings in Frankfurt by 34%.
- Initial high-density server configurations failed due to NIC queue contention, requiring a 50% density reduction.
- Latency spikes were traced to 'event loop starvation' in Rust, where network interrupts were fighting for CPU cycles.
- The engineering team had to build custom infrastructure to interface with Cloudflare's ephemeral container environment.
- The migration spanned one year and involved deep performance profiling across both application and kernel layers.
Decoder
- Event loop starvation: A condition where a program's main processing loop is blocked, preventing it from handling incoming tasks or events in a timely manner.
- NIC (Network Interface Card) queue contention: A performance bottleneck occurring when multiple processes or threads compete to access the network interface hardware buffers.
Original article
At Discord, the distance to a user's closest voice server matters. Every millisecond of network distance adds latency to every packet, and past a certain point calls stop feeling like your friend is in the same room as you.
For most of Discord's history, the closest voice server we could put you on was in one of about 30 cities worldwide, in places where the major cloud providers had data centers. That worked fine if you lived in the Bay Area or Frankfurt, and less well if you lived in Reykjavik, Auckland, or other places where hyperscaler coverage was thin.
Last year, we started migrating Discord voice and video traffic onto Cloudflare's edge network, which runs in over 300 cities. Today, more than 80% of our voice and video traffic runs there, and 70% of the regions show year-over-year quality improvements. Frankfurt leads the way, with ping averages down 34% and packet loss down 42% compared to the previous vendor.
This post is the story of how we got here: why we did it, what we had to build to make it work, and how we investigated quality issues in Europe earlier this year.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi