DEVOURED
The 58-Million-Key Freeze: What a HashMap Resize Taught Us About Memory Allocation at Scale
LinkedIn's Rust-based FishDB service froze for 10-15 seconds due to a HashMap resizing at 58.7 million keys, acquiring a process-wide mmap_lock and blocking all other threads.
What: LinkedIn's FishDB, a Rust-based retrieval engine using jemalloc and Tokio, experienced 10-15 second freezes. The root cause was a standard library HashMap resizing at 58,720,256 keys, which triggered a 3.5 GB mmap allocation, acquiring the Linux kernel's process-wide mmap_lock in write mode, blocking madvise and page faults.
Why it matters: This incident highlights how fundamental data structure behaviors and underlying OS memory management (specifically the mmap_lock) can cause catastrophic, silent failures in high-scale, multithreaded applications, even with modern languages and allocators.
Takeaway: For high-scale Rust services, pre-allocate HashMap capacity with HashMap::with_capacity() if the expected size is large and known, to avoid unexpected reallocations and mmap_lock contention.
Deep dive
- LinkedIn's FishDB service, a Rust application using jemalloc and Tokio, experienced recurring 10-15 second freezes, breaching availability SLOs.
- The problem was elusive: ephemeral, silent (no logs), sporadic, and without obvious external triggers.
- Correlation with RSS spikes led to suspicion of memory allocation issues.
- Traditional CPU profiling was ineffective as threads were blocked (off-CPU).
- An automated eBPF-based off-CPU profiling script was deployed to capture kernel stack traces during freezes.
- Off-CPU profiles revealed threads blocked on rwsem_down_write_slowpath (write lock for mmap), rwsem_down_read_slowpath (read lock for madvise and page faults).
- This pointed to contention on the Linux kernel's process-wide mmap_lock (VMA semaphore), which protects virtual memory area data structures.
- A large mmap allocation (requiring a write lock) blocked all other threads needing mmap_lock in read mode.
- The HashMap pkey_vs_docref (document reference index) was found to be the culprit. It held 56-59 million entries.
- At exactly 58,720,256 keys, the HashMap capacity doubled from ~1.75 GB to ~3.5 GB, requiring both buffers to coexist (total ~5.25GB, leading to observed ~4GB RSS spike).
- The fix involved pre-allocating the HashMap with HashMap::with_capacity(base_index_size
-
- to a sufficient size at startup, avoiding dynamic resizing.
- This prevented the mmap_lock contention and eliminated freezes.
Decoder
- mmap_lock: A process-wide read-write semaphore in the Linux kernel that protects the virtual memory area (VMA) data structures. Operations modifying the virtual address space (like large memory allocations or deallocations) require this lock, causing contention if held for too long.
- eBPF (extended Berkeley Packet Filter): A Linux kernel technology that allows programs to run in a sandboxed environment within the kernel, enabling powerful, flexible, and safe kernel-level tracing and profiling without modifying kernel source code.
- jemalloc: A general-purpose memory allocator that emphasizes fragmentation avoidance and scalable concurrency. It is used by many large-scale applications.
- Tokio: An asynchronous runtime for the Rust programming language, providing the necessary tools to build network applications and services.
- madvise: A system call that advises the kernel on how to handle a process's memory regions. MADV_DONTNEED is often used to tell the kernel that memory pages are no longer needed and can be reclaimed.
- RSS (Resident Set Size): The portion of a process's memory that is held in RAM (not swapped out).
Original article
Full article content is not available for inline reading.
Explore all 62 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi