DEVOURED
Kafka Share Groups and Parallelizing Consumption — Tuning max.poll.records
Kafka Share Groups improve parallelism but require careful tuning of max.poll.records to prevent a few consumers from "greedy capturing" large batches and degrading throughput.
What: Jack Vanlightly's tests on Kafka 4.2.0/4.3.0 show that the default max.poll.records of 500 is often too high for Share Groups, leading to "greedy-capture" where a few consumers hog records. Optimal performance at 60K msg/s with 5ms processing required setting max.poll.records to 30 and adjusting consumer count.
Why it matters: This highlights that new Kafka features like Share Groups change core performance bottlenecks, requiring re-evaluation of long-standing configurations and potentially custom solutions to ensure fair resource distribution in distributed systems.
Takeaway: When using Kafka Share Groups, calculate group.share.partition.max.record.locks / consumers-per-partition and set max.poll.records to that value, then tune slightly lower for optimal throughput.
Deep dive
- Kafka Share Groups allow consumption parallelization beyond partition count, unlike traditional Consumer Groups.
- The primary bottleneck with Share Groups shifts to the combination of
max.record.locks(broker-side) andmax.poll.records(consumer config). - The default
max.poll.recordsof 500 can lead to "greedy-capture," where a few consumers receive large batches, exhausting theinflight record budgetfor a partition. - This causes other consumers to sit idle, reducing overall throughput, even if the theoretical maximum is much higher.
- An "accidental fair-sharing" regime can occur at low loads, masking suboptimal
max.poll.recordssettings, but it collapses under higher load or consumer restarts. - The article demonstrates how a system with 300 consumers peaking at 60K msg/s initially dropped to 4800 msg/s with default
max.poll.records=500. - Tuning
max.poll.recordsto 30 and increasing consumers to 312 restored the 60K msg/s throughput with stable end-to-end latency. - A rule of thumb for
max.poll.recordsisgroup.share.partition.max.record.locks / consumers-per-partition, then tune slightly lower. - Increasing
group.share.partition.max.record.locks(up to 10000) can also help by providing a larger inflight budget. - The author suggests Kafka could benefit from an alternative broker-side fair-sharing enforcement mechanism in future versions.
- Dimster, a benchmarking tool, was used for live interaction to mutate workloads and observe config changes on the fly.
Decoder
- Kafka Share Groups: A new Kafka consumer group type that allows multiple consumers within a group to process records from the same partition in parallel, providing finer-grained control over workload distribution compared to traditional Consumer Groups.
- max.poll.records: A consumer configuration that determines the maximum number of records returned in a single call to
poll(). - group.share.partition.max.record.locks: A broker-side configuration that limits the total number of records that can be "in-flight" (locked by consumers for processing) simultaneously from a single partition within a Share Group.
- Greedy-capture regime: A state where a few consumers receive very large batches of records, occupying a significant portion of a partition's
inflight record budget, while other consumers remain idle, leading to reduced overall throughput. - Accidental fair-sharing regime: A temporary state at low producer rates where the broker doesn't have enough records to fill large
max.poll.recordsbatches for every consumer, unintentionally distributing records more evenly. - Dimster: A benchmarking and load testing tool for Kafka that allows real-time modification of workloads and configurations.
Original article
Full article content is not available for inline reading.
Explore all 61 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi