DEVOURED
Introducing Dimster, a performance benchmarking tool for Apache Kafka
Jack Vanlightly released Dimster, an open-source Kafka benchmarking tool designed for "Dimensional Testing" across various workloads and configurations, with Kubernetes support for standardized deployment.
What: Dimster is a new open-source performance benchmarking tool for Apache Kafka, created by Jack Vanlightly. It supports "Dimensional Testing" to explore the impact of specific configurations or workload aspects. It offers four test modes (Run, Explore, Drain-backlog, Correctness), produces shareable JSON results with logs and interactive charts/Grafana dashboards, and is designed to run on Kubernetes.
Why it matters: This tool addresses the complexity of comprehensive Kafka performance testing by providing structured, reproducible benchmarking that simplifies identifying bottlenecks and understanding performance envelopes across diverse deployment scenarios and configurations.
Takeaway: If you manage Apache Kafka clusters, explore Dimster on GitHub for more structured and reproducible performance testing, especially if you use Kubernetes for deployment.
Deep dive
- Dimster is an open-source performance benchmarking tool for Apache Kafka, created by Jack Vanlightly.
- It is designed for "Dimensional Testing," allowing users to systematically vary single or co-varying dimensions of configuration or workload to analyze performance impact.
- Results are self-contained and shareable, including JSON, CSV, source configs, log files, and interactive charts/Grafana dashboards (as HTML).
- Supports four test modes:
Run(fixed throughput, live interaction, optional availability),Explore(finds peak sustainable throughput under latency targets),Drain-backlog(times backlog processing), andCorrectness(detects data loss, corruption, out-of-order, duplicates). - Provides CLI commands for pre-benchmark resource calculation (
resources), comparing runs (compare), and pivoting results (pivot). - Kubernetes is a standardized runtime for Dimster, simplifying deployment and orchestration across various environments (local, EKS, GKE).
- Dimster can deploy Kafka clusters to Kubernetes or connect to external Kafka services.
- The tool is written in Java and leverages modern JVM features.
Decoder
- Dimensional Testing: A benchmarking technique where configurations or workload aspects (dimensions) are systematically varied to observe their impact on performance.
- p99 end-to-end latency: The 99th percentile of the total time taken for a message to travel from producer to consumer.
- mTLS (mutual TLS): A security protocol where both client and server authenticate each other using TLS certificates.
- OpenMessagingBenchmark (OMB): An existing open-source benchmarking framework for messaging systems, which inspired aspects of Dimster.
Original article
Full article content is not available for inline reading.
Explore all 62 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi