Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day

Back to Devoured - October 12, 2026

DEVOURED

Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day

Data infrastructureperformance Salesforce

Salesforce reduced weekly compute costs by 31% by right-sizing Spark workloads to match actual usage rather than over-provisioning executors.

What: After analyzing 8 million daily Spark jobs, Salesforce engineers discovered significant idle time in over-provisioned executors. By resizing based on measured demand, they achieved savings without increasing job failure or retry rates.

Why it matters: Resource efficiency in massive distributed compute clusters is increasingly shifting from 'default cluster settings' to data-driven, workload-specific tuning.

Decoder

Spark Executor: The process within a Spark application that is responsible for running tasks and storing data in memory or on disk.

Original article

Salesforce runs about 8 million Spark jobs a day and found some workloads left executors idle more than 90% of the time. Sizing targeted workloads to measured usage reduced weekly compute costs by 31%, while runtime, failures, and retries remained within acceptable levels.

Explore all 71 articles in today's digest