DEVOURED
How Yahoo optimizes resources with flexible VMs in Managed Service for Apache Spark
Yahoo reduced cluster provisioning failures by 85% by adopting flexible VM configurations in its managed Apache Spark pipelines on Google Cloud.
What: Yahoo implemented 'flexible VMs' in Dataproc to maintain infrastructure reliability during peak demand. The system ranks acceptable machine shapes and automatically searches regional zones, allowing Spark clusters to provision even when specific machine types are unavailable.
Why it matters: Static VM assignments are brittle in modern cloud environments where regional capacity for specific instance types can fluctuate; defining ranked fallbacks turns infrastructure scarcity into an automated policy rather than an operational failure.
Takeaway: If you use Dataproc, update your cluster creation scripts to include an 'instanceFlexibilityPolicy' that ranks secondary machine types for your workers.
Deep dive
- Flexible VM configs allow Dataproc to select from a prioritized list of machine shapes when a primary choice is out of stock.
- 'Auto-zone placement' enables Spark clusters to span multiple zones within a region to find capacity.
- To maintain performance, the system requires keeping core-to-memory ratios consistent across all ranked worker machine types.
- Spark and YARN resource properties may require manual overrides to handle different VM underlying hardware consistently.
- The approach prevents pipeline stalls by decoupling logical compute requirements from specific physical instance types.
Decoder
- Capacity Stockout: A scenario in public cloud environments where a specific VM type is temporarily unavailable for provisioning in a particular zone or region.
Original article
Full article content is not available for inline reading.
Explore all 55 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi