DEVOURED
From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines
Slack replaced over 700 SSH-based EMR data pipeline operators with a REST API architecture across 8 AWS regions without downtime.
What: Slack migrated 700+ SSH-based operators on AWS EMR to a secure REST architecture using Quarry, their internal job submission gateway, and YARN's Distributed Shell for resource management. The migration spanned 8 regions with zero downtime.
Why it matters: Reveals that even modern cloud platforms like AWS EMR still default to SSH-based access patterns that require custom API layers to achieve proper security controls and resource lifecycle management at scale.
Decoder
- EMR (Elastic MapReduce): AWS managed cluster platform for big data processing using Hadoop, Spark, and other distributed frameworks.
- YARN (Yet Another Resource Negotiator): Hadoop's cluster resource management system that schedules jobs and allocates compute resources across the cluster.
- Quarry: Slack's internal REST API gateway for submitting and managing EMR jobs, replacing direct SSH access with proper authentication, resource tracking, and lifecycle management.
- Distributed Shell: YARN component that runs arbitrary commands as managed applications with proper resource allocation and cancellation support.
Original article
Slack modernized its data pipelines by migrating over 700 SSH-based operators on AWS EMR to a secure REST-based architecture with zero downtime across 8 regions. Its team replaced direct SSH access with Quarry, their internal REST job submission gateway, and used YARN's Distributed Shell to run arbitrary commands for proper resource management, reliable tracking, clean cancellation, and server-side lifecycle handling.
Explore all 73 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi