Back to Devoured - November 16, 2026
DEVOURED
Real-time Alert in Spark

Real-time Alert in Spark

Data UiPath
UiPath cut memory usage for its Spark alerting engine by 66% by optimizing stream-to-static joins and state management.
What: UiPath re-engineered its Spark Structured Streaming system to process 20,000 alert definitions per minute. Instead of scanning entire datasets for each rule, the system joins incoming event streams with static alert definitions once and maintains local state, reducing memory consumption from 12 GB to 4 GB.
Why it matters: Stateful streaming often leads to memory bloat; optimizing join strategies is critical when the number of rules grows significantly in production.
Takeaway: If your Spark streaming jobs are hitting memory limits, move away from repeated scans and focus on joining event streams to cached static data structures to minimize state storage.
Decoder
  • Spark Structured Streaming: A scalable, fault-tolerant stream processing engine built on top of the Spark SQL engine.
  • SLA (Service Level Agreement): A contract defining the expected performance or availability; in this case, monitoring that checks if events meet specific time-bound conditions.
Original article

UiPath built a Spark Structured Streaming alerting system that evaluates 20,000 alert definitions per minute, including windowed conditions and SLA checks, without repeatedly rescanning source data. The key design was joining streams to static definitions once and managing state carefully, which cut memory from 12 GB to 4 GB and reduced downstream network calls.

Explore all 69 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi