Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs

Back to Devoured - July 02, 2026

DEVOURED

Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs

Data llmspark Medium

Expedia uses LLMs to interpret complex Spark execution plans, turning opaque DAGs into human-readable debugging insights.

What: Expedia Group engineers feed Spark SQL physical plans and DAG data into LLMs to automatically identify performance bottlenecks, data skew, and suboptimal operators in production workloads.

Why it matters: Parsing massive, multi-stage Spark plans manually is too slow for production; using LLMs as an analytical layer over Spark’s execution metadata democratizes tuning across engineering teams.

Decoder

DAG (Directed Acyclic Graph): A sequence of tasks representing a data pipeline execution, where nodes are operations and edges are data dependencies.

Original article

Instead of manually parsing complex physical plans and DAGs for debugging long-running Spark SQL jobs, Expedia feeds the plans (along with relevant context) by using LLMs to analyze and explain Spark execution plans to quickly identify bottlenecks, inefficient joins, skewed data, or suboptimal operators, significantly speeding up troubleshooting for production Spark workloads.

Explore all 51 articles in today's digest