Back to Devoured - August 17, 2026
DEVOURED
Spark observability skills (GitHub Repo)

Spark observability skills (GitHub Repo)

Data devopsopensource GitHub
Embrasure released open-source diagnostic agent skills that automate debugging and performance tuning for Apache Spark via the Spark History Server REST API.
What: The Embrasure Spark observability toolkit provides structured 'skills' for AI agents to diagnose crashes, identify bottlenecks like data skew or spills, and optimize Spark SQL plans. The project requires `spark.eventLog.enabled=true` and integrates with existing agent frameworks by pointing to a live or historical Spark UI/REST endpoint.
Why it matters: Standardizing observability as structured agent skills allows teams to move from manual log parsing in the Spark UI to automated, repeatable diagnostic patterns.
Takeaway: Install the skills by cloning `https://github.com/EmbrasureAI/spark-observability-skills` and symlinking the directories to your agent's skills path.
Deep dive
  • Provides modular skill definitions for common Spark issues: job failures, performance regressions, and query plan optimization.
  • Includes a Python-based REST client to extract runtime metrics from the Spark History Server.
  • Implements safety features like read-only access and response bounding to prevent agent-driven disruptions.
  • Supports comparative analysis by allowing agents to check a slow run against a known healthy baseline.
  • Detects common Spark pitfalls such as task skew, shuffle spill, and garbage collection (GC) pauses.
Decoder
  • Spark History Server: A monitoring UI and API that provides information about completed Spark applications.
  • Data Skew: A condition where data is unevenly distributed across partitions, causing specific tasks to take significantly longer than others.
  • Shuffle Spill: When a Spark transformation requires more memory than available, forcing it to write data to disk and slowing down execution.
Original article

Spark observability skills

Open-source agent skills from Embrasure for diagnosing and optimizing Apache Spark workloads. Each skill is a single SKILL.md with an ordered list of the highest-impact causes to check, plus a read-only Spark History Server REST client under scripts/ that collects the runtime evidence in one bounded snapshot.

Install

git clone https://github.com/EmbrasureAI/spark-observability-skills.git
ln -s "$PWD"/spark-observability-skills/skills/* ~/.codex/skills/

Point the symlinks at whichever skills directory your harness reads (~/.codex/skills, ~/.claude/skills, ...), creating it first if needed, then restart or reload the harness.

Or paste this into your agent:

Clone https://github.com/EmbrasureAI/spark-observability-skills and symlink each directory under skills/ into your skills directory, then tell me to reload.

Setup

The skills need HTTP access to a Spark History Server, or to the live UI of a running application (the driver UI on port 4040 serves the same REST API):

export SPARK_HISTORY_URL="https://<your-history-server>"   # e.g. http://localhost:18080 locally or via a tunnel
export SPARK_HISTORY_AUTHORIZATION="Bearer <token>"        # only if the server requires auth
  • History data exists only for applications that ran with spark.eventLog.enabled=true.
  • If the server is cluster-internal, open a tunnel first, for example kubectl port-forward svc/spark-history-server 18080:18080.
  • Behind an SSO proxy, reuse your browser session with SPARK_HISTORY_COOKIE or SPARK_HISTORY_HEADERS_JSON; pass --ca-file for a private CA.

Skills

  • Debug Spark failures: a run failed. Trace driver and executor crashes, out-of-memory kills, fetch failures, task exceptions, and aborted stages back to the earliest supported cause instead of the last retry error.
  • Debug slow Spark jobs: a run is slower or more expensive than it should be. Compare against a healthy run to localize the first divergence: skew, shuffle, spill, GC, poor parallelism, scheduler delay, or infrastructure.
  • Optimize Spark SQL plans: a query works but costs too much. Read its final adaptive plan and runtime metrics to cut scans, shuffles, joins, and unnecessary work without changing query results.

Safety

The collector is read-only, bounds large responses by default, redacts sensitive Spark properties, and keeps TLS verification enabled. Review every command against your environment and access policies before running it.

Contributing

Each skill directory is self-contained so it can be symlinked or copied on its own. As a result, scripts/spark_history_api.py is intentionally identical across the three skills. If you change one copy, sync all three.

License

Apache-2.0. Apache Spark, Apache Celeborn, and their respective marks belong to the Apache Software Foundation. This project is not an official Apache Software Foundation project.

Explore all 72 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi