DEVOURED
Plan Mode All the Time, Substrait over SQL, and the End of the DE Role ft
Chris Riccomini argues AI agents, when used with "plan mode" and declarative workflows, are already capable of most data engineering, suggesting a future where data engineers become general "data" roles and LLMs prefer formats like Substrait over SQL.
What: Chris Riccomini, author of "The Missing README" and co-author of "Designing Data-Intensive Applications", believes AI will handle most data engineering, emphasizing "plan mode" and strong quality gates for managing LLM non-determinism. He suggests LLMs should use Substrait, which expresses physical operations, rather than SQL, due to better optimization and fewer hallucinations.
Why it matters: This article provides a provocative view on how AI might reshape the data engineering profession and its tools, pushing for more declarative approaches and a shift towards agent-centric ergonomics in language and tooling design.
Takeaway: Experiment with detailed "plan mode" prompting and robust external testing frameworks when using LLMs for code generation, especially for critical data tasks, to manage non-determinism and ensure correctness.
Deep dive
- Chris Riccomini believes AI can handle the majority of data engineering work, especially with declarative workflows and strong quality gates.
- For financial data, correctness is maintained by defining invariants and using traditional verification tools, as well as pairing AI with human review for bug spotting.
- He advocates for LLMs to "speak" Substrait, a format representing physical data transformations (e.g., hash join vs. merge join), rather than SQL.
- Substrait could lead to fewer LLM hallucinations and allow for client-side query optimization.
- To make AI output more reliable, Chris recommends "plan mode all the time," where LLMs iterate extensively on a plan before implementation.
- Managing context by starting with fresh LLM contexts or using "Ralph Loops" (iterative autonomous AI development with external tests) can improve reliability.
- Implementing strong quality gates (defining, measuring, enforcing quality) is crucial, like enforcing test coverage with commit hooks.
- Non-determinism from LLMs can be mitigated by moving to incremental data loads, reducing the scope of potential errors.
- Security concerns for AI agents are high, with a need for "Okta for Agents" (identity/access management) to manage skills, marketplaces, lineage, and RBAC/ABAC.
- Agents are already good at inspecting failed workflows, running SQL queries, and writing Python, and could automate much of the "grunt work" of data engineering.
- The "data engineer" role may merge into a broader "data" role encompassing engineering, ML, and analysis, as tools become more agent-friendly.
- The choice of programming language may shift from human ergonomics to "agent ergonomics," favoring languages that lead to faster, cheaper, and more stable LLM output (e.g., Go over Python due to token cost/code size).
- He suggests that while AI might reduce some rote learning, it enables tackling more complex projects and learning about new domains (like FFI bindings) that would otherwise be too time-consuming.
Decoder
- Substrait: An emerging open standard that provides a cross-language serialization format for relational algebra expressions. It can represent both logical and physical query plans, allowing for more precise data transformation instructions than pure SQL.
- Plan Mode: An approach to interacting with AI where the LLM is guided to first generate a detailed, iterative plan for a task, which is then refined and approved by a human, before the LLM proceeds with implementation.
- Ralph Loop: An iterative, autonomous AI development technique where a bash loop (or similar mechanism) repeatedly prompts an AI agent with the same goal, forcing it to persistently iterate and fix errors until external tests pass.
- LLM (Large Language Model): A type of artificial intelligence program designed to understand and generate human-like text, often trained on vast amounts of text data.
- Declarative Workflows: A programming paradigm where you describe what you want the program to achieve, rather than how to achieve it (as in imperative programming). This allows the underlying system to determine the best execution strategy.
- Agent Ergonomics: The idea of designing programming languages, tools, and workflows to be optimized for AI agents to use, rather than primarily for human developers.
Original article
Full article content is not available for inline reading.
Explore all 62 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi