AI in Production: Panellists Highlight Silent Failures, Harness Engineering, and Agent Governance
Production AI agents fail in ways that bypass traditional monitoring, requiring dedicated observability, tiered permissions, and continuous evaluation harness testing.
Deep dive
- Semantic drift can cause LLMs to interpret business terminology differently over time.
- Embedding drift in vector databases can lead to severe false positive/negative spikes in compliance and search systems.
- Traditional human code review cannot keep pace with autonomous agent output volume.
- Security relies on systematic red teaming rather than prompt engineering guardrails.
- Agents should operate with distinct, restricted identities and tiered write permissions.
- Evaluation must be continuous, feeding production edge cases back into benchmarks.
Decoder
- Semantic drift: When an LLM interprets a concept differently over time or across contexts, leading to logical errors despite correct syntax.
- Embedding drift: The degradation or shift in accuracy of a vector-based system after the underlying embedding model is updated.
- Mutation testing: Introducing deliberate, small bugs into code to verify that the automated test suite successfully detects them.
Original article
AI in Production: Panellists Highlight Silent Failures, Harness Engineering, and Agent Governance
During a recent InfoQ webinar titled "AI in Production: What Breaks, What Works, and Who Approves It?", a panel of senior engineering practitioners met to dissect the real-world operational challenges of deploying agentic systems. The discussion was moderated by Hien Luu and featured Zichuan Xiong and Prem Chandrasekaran from ThoughtWorks, privacy specialist Katharine Jarmul, and financial AI practitioner Lan Chu.
Luu opened the session by noting three core drivers: AI fails differently from traditional software by generating convincing inaccuracies rather than crashing; coding agents are generating production code faster than humans can review; and sensitive data has new places to leak through prompts, context windows, and retrieved documents.
Xiong highlighted the widespread issue of semantic drift in LLM pipelines. He described how text-to-SQL translation agents frequently flatten domain-specific concepts, such as treating "member" identically across sales and customer support domains, leading to subtle, hidden errors in downstream business analytics that pass standard tracing.
Similarly, Chu recounted a critical failure caused by embedding drift in financial compliance systems. After deploying an agent to block prohibited investment advice based on vector similarity thresholds, an unannounced background update to the underlying embedding model altered the vector distance distribution. This caused severe false positive and false negative spikes in live traffic while all infrastructure health checks remained completely green.
Additionally, Chandrasekaran highlighted how RAG pipelines can inadvertently retrieve sensitive documents, while Jarmul noted that models inherently memorize training data and can be used to de-anonymize information or expose API keys. Chandrasekaran also referenced the GitSpawn vulnerability, showing how malicious repository configurations can execute unauthorized code during routine commands.
As coding assistants generate an increasingly massive volume of production code, conventional human code review processes face unsustainable bottlenecks. Chandrasekaran, tech and business services lead at ThoughtWorks, emphasized that line-by-line manual code reviews cannot scale alongside high-velocity agent outputs. Instead, engineering teams must automate tactical style and syntax checks within CI/CD pipelines, reserving human oversight strictly for high-level architecture, business intent, and non-functional requirements.
Xiong added that teams must shift left to evaluate intent at the harness level before pull requests are generated. Chandrasekaran also highlighted the resurgence of mutation testing as an essential practice. By introducing deliberate code mutations, teams can verify whether automated test suites truly evaluate underlying logic or merely satisfy surface-level code coverage metrics.
When evaluating agentic systems, Chu stressed that evals must be treated as permanent, living infrastructure rather than static pre-release checklists. Effective evaluation requires continuous online tracing at the span level, log filtering, and using LLMs to categorize failure patterns, enabling teams to feed production edge cases back into offline benchmark suites.
When evaluating agentic systems, Chu stressed that evals must be treated as permanent, living infrastructure rather than static pre-release checklists. Effective evaluation requires continuous online tracing at the span level, enabling teams to capture real-world user interaction patterns and systematically feed production edge cases back into offline benchmark suites.
To address privacy and security risks, Jarmul cautioned against relying on prompt engineering for system defence. She advocated for systematic red teaming, continuous threat modeling, defining strict functional scopes for autonomous agents, and scaling observability to handle expanded event volumes. To limit the blast radius of autonomous agent actions, Chandrasekaran advised treating every AI agent like a new engineering hire.
Organizations should grant agents distinct non-human identities, implement tiered access controls distinguishing read-only operations from reversible write actions, version third-party Model Context Protocol (MCP) dependencies, and strictly isolate API secrets outside the agent's runtime environment in sandboxes. Ultimately, while engineering, security, and privacy teams validate safety guardrails, deployment approval remains a strategic business decision.
To limit the blast radius of autonomous agent actions, Chandrasekaran advised treating every AI agent like a new engineering hire. Organizations should grant agents distinct non-human identities, implement tiered access controls distinguishing read-only operations from reversible write actions, version third-party Model Context Protocol (MCP) dependencies, and strictly isolate API secrets outside the agent's runtime environment. Ultimately, while engineering, security, and privacy teams must validate safety guardrails, deployment approval remains a strategic business decision.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi