How to monitor AI agent and MCP activity

Broken MCP tool calls silently cost real money. Across our agent fleet, seven small MCP-server bugs burned ~$499K/year in tokens and 12,000 eng-hours/year ($1.2M lost) because agents quietly retry instead of surfacing failures.

Databricks engineers rely heavily on AI agents to streamline and accelerate their work. In turn, these agents require access not only to different Foundation Models but also to MCP servers with tools that enable access to relevant artifacts (e.g., system logs, usage tables, support tickets, wikis). We describe how we looked for cost savings in our agents' use of tools, the challenges we hit along the way, and how OTel tracing in Unity Gateway cut the path from analysis to $1.2M/year in savings to a single hour.

Enabling our developers to build their agents was a huge unlock on productivity, but as usage ramped up, we also faced increasing costs. We started investigating several optimizations, and one suspicion that we had was the hidden cost of failing tool calls. Specifically, when tools misbehave, the calling agent rarely fails loudly. Instead, it retries, guesses, and eventually works around the problem, quietly burning tokens and developer time the whole way.

We investigated this suspicion in our agent fleet using Unity Gateway's tracing and Genie One. We found seven small bugs in our tool servers that were costing an estimated $499K/year in wasted tokens and about 12,000 engineering hours per year in agent wait time. Overall, this is an estimated $1.2M/year in lost productivity.

When we first deployed AI agents widely at Databricks for coding and internal workflows, it was impossible to manage or even fully understand costs because we lacked visibility into the agents’ tool calls and overall activity. To solve this, we leveraged Unity Gateway, which automatically emits an OpenTelemetry trace for all MCP tool invocations, including the tool name, arguments, error (if any), token counts, latency, and a session ID that ties calls together. Those traces land in a single table that records exactly what our agents did over any time window.

This makes AI agent cost management more actionable, where instead of seeing only aggregate token spend, we can attribute wasted spend to specific tools, errors, and agent sessions.

Now that the data is available, the next step is exploration:

• Which tool errors recur the most?
• When an agent hits one, how many turns does it take to recover?
• What does each error cost in tokens and wall-clock wait time?

Normally, the expensive part of this kind of analysis is the SQL and the schema spelunking. But with Genie One, we just pointed it at the trace table, asked these exact questions in plain English, and got answers back in minutes.

What the traces revealed: How MCP tool failures drive up AI agent costs

Genie One turned a vague suspicion into a ranked, quantified bug list in minutes. Here is an example from a single 24-hour window, showing bugs in our Jira and Google Drive/Docs tool servers:

Take the highest-volume bug, 535 failures a day, as an example. The Jira issues.search tool takes a fields parameter. A list has no .split(), so the server raised 'list' object has no attribute 'split', a raw Python traceback that tells the agent nothing about what it did wrong. So the agent guessed again. On average, it took 12 turns to recover, and 30% of sessions hit the error more than once.

The real lesson: How to design MCP tools for AI agents and LLMs

The obvious takeaway is "write better error messages," and the data backs it up. Recovery cost tracks error-message quality almost perfectly:

But "good error messages help" is old news. The more interesting question is why the model called these tools "wrong" in the first place. In most of these cases, it didn't.

MCP tool signatures are often deliberately under-specified. When a signature is vague about fields, the model fills the gap with a reasonable guess, and a JSON array is a reasonable guess for a list of fields. The bug was not that the model called the tool incorrectly. It was that the server accepted only one of several reasonable interpretations and crashed on the rest.

So the design principle is the reverse of the reflexive one: tools for agents should adapt to the way LLMs naturally call them, e.g., coerce the list into a string, default the omitted parameter, absorb the unexpected argument, and so on.

The easy part: How we reduced wasted AI agent spend in one hour

The fixes themselves were simple. Once Genie One had handed us a ranked list of which errors to fix and what the model was actually sending, applying the fixes across the tool servers was a quick pass with a coding agent. The whole loop (find, quantify, fix) took about an hour.

The scarce, expensive step was never writing the fix. It was knowing what to fix. Tracing plus Genie One turned that step from a research project into a question you can ask out loud.

Closing the loop: How to continuously monitor and reduce AI agent costs

As more real work shifts onto agents, silent tool failures become a first-class cost center, the kind that hides inside "usage growth" and never pages anyone. The loop for catching them is cheap and repeatable: Unity Gateway makes agent behavior observable, and Genie One makes that behavior queryable without SQL.