How we built LangChain’s agent-first data stack

Key Takeaways

• Reliable data agents need more than access to tables. They need clear models, metric definitions, business context, and signals about which sources to trust.
• An agent-first stack can dramatically expand self-service. Our data agent now handles roughly 40x the request volume our three-person data team could manage directly.
• The data team’s role has shifted from answering every question to improving the system. The data team now focuses on building the models, context, guardrails, and feedback loops that help make agent answers more useful over time.

Over the past year, our data team has been rethinking how our stack needs to evolve to support agents. Most company data stacks are built around dashboards, reports, and SQL workflows. Those are still useful, but agents change what the data layer needs to provide.

An agent can answer more questions when it has clear definitions, trusted sources, business context, and access to the logic behind the data. Without that context, it can still generate SQL, but the answers are harder to trust. The agent might miss company-specific definitions, use the wrong table, or answer a question in a way that is technically valid, but not actually useful for the business.

We wanted to make data easier to access across the company, while giving agents enough context to answer questions accurately and explain how they got there.

We did a big architectural shift, moving from a data stack centered around a traditional BI tool to one designed for self-serve analysis, shared context, and agent use.

Key results

• Our self-serve data agent now handles ~40x the volume of requests our 3-person data team could field directly.
• In the last 30 days, nearly 100% of provisioned users (a third of our company) used the data agent. We saw roughly 2,200 agent conversations in the period, or an average of 23 conversations per month per user.

Where we started

Before this migration, almost every data ask went through the data team. At the time, that team was just one person.

The existing traditional BI tool worked for predefined reporting, but it was rigid. Exploratory analysis was hard to collaborate on, hard to share, and hard for anyone outside the data team to explore on their own unless the data had already been modeled and exposed in the BI layer.

This all created a bottleneck. People across the company had good questions, but answering them usually required a data team member to translate the question, find the right model, write or adjust the query, validate the result, and send back an answer. The data team spent a lot of time handling one-off requests instead of focusing on deeper analysis, modeling, and cross-functional projects.

We needed an agent-first stack that could support several different kinds of users at once.

Some people wanted polished dashboards. Some wanted notebooks and SQL. Some wanted a conversational interface that could help them answer a question without needing to know where the data lived. Engineers and technical operators needed flexibility, while many business users needed a safer and more guided way to explore.

We also wanted one central place for data work. We considered keeping our old BI tool and also adding a new one, but that would have split usage, context, and trust across two systems.

How we evaluated our tooling

We were looking for more than a dashboard replacement.

When we evaluated different vendors, we wanted one that was building AI features directly into the product and treating the agent experience as part of the core workflow. We also wanted a notebook layout that would make it easy for more technical users and engineers to build what they needed without waiting on the data team for every iteration.

For non-technical users, we needed a place where someone could ask a business question, get a reasonable first answer, and understand which sources the agent used.

We ended up choosing Hex. It worked for dashboards, notebooks, and conversational analysis, which made it easier to choose a single central data workspace.

People now interact with the Hex agent through several surfaces:

• The Hex UI, including Threads and notebooks
• Slack
• CLI workflows
• MCP
• LangSmith Fleet, through MCP and CLI integrations

Having a wide surface area is important for adoption. Agent access is most useful when people can use it where they already work. A product manager may want to ask about user behavior in Hex. A GTM teammate may want to ask a pipeline question from Slack. A technical user may want to interact through the CLI or MCP.

We have already seen a wide range of use cases across teams:

• Marketing uses the agent for weekly pipeline analysis
• Product uses it to understand user behavior and usage trends
• Sales and deployment engineering use it for customer health and usage questions
• Customer engineering uses it to analyze churn, expansion, and account trends

What changed after we migrated

We migrated 100% off our old BI tool in six weeks.

Today, 100% of the company uses our agent-first data stack via Hex in some form.

Roughly 70% of users have read-only access, and about 30% have agent access. Those roles are self-serve through IT, so anyone can request agent access when they need it.

A lot of those conversations represent questions that previously would have gone through the data team. However, the data team has not disappeared from the workflow entirely. The questions that reach us are more complex and higher leverage. We spend more time on the work that needs deeper business context, stronger data modeling, or cross-functional alignment.

How we think about context

The agent experience depends on context.

The more clearly we can describe our business, data, metrics, and internal processes, the better the agent can answer questions about the company. Providing context is what helps agents generate useful analysis from raw table access.

How we define the data models

dbt is one of the main places where we manage context about our data models. That context exists in both SQL and written definitions.

Every table and column should help a person understand what the data represents, how it should be used, and where the edge cases are. Those same definitions also help the agent.

A weak column definition might say: account_status: The status of the account.

A stronger definition would say:

account_status: The current lifecycle status of the account in Salesforce. Active means the customer has an active paid contract. Churned means the customer previously had a paid contract that has ended. Prospect means the account has not yet become a customer. For customer reporting, filter to Active unless the analysis explicitly includes churned or prospective accounts.

The semantic model defines metrics and relationships

The semantic model gives the agent context about metrics and how models relate to each other. This is where we define concepts like ARR, pipeline, active usage, customer health, and other metrics that people ask about repeatedly.

Capturing business context

Some context might not belong neatly in a table or metric definition. We use Hex workspace guides for this. They give the agent written instructions about parts of the business, specific metrics, common workflows, and how people should interpret certain data.

A guide might explain:

• How we define pipeline for weekly GTM reporting
• Which dashboards are considered canonical for a given metric
• How to interpret product usage across different deployment types
• What filters to apply when analyzing customer health
• When a question should be routed to the data team for validation

Endorsements tell the agent what to trust

Endorsements help the agent understand which data sources and assets are trusted. We keep guardrails around endorsements so they stay meaningful. Only the data team can mark something as endorsed. Endorsed dashboards also require data team review before changes go live.

GitHub provides deeper implementation context

The agent can also look at our dbt repo for deeper context about how a data point is sourced. The dbt repo gives it access to the SQL and logic that define a column or metric from start to finish.

How we improve the system

Context needs a feedback loop. We wanted a way to see how people are using the agent and where the agent needs more help. Observability tools help with this. We look at trends in conversation topics, common warnings, issues, and context gaps to decide where to improve the stack.

The feedback loop:

1. Users ask questions through Hex, Slack, CLI, MCP, or LangSmith Fleet
2. Agent uses dbt definitions, semantic model, workspace guides, endorsements, dashboards, and GitHub context
3. Observability surfaces gaps, warnings, and repeated topics
4. Data team reviews patterns and suggestions
5. Data team updates models, definitions, guides, metrics, endorsements, or dashboards
6. Agent responses improve over time

Where we're going next

We are still early in this workflow. Over the next few months, we are focused on leveraging evals to test whether context changes improve responses, and automating more of the process around context suggestions.

What we’ve learned

• Solid data modeling foundations make everything easier. Clear grains, well-named models, consistent definitions, and reliable transformations make the agent more useful.
• Good context is one of the highest leverage investments. The best context is specific. It explains how the business works, what values mean in practice, which filters are expected, and where the data should or should not be used.
• The semantic model works best on top of strong foundations. If you are starting this work, fix the foundations first.
• Start with the questions that matter most. If you can cover roughly 80% of the questions people ask, you’ll create a much better experience for most users.
• Self-service still needs data education. Our goal is not to make every person a data expert, but we want to give people better access to data while preserving trust.

Closing

Building an agent-first data stack is mostly about making context explicit. The agent needs to know what the data means, which sources to trust, how the business works, and when a question requires extra care. The better we describe those things across our stack, the better the agent becomes.