Inside LinkedIn's data context layer: curating trustworthy context for humans and agents

Answering a data question often depends less on writing the query than on finding the right datasets and knowing how to use them correctly. We built a context layer that helps people and AI agents discover the right assets and understand their meaning, usage, and limitations. Suppose you want to know how many members engaged with a new feature last week.

The query may take a minute to write, but identifying the right datasets, applying the required filters, and understanding what each row represents can take a whole day before getting to the result you can trust.

At LinkedIn’s scale with a very large catalog of datasets and metrics, finding the right dataset is challenging, especially when similarly named datasets or different versions contain slight differences in underlying data. This often requires finding someone who knows which dataset to use and what each row represents, but their guidance may remain buried in a chat thread, leaving others to repeat the same search. Business teams rely on analysts to answer questions that inform critical decisions, often requesting additional details or clarifying requirements as they review the results. Each follow-up can require analysts to find additional data and understand its context, prolonging the analysis and delaying business decisions. AI agents can write syntactically correct SQL in seconds, yet confidently select the wrong dataset or miss a required filter. Even when the resulting query runs successfully, its results can be incorrect if the agent lacks the context needed to use and interpret the data correctly. To help people and agents find the right data and use it correctly, we built a shared data context layer that combines automatically generated dataset profiles with team-maintained knowledge about meaning, usage, and limitations.

This post explains how that context is generated, curated, and organized, and how agents retrieve the information relevant to a question without loading everything into their context window.

Generated and human-curated context

At LinkedIn’s scale, manually documenting millions of datasets is impractical, so the AI enrichment pipeline generates baseline context from schemas, source code, and query logs. Each profile describes the dataset’s purpose, fields, join keys, and representative queries, providing a starting point even when its owners have not written documentation.

Teams supplement this baseline with knowledge the pipeline cannot easily capture, including business definitions, exceptions to standard usage, known data-quality issues, and corrections to generated descriptions. For example, a metric may be appropriate for engagement reporting but unsuitable for financial close, or a join key may be valid only for data after a particular date. This guidance helps people and agents use the data correctly. For asset-specific context, generated and curated contexts are stored separately but are associated with the same asset identifier, so a single fetch returns the generated profile alongside any applicable curated guidance. Corrections in curated context can also reveal gaps in the generated descriptions and help identify improvements to the enrichment pipeline.

The generated layer: building baseline context

Teams can capture details about the data, including valid joins, required filters, known gaps, and what each row represents. Yet manually curated context rarely covers the full catalog and can quickly become outdated. The generated layer fills that gap through an AI enrichment pipeline by regularly processing source code, query logs, wiki pages, and other information to generate baseline context for analysis.

From source signals to generated context

The pipeline automatically nominates datasets with selected catalog labels for business criticality, cross-organization sharing, or powering critical dashboards. Teams can also request context through DataHub, LinkedIn’s metadata catalog platform, or supply lists of important assets. These additional paths allow teams to nominate assets that are not covered by catalog-based selection.

The pipeline moves through four stages of nomination, collection, generation, and persistence. Nomination checks for existing context and skips assets that do not need regeneration, while allowing explicit on-demand requests to refresh their profiles. Collectors gather evidence about what those assets contain and how they are used, including masked data samples, query logs, column and metric usage, source code, and wiki definitions. A large language model turns those signals into structured dataset profiles, which the final stage persists in the data context layer, where embeddings are also generated to support semantic retrieval.

The pipeline supports pluggable collectors and asset-specific prompt templates, allowing new context sources and asset types to fit into the same four-stage workflow.

Schema metadata describes an asset’s structure, while query logs and supporting documentation help explain how it is used. For example, recurring joins and filters can reveal usage patterns that are not apparent from table definitions alone.

Mining how a dataset is really used

While the schema lists the columns and their data types, users need to know how they are actually used in queries. Certain unwritten rules are hard to discover by merely looking at the schema and table definitions. For example, people use partition filters to avoid double-counting, use specific keys to join, and include a few specific columns in their queries to answer business analytical questions.

We identify these usage patterns by analyzing runtime query logs and source code, drawing on historical query logs from our shared query engine. The logs include structured input-table metadata that links each query to the datasets it uses.

With so many queries to analyze, we normalize their SQL text to group queries with the same structure and surface recurring usage patterns. Before counting executions, the collector canonicalizes each query by stripping comments, masking literals, collapsing IN lists, and normalizing whitespace. Queries that differ only in their WHERE values then share a canonical form, allowing their execution counts to be combined. We rank these canonical patterns by execution count to identify the most frequently used query structures. To capture usage outside the shared query engine, a second collector searches source code in Spark jobs, Apache Airflow pipelines, analytics scripts, dbt sources, and metric definitions. The code provides insight into each asset’s intended use and identifies assets used primarily by pipelines that might be missing in the query logs.

The collector distinguishes ad hoc queries from dashboard-generated traffic and identifies recurring analytical questions and the joins and filters used to answer them. These usage patterns provide reusable context that helps people and agents find relevant data and apply established query conventions when answering similar questions.

Learning from sample values

Schemas describe a column’s type, but sample values help show whether a string field contains a business category, a formatted date, or a structured identifier. Concrete examples help the model interpret those conventions and generate more useful descriptions when column names and types alone are ambiguous. The collector requests five rows by default to clarify field meanings and value formats, while a separate profiling step can identify common values and their frequencies for selected dimension columns. Profiling uses a recent partition or a bounded sample to complement the row examples with value-frequency information, helping the model describe the contents and patterns observed in each field. The sample-data collector applies masking rules based on field-level annotations and security classifications, along with additional safeguards, before including sampled values in model context.

Human-curated context as code

Automated enrichment provides a baseline from schemas, query patterns, and other available signals, but those signals do not capture everything teams know about their data. Business rules, exceptions, and lessons from past investigations also need to be contributed and maintained by the people who understand them.

To support this human-curated layer, we chose a familiar authoring workflow where teams capture that knowledge in a Git repository and review changes through pull requests. Asset owners approve curated context as they would code changes, while automated skills and checks catch missing or duplicate content before review.

We store user-provided context separately from generated context so that the pipeline does not overwrite the corrections, exceptions, and usage guidance contributed by teams. The pipeline refreshes only the generated layer, leaving curated content intact. The context retrieval will return both contexts into a single view.

Three levels of ownership

Context serves different audiences, from company-wide metric definitions to a team’s investigation playbooks. The repository reflects these differences in its directory layout, organizing context into global, team, and analysis scopes.

• Global: Authoritative, company-wide definitions of shared datasets and metrics, served alongside the generated context for every asset. Ownership resolves dynamically to the asset's catalog owner.
• Team: A team's operational knowledge such as playbooks, known issues, domain definitions, and notes on the datasets and metrics they own, grouped by the product area they align to.
• Analysis: A certified investigation answering a specific question, with the headline finding up front, backed by its methodology, assumptions, data sources, and the query behind it. Authored, dated, and peer-reviewed, so a past answer is reusable instead of stranded in a chat thread.

Within a team's folder, context is organized by type, so both humans and agents know what they are looking at. Every team directory carries the same eight sections.

What a context file looks like

Each context file is plain Markdown with a small block of YAML frontmatter which contains a few fields including summary, a sentence or two that becomes the file's preview wherever it appear. Authors can write the summary themselves or generate one with a provided skill and review it before submitting. If no summary is provided, the context layer generates one automatically.

From a pull request to queryable context

Teams contribute context through pull requests, with ownership and review requirements determined by the scope of the change. Team context leverages team information to identify the responsible team; global dataset and metric context uses asset ownership information in DataHub to identify the appropriate reviewers. After merge, an automated workflow synchronizes changed files into the data context layer, where their full content and summaries become available for retrieval. Derived indexes are maintained automatically.

Once published exclusively within LinkedIn, where agents access it through MCP tools, applications through a gRPC API, and employees through our DataHub deployment. Authors maintain the knowledge in one place rather than copying it into separate agent configurations or prompts.

Agents access the published context through MCP tools that support asset discovery, context retrieval, and index navigation. Relevant team context can be retrieved through embedding-based semantic search or located by browsing indexes, while known documents can be fetched directly by identifier. Tool responses bring the selected content into the agent’s working context without requiring a repository clone or a custom storage integration.

Progressive disclosure: navigating to the right context

Even though the Git repository organizes context by organization and team, an agent still needs a way to identify which files matter. Without an index, it would have to read every file to determine what is relevant, quickly filling its context window and reducing both efficiency and answer quality. The context layer organizes indexes into a tree, with each index containing a brief summary of its node and a map of its children to their summaries. This lets agents explore the relevant branches without loading the full hierarchy into their context window. When content changes, the affected indexes are regenerated automatically from the leaves upward.

Agents can traverse this hierarchy through navigate team index MCP tool, which retrieves an index at the repository, organization, team, or section level without requiring a local checkout. The agent reads the returned summaries, selects a relevant child, and makes another call for that scope. Once it identifies a document, it passes the document’s identifier to fetch team context to retrieve the full content. Agents with a local clone can instead read the corresponding index and Markdown files directly.

Progress and results

Today, people and agents at LinkedIn can retrieve an asset’s meaning, usage patterns, and known limitations through a shared context layer, reducing the time spent searching schemas, reading code, and piecing together guidance from dashboards and chat threads. Our MCP tools bring generated context and owner-reviewed guidance into an agent’s workflow, reducing the effort of gathering information and providing a more consistent basis for deciding which data to trust.

The question we opened with was never just about SQL syntax, but about knowing which dataset to trust, which filters matter, and which caveats could change the answer before filling in the FROM and the WHERE. Our context layer gives onboarded assets a generated baseline, while their owners contribute the business rules, exceptions, and lessons that automated signals alone cannot reliably capture.

Participating teams that had built their own context layers have consolidated their knowledge in our shared context layer. This simplifies management and makes that knowledge easier to discover and reuse across LinkedIn, while teams retain ownership of their domain guidance.

That shared context is already being used at scale across LinkedIn, with a large number of search and context-retrieval requests recorded consistently. Early evaluations also show improvements in agent performance. In one business-area benchmark, the permission-adjusted acceptable-answer rate rose from 25.9% with the earlier analytics-agent experience to 79.5% after the integration. A separate comparison found that the agent’s median end-to-end time to answer a question fell from 10.7 to 4.7 minutes. Together, these early findings suggest potential benefits in answer quality and speed and reinforce our confidence in giving agents access to maintained, relevant context.

What comes next

Authoring context is only part of the work, because its value depends on whether people and agents can find and apply it correctly. We are continuing to improve search, evaluation, and agent integration together so the system surfaces the right assets, identifies gaps, and retrieves relevant knowledge as a question is answered.

The goal is not to eliminate judgment, but to make the knowledge behind it easier to find, reuse, and improve so that finding trustworthy data becomes less of an investigation and more of an everyday expectation.