Rebuilding Data Lineage at LinkedIn: From Storage to Compute
LinkedIn overhauled its lineage platform to track dependencies at the compute level, processing 10 million edges across 6 million datasets.
Deep dive
- Instrumentation: Unified SDK for Spark, Trino, Flink, and Airflow emits structured lineage events.
- Storage: Append-only Kafka log for events; MySQL-based graph for traversal.
- Retrieval: gRPC-based engine handles bread-first traversal with edge-type filters.
- Semantics: Edges are classified by type (data flow vs. metadata), granularity, and confidence.
- RDD Handling: Lineage captured at the task-level for non-SQL Spark workloads.
- Metadata: 'VIEW' and 'REFERENCE' edges capture logical relationships, accounting for 20% of the graph.
Decoder
- Blast-radius analysis: Assessing the potential impact or 'damage' of a change or failure across downstream dependencies.
- RDD: Resilient Distributed Dataset; the primary data abstraction in Spark representing immutable partitioned collections.
Original article
Rebuilding Data Lineage at LinkedIn: From Storage to Compute
Data is foundational to how modern digital products are built and operated. Reliable data helps teams make better decisions, respond to issues faster, and deliver experiences that are relevant, dependable, and secure for members and customers.
At LinkedIn, this challenge exists at significant scale: tens of thousands of data-processing workflows process data from millions of datasets, generating tens of millions of lineage events each day. Data flows from online databases through ingestion pipelines and batch and streaming processing engines into serving systems that support experiences across LinkedIn.
In this blog, we share how LinkedIn’s lineage requirements evolved, why that evolution led us to rethink where and how lineage is captured, and how the rebuilt platform now supports compliance, observability, reliability, and infrastructure efficiency at scale.
How Our Lineage Requirements Evolved
LinkedIn’s original lineage platform was designed for broad visibility across the data ecosystem. Over the past five years, as our data ecosystem expanded in scale and complexity, teams began relying on lineage for a broader set of decisions: investigating incidents, assessing the impact of changes, supporting compliance workflows, and identifying infrastructure that could be safely retired.
This shift raised the bar for what the platform needed to provide. Broad visibility remained valuable, but these new workflows also required precise dataset relationships, end-to-end reachability, timely results, and explicit confidence in each lineage signal. The platform now needed to support day-to-day operational decisions across four areas:
- Compliance. Governance workflows benefit from a clear understanding of how data moves through complex systems. Reliable lineage provides factual context that facilitate those processes; it does not by itself determine compliance outcomes.
- Observability. At this scale, identifying stale upstream data, data-quality regressions, and freshness drift requires reliable lineage.
- Data infrastructure reliability. Trustworthy lineage enables engineers to understand how failures propagate across complex data pipelines.
- Resource efficiency. Understanding dependencies helps teams identify unused datasets, redundant pipelines, and storage that can be safely retired.
Across these use cases, engineers repeatedly return to a small set of questions:
- “Where did this data come from?”
- “What breaks downstream if I change this column?”
- “What caused the issue in my dataset?”
At LinkedIn’s scale, answering these questions accurately and quickly required capabilities beyond the original design. This evolution in requirements motivated us to rethink the platform from the ground up.
What We Learned From the Previous Design
Our previous lineage platform treated every access path equally, providing broad visibility across LinkedIn’s expanding data ecosystem. As operational use cases grew, we identified opportunities to improve its precision, granularity, relationship semantics, and transparency about signal confidence.
Three design choices highlighted where the platform could evolve:
1. Capture lineage where computation happens. Storage systems can observe reads and writes, but they cannot determine how a compute engine uses them. As a result, storage-level signals are less precise in identifying the producer-consumer relationship and in distinguishing meaningful data movement from temporary files, staging outputs, checkpoints, and shuffle data. They also cannot see query semantics such as projections, filters, or column pruning, which are required for column-level lineage.
2. Preserve exact dataset-to-dataset relationships. Workflow-level aggregation captures the combined inputs and outputs of a workflow, but not the exact mapping created by each computation. Preserving that mapping enables more precise root-cause and impact analysis.
3. Model data-flow and metadata relationships explicitly. A logical dataset may be represented as a Hive table, an HDFS path, or a governed view. Modeling relationships such as VIEW and REFERENCE as first-class edge types preserves their meaning while connecting these representations in a unified graph.
Together, these lessons shaped three requirements for the rebuilt platform: capture lineage from compute execution, preserve precise dataset-to-dataset relationships, and attach explicit type and confidence semantics to every edge. These capabilities support operational workflows such as blast-radius analysis, incident response, and compliance evaluation.
How the Rebuilt Platform Works
The rebuilt platform captures and serves lineage across more than 15 compute, orchestration, and data-movement systems, including Spark, Trino, Flink, Beam, Airflow, Flyte, change-data-capture, and ETL systems. Supported engines emit high-confidence lineage directly, while other sources are integrated as confidence-rated fallbacks.
The platform consists of three major stages: instrumentation, ingestion and storage, and retrieval.
Instrumentation. Every lineage signal uses the same emission format. Native integrations, including Spark, Trino, Flink, Beam, streaming ingestion jobs, change-data-capture streams, and push jobs, emit lineage directly through a lightweight SDK. Where native instrumentation is unavailable, conversion jobs translate legacy signals, such as relationships between Hive tables and their underlying HDFS paths, into the same format. All events then land in a single Kafka topic.
Ingestion and storage. Incoming lineage events are processed by Flink and written to an append-only log that supports auditing and replay. The processor also maintains a merged graph optimized for online traversal. For offline analysis, ETL jobs materialize the same lineage data in warehouse tables.
Retrieval. Online, a gRPC graph engine serves upstream and downstream traversal queries bounded by depth, breadth, edge type, and optional stop conditions. This supports latency-sensitive workflows such as incident response.
The same graph is exposed through three logical views: dataset-to-dataset for tracing data flow, dataset-to-job for execution debugging, and workflow-to-workflow for reasoning about workflow dependencies. One graph, three views.
Offline, the same lineage materializes as warehouse tables for compliance, cost analysis, and pipeline health reporting. The same APIs are also integrated with internal AI-powered engineering tools, allowing engineers to explore lineage using natural language.
Key Design Decisions Behind the Rebuild
Five design decisions shaped the rebuilt lineage platform. Capturing lineage from executed query plans is the foundation; the remaining decisions determine how lineage is transported, stored, queried, and presented.
1. Capture lineage from executed query plans. Lineage can be observed from runtime execution, declared in workflow configuration, or inferred from storage-layer audit logs. We prioritize runtime observation because it reflects the computation that was actually executed.
Engines such as Spark SQL, Trino, and Flink SQL build query plans that describe their inputs, outputs, and column-level relationships. We instrument supported engines to emit structured lineage directly from those plans. Declared and inferred sources remain available as fallbacks and are assigned confidence levels based on how the lineage was captured.
2. Separate lineage submission from presentation. Producers emit high-volume events containing the complete context of an individual job execution. Consumers, however, need to traverse relationships efficiently across many jobs and datasets.
Because these access patterns have different requirements, we maintain an append-only submission log as the source of truth and build query-optimized representations from it. These representations support both online traversal and offline analysis, and they can be rebuilt without requiring producers to re-emit historical events.
3. Use an interoperable, strongly typed emission schema. We aligned our event format with OpenLineage and extended it to support capabilities needed across LinkedIn’s production environment, including RDD-based jobs, Flink column-level lineage, Flyte workflows, and Iceberg-based datasets. Events are transported through strongly typed gRPC APIs, providing consistent schema enforcement across producers and consumers.
4. Build a traversal engine for lineage-specific queries. Lineage traversal differs from generic graph traversal because every query combines graph navigation with lineage-specific semantics: confidence, edge type, granularity, stopping rules, and nodes with large numbers of dependencies. These rules must work together within the same traversal.
We store the graph as an indexed edge table in MySQL and traverse it using a breadth-first-search engine with pluggable rules for stopping, node selection, database batching, and result pruning. This approach provides predictable operational characteristics while allowing traversal behavior to evolve independently of the underlying graph representation.
5. Attach explicit semantics to every edge. Rather than maintaining separate graphs for compliance, debugging, and workflow analysis, we describe each relationship using three dimensions:
- Type: distinguishes data flow from metadata associations such as views and dataset aliases.
- Confidence: indicates whether lineage was observed from runtime execution, declared in configuration, or inferred from audit logs.
- Granularity: identifies whether the relationship was captured at the query, task, or workflow level.
Consumers can select the combination appropriate for their use case. This allows the same graph to support compliance, incident response, debugging, and infrastructure efficiency workflows without maintaining separate copies.
Two Capabilities Beyond Warehouse-Centric Lineage
Spark RDD lineage
Unlike Spark SQL and DataFrames, RDDs execute arbitrary user code rather than exposing a structured query plan, making precise lineage capture more challenging. RDDs remain important for performance-sensitive Spark workloads at LinkedIn. We capture their lineage by instrumenting input and output operations in the Spark integration layer and connecting them at the task level. For arbitrary user code, the platform captures dataset-level rather than record-level lineage, assigns the appropriate confidence, and integrates the relationships into the same graph.
Non-dataflow lineage: VIEW and REFERENCE
Some relationships describe how datasets are represented or referenced rather than how data flows. These metadata relationships are important to compliance, governance, and machine learning workflows.
- VIEW: A governed logical view over physical tables. Preserving VIEW as a first-class edge type maintains the access-policy context associated with consumer and usage intent.
- REFERENCE: Connects a Hive table with its underlying HDFS path. Preserving this relationship as a first-class edge allows workloads using either representation to resolve to the same logical dataset.
VIEW and REFERENCE relationships account for roughly 20% of the edges in our production graph, making them important to broad lineage coverage.
Scaling the Rebuilt Platform Across LinkedIn
- Build a new foundation. As lineage expanded from governance into operational workflows, we designed instrumentation, ingestion, storage, and retrieval together around precise relationships, explicit confidence, and fast traversal. This unified foundation enabled broad deployment across supported platforms in less than six months.
- Focus on strategic platforms. LinkedIn’s data ecosystem spans dozens of compute engines, data movement tools, and access patterns. We began with widely used platforms, including Spark, Trino, Flink SQL, Beam, Airflow, and Flyte, where native integrations provide high-confidence, query-plan-level lineage. For other systems, declared and inferred signals provide broader coverage with explicit confidence labels.
- Use lineage to support platform adoption. Teams that move workloads onto these strategic platforms gain more precise lineage, stronger usage visibility, better support for governance workflows, and better tools for pipeline reliability and infrastructure efficiency. These benefits helped adoption of lineage and strategic platforms reinforce one another.
To measure progress, we set a clear rollout goal: end-to-end reachability from business-critical datasets back to their source events and tables. A daily coverage job identifies where lineage chains break, new engine integrations close coverage gaps, automated regression tests prevent regressions, and confidence-rated fallbacks preserve continuity where direct observation is not yet available.
This approach achieved our reachability goal for business-critical datasets across supported data platforms. The production graph now represents roughly 6 million datasets and 10 million edges.
What This Unlocked
Lineage Explorer in DataHub provides an interactive view of multi-hop dataset and job dependencies, turning investigations that once required manual tracing into graph queries that return in seconds. The value, however, is not the graph itself, but the operational workflows it enables.
Trustworthy blast-radius analysis. A single graph traversal returns a current, filtered view of downstream dependencies in seconds, supporting incident response and infrastructure cleanup.
Natural-language lineage exploration. By integrating the lineage platform with internal AI-powered engineering tools, engineers can explore lineage relationships using natural-language questions instead of graph queries.
Final Thoughts
By making lineage a reliable part of LinkedIn’s data infrastructure, teams can better understand how data moves and how changes or failures propagate. This enables safer product development, faster incident response, more effective compliance workflows, and more efficient infrastructure use.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi