Summary

• LinkedIn’s observability team runs distributed tracing and metric metadata discovery, and analytics on ClickHouse, after proving the database on tracing first.
• Even at 1% head sampling, that system generates around 0.8 trillion spans and 200 TB of uncompressed data per day.
• The metric metadata index consolidates three systems behind a single ClickHouse cluster, serving 150k+ queries/minute at 68ms average latency across 13B+ metrics.
• Migrating off a legacy stack cut memory use to roughly one-fifth and compute to about two-thirds, with headroom to double metric volume.

With over a billion members worldwide and thousands of interacting services, a single LinkedIn feed load or job search touches dozens of backend systems. When one of them slows down, finding the culprit means being able to see inside all of them.

That visibility falls to the company’s observability team, who own the full stack that keeps the platform measurable, from hosts, containers, services, and storage up through online, nearline, and offline workflows and real user monitoring, along with the ingestion, alerting, triaging, on-call, and visualization layers built on top. That stack covers every kind of observability signal, including metrics, logs, traces, events, profiles, and exceptions.

LinkedIn’s ClickHouse journey began two years ago with distributed tracing. The team put it into production across three data centers, building a system that handles roughly 800 billion spans a day and gives engineers a near-real-time troubleshooting surface.

Staff software engineer Jacob Zelek shared the next step in LinkedIn’s observability journey: how the team consolidated a legacy metric metadata system onto a single ClickHouse index, resulting in a system that costs less to run and answers questions the old stack couldn’t, with plenty of room to grow.

LinkedIn’s initial chapter with ClickHouse

In 2024, the team was ramping up an OpenTelemetry-based distributed tracing system that follows a request end-to-end, from a member’s app down through every backend service it touches. Even at 1% head sampling, that system generates around 0.8 trillion spans and 200 TB of uncompressed data per day. They needed storage that could keep up with that volume on the write side and stay fast on the read side.

Among their targets for the system, end-to-end ingestion had to complete in under 10 seconds, listing traces over a two-hour window had to return within 2.5 seconds at P99, and fetching a full trace by ID had to come back in 750 milliseconds or less. They also wanted a 7-day retention policy, tiered storage, customizable indexes, and query workload isolation. Finally, whatever solution they picked had to support KQL, one of LinkedIn’s preferred query languages.

Today, the deployment spans three data centers. Each has its own ClickHouse cluster of 22 shards fed by PubSub and a tier of ingesters. The team runs a red-black setup, writing to two clusters and reading from one, with the second ready to take over if the first has trouble, and a third handling validation. The schema is flattened, with span attributes carried alongside, and runs 3x local replication over a distributed table. Compression is around 5x. Each node takes in 350,000 spans a second at steady state, bursting past 1.2 million at max load.

The team reordered the table to put low-cardinality columns first, repartitioned on the field they actually query, and tightened Bloom filter false-positive rates on the columns where it was cheap to do so, bringing queries to run in under two seconds.

The next challenge: legacy metric metadata

Years ago, LinkedIn relied on a legacy time-series data logging and graphing system called RRDtool. However, the naming convention it introduced never left, and at LinkedIn’s scale and size, it had stayed.

Today, this metric is still referenced by an RRD, a single string that concatenates two to six standardized dimensions. Originally, users could target a single metric, but at some point the team introduced the ability to apply a regex across many of them to generate multiple series in one graph, or a separate graph per match.

That problem is compounded by the scale at which LinkedIn operates. The company has more than 13 billion metrics that are still referenced using RRD semantics, with roughly 30% daily turnover. Millions of graphs and alerts are defined this way, with dozens of tools and services still querying RRD metrics.

The challenge for LinkedIn’s observability team was letting people discover and analyze across all of it. For example, engineers might want to ask which hosts emit a given metric, which RRDs a service emits, which metrics match a pattern across every service, and how unique metric counts are trending for growth tracking.

The older stack had grown into three separate services behind a single gateway: a custom in-memory search index, a custom in-memory KV store, and Elasticsearch. By 2025, all three existing services had reached their scaling limits, managing three separate systems had become an operational burden, and even together they couldn’t serve some of the queries engineers were asking for.

Developing a ClickHouse ecosystem at LinkedIn

The team considered a number of options. Building yet another custom solution was unappealing, and sharded MySQL couldn’t scale reads. Elasticsearch was already in hand but didn't handle larger spanning aggregations very well.

ClickHouse, meanwhile, supported all existing queries. Benchmarked against the custom in-memory databases, most ran faster on ClickHouse. Its quotas and query logs let the team see which team or service was driving each query, and as a generic OLAP store holding every dimension, it allowed the team to support analytical queries they’d never thought of.

“The expertise we gained from using ClickHouse for tracing made onboarding a new project very easy. We’re starting to develop a ClickHouse ecosystem at LinkedIn.” — Jacob Zelek, Staff Software Engineer

The new ClickHouse-based architecture for legacy metric metadata

The team’s time-series database feeds ClickHouse through a continual change-capture system, while the gateway that fronted every metadata query now rewrites those queries into SQL and runs them against ClickHouse.

That gateway made the migration “transparent.” Every tool and service in the company already routed through it, so when the gateway stopped fanning out to three separate services and started writing SQL to ClickHouse instead, nothing downstream had to change.

Underneath, the table is a replicated ReplacingMergeTree, ordered to keep the discovery dimensions efficient, with deduplication keyed on a unique ID. Projections handle the heavier aggregations the old systems couldn’t, including service-to-metric counts, service-to-unique-RRD counts, and datacenter-to-host listings.

The results, and the road ahead

Today, the consolidated index serves more than 150,000 queries per minute across the fleet, with an average latency of 68 milliseconds. In terms of resources, the migration cut memory use to roughly one-fifth of what the previous three-system stack required, and reduced compute to about two-thirds. Because the metadata index is a single shard the team doesn’t have to think about resharding, it also has enough headroom to double the metric volume.

Now that every dimension is queryable, major customers are rewriting their old regex-heavy queries into more efficient native ones. This means the cluster may actually scale down over time, even as metric volume grows.