Rebuilding LinkedIn's Tracking Infrastructure: Scaling, Migrating, and Hardening a High-Volume Event Pipeline

LinkedIn's tracking infrastructure turns product interactions such as impressions, clicks, page views, and media playback into signals used for experimentation, product analytics, platform health, and product experiences across LinkedIn. The quality and freshness of this data affect how quickly teams can understand product behavior and make informed decisions.

The system processes hundreds of billions of events per day—millions of events per second at peak Member activity—across several hundred event types. At this scale, even a small inefficiency or correctness issue can have a large cumulative impact.

That system had carried LinkedIn's tracking traffic for years, but the technical environment around it had changed. It ran on Java 8 and the Play Framework while LinkedIn's service ecosystem standardized on gRPC and a modern Java runtime—and adopting gRPC meant a new request contract and a different serialization path, not a transport-library swap. It also served every event category from one shared capacity pool, so compact engagement events, bursty media events, and deeply nested observability payloads competed for the same resources. Changing the request contract, the serialization path, and the deployment model together is what turned an upgrade into a rebuild.

Rebuilding this system was not simply a REST-to-gRPC migration. We modernized the service architecture and performance, built controls for a safe production migration, and strengthened failure handling, data validation, and release safety while traffic continued to flow. This post focuses on how we rebuilt and migrated the pipeline safely, what production traffic revealed about its performance, and how we strengthened correctness and failure handling along the way.

At a high level, the ingestion service is one stage in a broader path. Product interactions are captured and batched by client SDKs, then routed through LinkedIn's edge and traffic layers. The ingestion service sends latency-sensitive signals directly to real-time processing, while Apache Kafka provides the asynchronous boundary for near-real-time and offline processing.

The Asynchronous Pipeline Behind the API

Tracking API is an ingestion service within a distributed event pipeline. Mobile and Web SDKs batch interactions, our edge and traffic-routing layers send each batch to the appropriate category-specific Tracking API service, and the service decodes, validates, enriches, classifies, converts, and submits the records through Kafka's asynchronous producer. Once the Kafka client accepts a record into its local producer buffer, Tracking API returns gRPC success. It does not keep the request open until the Kafka broker replies. A producer callback reports the acknowledgement or failure later, and downstream systems consume acknowledged records independently.

This design prevents Kafka response time from becoming API response time. It also means that success at one boundary does not prove success at the next.

These are not three synchronous checks performed for every production request. Request and callback metrics describe the live path; tagged comparisons and CI/CD tests verify downstream message correctness. Together, they shaped both the migration and the production-hardening work.

Why We Rebuilt Instead of Refactoring

Modernizing the service meant a new request contract, a different serialization path, and a different deployment model. Those changes affected both how requests were processed and how the service was deployed and scaled. Changing them one at a time inside the only production service would have left old and new assumptions mixed together for a long period. We instead built a parallel destination. The legacy path continued serving traffic while we compared outputs, moved one controlled slice at a time, and kept rollback available through routing rather than a new deployment or mobile release.

Takeaway: A parallel rebuild is useful when an in-place migration would create a long and risky intermediate state. It is safe only when the old and new outputs can be compared and traffic can be moved back quickly.

Isolating Workloads by Cost and Failure Risk

The rebuilt service separated traffic into broad workload categories. Each category received independently scalable hardware, deployment controls, and operational metrics.

This was not merely code organization. Isolation changed how the system behaved:

• Capacity: when a member scrolls through a media-rich Feed, render, play, pause, and exit events arrive close together; that burst can no longer consume capacity reserved for experimentation traffic, and a deeply nested observability workload can be provisioned for its higher CPU and allocation cost.
• Measurement: performance numbers no longer average unlike workloads together.
• Blast radius: categories can move through migration at different speeds, and failures or rollback stay contained to one platform and category.

Compression can reduce the wire size of a nested observability event, but the service must still traverse its collections, validate its values, allocate intermediate objects, and convert the complete structure. Separate capacity allowed that heavier work to be tuned without forcing every event category onto the same resource model.

Reducing Repeated Context on the Wire

Each event contains unique data, but many events in the same request share the same client and session context. The legacy contract repeated much of this context for every event. A few hundred repeated bytes appear small until clients send millions of events per second. Clients serialize the same fields repeatedly, the network carries them repeatedly, and the server parses them repeatedly.

The new contract sends shared context once and reuses it across events where the client path allows it. In the observed production windows, the average aggregate inbound network receive metric decreased from 210.7G on the legacy service to 21.7G on the rebuilt service, a reduction of approximately 89.7%, or about 9.7x. The rebuilt service carried 90% of production traffic during that window; adjusting to full traffic puts the reduction near 89%, or close to 9x. The improvement reflects the combined effect of shared-context reuse, Protocol Buffers, and related serialization changes.

Moving Production Traffic with Evidence

Building the new service created a destination. Migration controls made it safe to use. Server-controlled feature flags routed traffic independently by platform and workload category. Each slice could move through small percentages, remain in comparison mode, or return to the legacy path within minutes.

Comparing a tagged 10% sample

Before moving a traffic slice, we sent 10% of eligible events through both pipelines and compared the records they produced. Each sampled event carried a comparison identifier. That identifier let us filter the same events from the legacy output and match them with records from the rebuilt output. During comparison, the legacy pipeline continued writing sampled events to the production Kafka topic, while the rebuilt pipeline wrote its copies to a dedicated shadow topic. Comparison jobs matched records across the two topics using the identifier, while normal downstream processing did not consume the shadow copies, a design intended to keep duplicate events out of production datasets.

We deliberately avoided dual-firing 100% of traffic. Full duplication simplifies matching, but it also doubles ingestion work and creates unnecessary downstream data, storage, and compute costs. At LinkedIn's scale, that is a significant price for validation. The sample carried real product-instrumented values across the SDK versions in use. It therefore covered uncommon value combinations and legacy conventions that unit-test fixtures might not represent.

Starting with high-volume events

After the comparison process and rollback path were proven, we migrated a small set of high-volume events first. This was more informative than beginning only with low-volume traffic:

• small mismatches in event volume or shifts in end-to-end event latency became measurable sooner;
• concurrency, queues, and capacity limits were exercised early;
• rare defects appeared more frequently; and
• Each successful move removed a meaningful share of load from the legacy service.

Takeaway: A tagged, representative sample can provide strong migration evidence without the cost of duplicating everything. Once comparison and rollback work reliably, high-volume traffic can reveal correctness and capacity problems faster.

Profiling the Real Workload

The ingestion path performed little blocking application-side I/O. After validation and enrichment, events were handed to the asynchronous Kafka producer. At high throughput, CPU—not waiting for remote calls—was the main constraint.

Serialization and compression

The migrated path initially converted each protocol-buffer event into an intermediate Apache Avro object before writing the downstream binary record. We replaced that step with a precompiled inline encoder for both low-latency and high-throughput producers, avoiding generic records, nested maps, and other temporary objects while preserving the same Avro binary output.

The legacy producer also used gzip, whose Java compression path consumed substantial CPU. We switched Kafka batch compression to zstd, which uses a faster native implementation while maintaining a similar compression ratio for our event payloads.

Finding a concurrency problem that benchmarks missed

As more high-volume traffic moved, profiling showed that conversion had again become a major CPU cost. A schema comparison deep in the path repeatedly accessed a shared cache. It looked inexpensive in a sequential benchmark, but under concurrent traffic many workers competed for the same resource. We replaced repeated field-level comparisons with a direct conversion path for supported events while keeping the legacy conversion as a fallback.

Takeaway: Optimize the real bottleneck, not a generic benchmark. Sequential tests can hide threads competing for shared resources, so profile under realistic concurrency and protect every optimization with output checks.

Turning Delayed Kafka Failures into Backpressure

Matching output under healthy traffic was necessary but not sufficient. The rebuilt service also had to fail safely under Kafka slowdowns and recover without losing events unnecessarily.

The Kafka producer runs asynchronously for throughput, so broker and network failures can arrive in callbacks after the caller has received success. We connected callback outcomes to request admission through an overload detector. When failures exceed the configured threshold, the host rejects requests before processing, giving its Kafka buffer time to drain. Rejection preserves an opportunity for the load balancer—and subsequently the client—to retry on a healthy host. Recovery requires positive evidence: enough healthy callbacks must arrive before admission resumes.

Takeaway: In an asynchronous producer path, delayed failures must influence which requests are accepted next. Reduce load only for failures that less traffic can relieve, reject before expensive processing, and require healthy evidence before reopening.

Impact and Lessons

The rebuilt service now handles migrated traffic with more granular scaling and failure boundaries. Workload isolation, lower inbound network usage, and reduced Kafka-producer CPU and allocation all contribute to a more efficient operating model.

The broader lessons extend beyond tracking:

• Make structural migrations reversible. Run old and new paths together, move independently controlled slices, and keep rollback separate from deployment.
• Compare what downstream systems receive. API success is only one point in an asynchronous pipeline.
• Use representative samples rather than duplicating everything. Tagging enables precise comparison at lower operational cost.
• Isolate workloads by cost and failure impact. Logical similarity does not imply identical capacity needs.
• Measure the envelope, not only the payload. At high volume, context repeated on every event can cost more than the unique event data itself.
• Profile under realistic concurrency. Sequential benchmarks can hide contention and allocation costs.
• Connect delayed failures to future admission. Callback telemetry should influence whether a host accepts more work.