Codex now works directly in Chrome on macOS and Windows
OpenAI's Codex now runs natively in Chrome on macOS and Windows, enabling it to automate repetitive browser tasks across tabs in the background by writing code to navigate complex data flows.
Summary
Decoder
Original Article
OpenAI Released Realtime Audio Models
OpenAI has launched a new suite of real-time audio models through its API, including GPT-Realtime-2 for conversational reasoning, GPT-Realtime-Translate for live multilingual translation, and GPT-Realtime-Whisper for streaming transcription.
Summary
Original Article
Meta prepares Hatch AI Agent with waitlist and social skills
Meta is developing "Hatch," a consumer-grade AI agent designed to compete with OpenAI's OpenClaw, integrating image/video generation, shopping, and learning features deeply into Facebook and Instagram, with internal testing targeted for June.
Summary
Decoder
Original Article
Improving token efficiency in GitHub Agentic Workflows
GitHub is actively optimizing token usage in its AI agentic workflows to reduce growing costs, as these automatically scheduled and triggered jobs can accumulate significant expenses out of sight for developers.
Summary
Decoder
Original Article
The Six-Hour Codex Run That Survived a Five-Hour Pause
Codex CLI v0.128.0, released April 30, 2026, introduced a headline feature called `/goal` that allows AI-driven development tasks to persist across terminal restarts and laptop sleeps, automatically resuming work without user re-prompting.
Summary
Deep Dive
Decoder
Original Article
Good QC for RL Data
The quality control (QC) bar for reinforcement learning (RL) data sold to frontier AI labs in 2026 is critically low, with most vendors failing multiple internal QC gates and shipping data that often proves unusable or problematic downstream.
Summary
Deep Dive
Decoder
Original Article
AlphaEvolve: How our Gemini-powered coding agent is scaling impact across fields
Google DeepMind's Gemini-powered coding agent, AlphaEvolve, significantly improved DNA sequencing error correction for PacBio, achieving a 30% reduction in variant detection errors.
Summary
Decoder
Original Article
Meta's Optimized RecSys Inference
Meta has achieved up to a 4x speedup and a 2/3 reduction in latency for its recommendation system inference by implementing In-Kernel Broadcast Optimization (IKBO), which eliminates redundant user embedding replication on both GPUs and Meta's MTIA accelerators.
Summary
Deep Dive
Decoder
Original Article
Featured projects
TL;DR:
- Traditional RecSys inference explicitly replicates shared user embeddings/sequences for every candidate. In-Kernel Broadcast Optimization (IKBO) eliminates this overhead via a kernel-model-system co-design that fuses broadcast logic directly into user-candidate interaction kernels. By decreasing both the memory footprint and IO utilization, IKBO unlocks even higher throughput.
- IKBO delivers up to a 2/3 reduction in compute-intensive net latency, serving as the scalability backbone for the request-centric, inference-efficient framework that powers the Meta Adaptive Ranking Model.
- Deployed end-to-end across Meta’s multi-stage recommendation funnel on both GPU and MTIA (Meta Training and Inference Accelerator).
- The IKBO Linear Compression kernel achieved a cumulative ~4× speedup on H100 SXM5 after four stages of progressive co-design, culminating in warp-specialized fusion via TLX.
- The IKBO co-design shifted the Flash Attention kernel from IO-bound to compute-bound (hitting 621 BF16 TFLOPs on H100 SXM5). Coupled with TLX warp-specialized optimization, this results in a 2.4x/6.4× throughput gain over the non-co-designed CuTeDSL FA4 Hopper baseline (kernel only/kernel + broadcasting).
In this post, we present In-Kernel Broadcast Optimization (IKBO), a kernel-model-system co-design approach that eliminates redundant user-embedding broadcast in recommendation model inference. In production RecSys, user embeddings are identical across all candidates for a given request, yet standard approaches require explicit replication, wasting memory bandwidth and compute that scale with candidate count. IKBO encodes a simple insight: broadcast is a data layout concern, not a computational necessity. Each IKBO kernel accepts user and candidate inputs at their natural, mismatched batch sizes and handles broadcast internally, so no replicated tensors ever materialize. We showcase the methodology through two kernel deep dives: Linear Compression and Flash Attention.
Deployed across Meta’s RecSys inference stack—from early-stage to late-stage ranking models, spanning both GPU and MTIA (Meta Training and Inference Accelerator)—IKBO delivers up to a 2/3 reduction in compute-intensive net latency on co-designed models. It serves as the scalability backbone for the request-centric, inference-efficient framework underlying the Meta Adaptive Ranking Model (serving LLM-scale models in production). On H100 SXM5, our IKBO Linear Compression kernel achieves ~4× speedup through four progressive co-design stages: matmul decomposition, memory alignment, broadcast fusion, and warp-specialized multi-stage fusion via TLX (Triton Low-Level Extensions). For Flash Attention, IKBO delivers a 2.4×/6.4× throughput compared to non-co-designed CuTeDSL FA4-Hopper (kernel only / kernel + broadcasting) with 621 BF16 TFLOPs. Unlike system-level broadcast or net-splitting that work around replication, IKBO eliminates it at the computational primitive layer, achieving dense interaction quality at near-independent cost.
Code Repository: https://github.com/pytorch/FBGEMM/tree/main/fbgemm_gpu/experimental/ikbo
† Work done while at Meta
1. In-Kernel Broadcast Optimization: Eliminating Memory and Compute Redundancy
When a user opens their feed, the recommendation system must score hundreds to thousands of candidate items to decide what to show. The model’s inputs split into two categories: user features (e.g., browsing history, profile, context) that are identical for every candidate in a request, and candidate features (e.g., item ID, category, engagement statistics) that are unique to each item. Both pass through embedding lookups and subsequent processing to produce embedding representations. At various points in the model, interaction layers (e.g., linear projections, feature crosses, target attention) combine user and candidate embeddings. We call embeddings shared across all candidates in a request Request-Only (RO), and per-candidate embeddings Non-Request-Only (NRO).
Fig. 1. A very simplified RecSys inference data flow. Request-Only (RO) user embeddings must be broadcast (replicated) to match the Non-Request-Only (NRO) candidate batch dimension before interaction layers. IKBO eliminates this materialization by handling broadcast internally within each kernel.
Interaction layers require tensors with matching batch dimensions. In a batch of 1,024 candidates served by ~15 users, RO embeddings must be broadcast, replicated ~70 times, to match the NRO batch size before any interaction (Fig. 1). As architectures have evolved from DLRM [1] and DCN [2] through sequential models like HSTU [3] and X’s Phoenix [4], they have steadily enriched user-candidate interaction. But richer interaction comes at a cost: user features must be broadcast across all candidates. For batch sizes of 10 – 10,000+ in inference, this replication overhead incurs significant computation and memory cost that scales linearly with candidate count.
Broadcast is a data layout concern, not a computational necessity. Viewing the model and inference system through this lens opens optimization at every layer: the inference runtime eliminates system-level broadcast, user-only model layers run at the smaller user batch size, and kernels that mix both are redesigned to handle broadcast internally—no replicated tensors ever materialize. Deployed across Meta’s RecSys inference stack, from early-stage to late-stage ranking models, spanning both GPU and MTIA, IKBO delivers up to 2/3 reduction in compute-intensive net latency on co-designed models.
This post focuses on the kernel layer through two deep dives: Linear Compression and Flash Attention.
1.1. Kernel Optimization Type
Type I — Decomposable Operations. Mathematical restructuring lets the Request-Only (RO) portion be computed independently at small batch size, combining with the Non-Request-Only (NRO) portion only at the end. This saves both memory bandwidth and compute.
Type II — Memory-Only Optimization. Handling RO-NRO broadcasting within the kernel avoids redundant data movement, pushing the kernel away from IO bound.
1.2. E2E System Design
Deploying IKBO touches three layers of the infra stack:
- Kernels: Custom GPU kernels that accept mismatched RO/NRO batch sizes and handle broadcast internally (Sections 2 and 3).
- Compilation Specification: The ML compiler needs per-operator dynamic shape ranges to select appropriately shaped kernels. With one batch size this is trivial; with two (user and candidate) or even more, reliably resolving which each operator uses—across production models where interactions obscure batch lineage—requires systematic automation.
- Inference: The runtime passes the candidate-to-user mapping into the model instead of materializing the broadcast.
These kernels enter the model through one of two paths:
- Direct adoption: Model authors integrate IKBO kernels directly into their model definitions. When candidate-to-user ratio > 1 during training, the same kernels reduce training cost as well.
- Inference-time transformation: A pass automatically swaps standard ops for IKBO equivalents at inference time — no model code changes required.
The net effect: broadcast disappears from every stage of inference, with no architectural constraints on the model and no infrastructure changes beyond the inference runtime’s mapping interface.
1.3. Comparison with Other Approaches
Existing approaches work around broadcast rather than eliminating it.
- System-level broadcast materializes the replicated tensor before GPU dispatch—simple but wasteful, with cost scaling linearly with candidate count.
- Net-splitting (ROO) [5] partitions the model into RO and NRO sub-networks, reducing redundant work but constraining where user-candidate interactions can occur and still introduce extra cost at small RO batch sizes.
Both preserve broadcast as a materialized tensor. IKBO eliminates it at the computational primitive layer: savings scale with the candidate-to-user ratio, any interaction pattern works without broadcast cost, and the full NRO batch dimension provides GPU occupancy within fused kernels.
IKBO has been deployed on both GPU and MTIA accelerators. In this blog post, we focus on H100 GPU kernel design to illustrate the core optimization principles.
2. Kernel Deep Dive I: IKBO Linear Compression
Linear Compress Embedding (LCE) compresses input embeddings (B, K, N) via a learned projection (M, K) @ (B, K, N) → (B, M, N), and is widely adopted in Meta RecSys models, e.g., Wukong [6]. We go through four progressive optimization stages.
2.1 Matmul Decomposition
Fig. 2. LCE decomposition: baseline batched matmul (top-left), embedding separation and user deduplication along K (top-right), two independent GEMMs with broadcast-add on compressed output (bottom).
The baseline LCE computes a single batched matmul across all B candidates. The input embeddings concatenate user and candidate parts along K — but user embeddings are identical across all candidates for the same user.
Push broadcast past the matmul. Since W is batch-independent, we decompose by linearity: separate user and candidate embedding blocks along K, deduplicate the repeated user embeddings, and compute two independent GEMMs at their natural batch sizes. Instead of replicating user embeddings before the matmul, we broadcast only the small compressed result. See Fig. 2. With a candidate-to-user ratio of ~70 (a representative setting), the user batch shrinks from B=1024 to B_user ≈ 15 — a 70x reduction in user-side compute. The decomposition is implemented in standard PyTorch.
Result. 1.944 ms → 1.389 ms (28.5% reduction; benchmark setup in Appendix 1). Both the original batched GEMM (arithmetic intensity ~ 356 FLOPs/Byte, below H100’s ~495 FLOPs/Byte machine balance point; see Appendix 2 for derivations) and the two decomposed GEMMs are memory-bound, so the speedup is driven by memory cost reduction. Deduplication cuts memory cost more than half — as the user-side GEMM (B_user ≈ 15 vs. B = 1024) becomes negligible in cost.
Note that the decomposition pushes broadcast past the matmul: instead of replicating full K-dimensional input embeddings before the GEMM, we broadcast only the small compressed result, which is far cheaper. In Section 2.3, we will further eliminate this remaining broadcast entirely via in-kernel broadcast fusion.
The current bottleneck is L1/TEX pipeline utilization (84%) rather than DRAM utilization — a suspicious imbalance we will zoom into in the next section. Detailed profiling breakdown in Appendix 3.
2.2 Memory Layout Optimization
Detailed result analysis of the decomposed GEMM reveals an imbalance: L1/TEX sits at 84% of peak while DRAM reaches only 19%, indicating unnecessarily narrow memory loads. SASS confirms: every cp.async copies only 4 bytes instead of a single 128-bit load.
LDGSTS.E.LTC128B P0, [R203], [R38.64] // 4 bytes
LDGSTS.E.LTC128B P1, [R203+0x4], [R38.64+0x4] // 4 bytes (×4 total, only 16B load in total)
cp.async width is capped by the source pointer’s natural alignment. Matrix A is (M, K) row-major with stride K × 2 bytes, so when K is not a multiple of 8, the stride breaks 128-bit alignment.
Model-kernel co-design insights. Memory alignment is a well-understood GPU optimization — but decomposition turns it into a model-kernel co-design challenge. K is formed by torch.cat of embedding tensors whose sizes depend on many model config factors. Decomposition makes it very hard to manually engineer these factors so that decomposed embeddings remain perfect multiples. A systematic solution is needed.
Solution. Pad each decomposed K to the next multiple of 8 by appending zeros to the concat list. We prove this is mathematically equivalent in both forward and backward passes (see Proof 1 below), and with the ML compiler’s memory planner, reduces to a cheap constant copy.
Proof 1. Zero-padding K preserves exact numerical equivalence in both forward and backward passes.
Result. 1.389 ms → 0.798 ms (42.5% reduction). Padding enables CUTLASS to select a TMA-based kernel, bypassing L1/TEX entirely (sectors 351M → 0) and cutting GEMM latency from 0.984 ms to 0.400 ms. With the GEMM resolved, the unfused broadcast and add (0.398 ms) now accounts for half the total latency — to be addressed in the next section. Detailed result analysis in Appendix 5.
2.3 Candidate GEMM In-Kernel Broadcast Fusion
The unfused broadcast and add are memory-bound: write the candidate GEMM result to HBM, read it back alongside the user result, add, and write again. We eliminate this by fusing the broadcast into the candidate GEMM epilogue (Fig. 3). After each tile’s accumulation, the epilogue looks up the user index, loads the pre-computed user result, adds it in registers, and writes the final sum — the intermediate tensor is never materialized. We implement this as a Triton kernel: a standard batched GEMM with a custom post-accumulation epilogue block.
Fig. 3. In-kernel broadcast fusion: the GEMM epilogue loads the pre-computed user result via index lookup and adds it in-register.
Result. 0.798 ms → 0.580 ms (27.4% reduction). Fusion eliminates 0.87 GB of intermediate DRAM traffic, contributing to the latency win. However, occupancy is just 6.25% (1 warp per scheduler), leaving every stall fully exposed. Beyond 42% of cycles waiting on global loads, 20% are spent waiting on WGMMA — stalls that cannot be hidden by the epilogue, and without persistence there is no next-tile load to overlap with. This is a challenging tradeoff: large tiles and deep pipelines are needed to keep tensor cores fed, but they consume most of the shared memory budget, leaving little room to hide latency through occupancy. Detailed result analysis in Appendix 6.
2.4 Warp-Specialized Multi-Stage Fusion with TLX
TLX (Triton Low-level Language Extensions) exposes Hopper’s warp specialization, TMA, mbarriers, and named barriers while preserving Triton’s Python DSL and autotuning infrastructure.
Using TLX, we address the occupancy limitation from Section 2.3 with warp specialization — hiding latency through functional partitioning rather than additional warps.
Sections 2.1 – 2.3 decomposed the original LCE into two independent computations: the user GEMM (Stage 1) and the candidate GEMM with fused broadcast-add epilogue (Stage 2). We first optimize latency hiding within Stage 2, the dominant bottleneck, then fuse both stages into a single persistent kernel.
Intra-Stage Latency Overlap
The candidate IKBO kernel is memory-bound — the design goal is to keep the memory pipeline continuously fed. Triton’s software pipelining (Section 2.3) already overlaps Loads with WGMMA, but the epilogue remains serialized — it blocks future Loads and exposes the WGMMA wait stalls. We resolve both by partitioning each CTA into specialized warp groups: a dedicated producer issues TMA loads continuously (Overlap #1, analogous to Triton’s software pipeline), while two consumers ping-pong tiles so one’s epilogue overlaps the other’s WGMMA (Overlap #2). With persistence, tiles flow continuously with no cross-tile gaps. See Fig. 4.
Fig. 4. Candidate IKBO kernel structure with two intra-stage latency overlaps and warp group role assignments.
Multi-Stage Fusion
We fuse user IKBO (Stage 1) and candidate IKBO (Stage 2) into a single mega-kernel to reduce wave quantization, eliminate kernel launch overhead, and improve L2 cache utilization. High candidate-to-user ratios amplify wave quantization in Stage 1. Since the candidate GEMM is independent of user results until its epilogue, we schedule both stages concurrently.
This concurrent scheduling unlocks two additional cross-stage overlaps, bringing the total overlaps to four. See Fig. 5.
Fig. 5. Concurrent stage scheduling: SMs without user tiles enter Stage 2 immediately, overlapping with Stage 1’s partial wave. All four latency overlaps after multi-stage fusion, showing intra-stage (#1, #2) and cross-stage (#3, #4) overlap opportunities. SM 0-49, 50-131 are example numbers.
Warp Group Specialization & Synchronization Setup
To realize all four overlaps, each CTA is partitioned into one producer and two consumer warp groups. Critically, both stages share the same circular buffer and mbarrier infrastructure — no pipeline drain or barrier reinitialization occurs at the stage boundary. The last user K-block and the first candidate K-block coexist in different buffer slots simultaneously. See Fig. 6.
Fig. 6. Per-CTA warp group setup and the three synchronization mechanisms.
Bidirectional Stage-Alternating Tile Scheduling
When neither stage’s tile count divides evenly by the SM count, naive unidirectional dispatch causes workload imbalance. We reverse tile assignment direction between stages: Stage 1 starts at pid, Stage 2 at NUM_SM - 1 - pid. See Fig. 7.
Fig. 7. Unidirectional (left) vs. bidirectional stage-alternating dispatch (right), balancing per-SM workload across partial waves.
Tile-Granularity Cross-CTA Synchronization
User and candidate tiles may execute on different CTAs, requiring cross-CTA synchronization — but a device-wide barrier would serialize all work and destroy the overlap. We synchronize at per-tile granularity using a three-step release-acquire protocol:
- A single thread per warp group spins on the tile flag with
ld.relaxed, minimizing memory traffic - Once set, a single
ld.acquireestablishes the happens-before edge - A named barrier broadcasts readiness to all 128 threads in the warp group
This avoids expensive fences during polling and lets candidate CTAs on different user tiles proceed fully independently. Details in Appendix 7.
Results
With all optimizations combined, latency improves from 0.580 ms to 0.482 ms (16.9% reduction). The clear intra-warp Proton tracer timeline confirms all four overlaps are realized in practice.
Fig. 8. Proton profiler timeline for two CTAs, with all four overlaps color-coded. The memory pipeline remains continuously fed.
The primary gain comes from Overlap #2: ping-ponging consumers hide WGMMA and epilogue stalls on every tile — directly addressing the dominant wasted cycles from Section 2.3. Overlap #1 (Load↔WGMMA) carries forward from Triton’s existing software pipelining. Overlaps #3 and #4 hide idle time at the user-to-candidate stage transition. See Fig. 8.
NCU confirms: occupancy rises from 6.25% to 18.75% (3 warp groups vs. 1), DRAM throughput from 39% to 52%, and L2 — the bottleneck — from 74% to 84% of peak. This is not occupancy alone: the aggressive latency hiding across all four overlaps keeps the memory pipeline saturated, which is what pushes L2 past 80%. Detailed NCU metrics in Appendix 8.
We benchmark across batch sizes and candidate-to-user ratios, with the default (batch=1024, ratio=70) settings. See Fig. 9.
Fig. 9. Cumulative IKBO speedup across batch sizes (left, ratio=70) and candidate-to-user ratios (right, batch=1024).
The IKBO fusion delivers robust gains across scenarios: ~4x speedup across batch sizes (left) and candidate-to-user ratios (right). Even at low candidate-to-user ratios, the kernel still achieves meaningful speedup.
3. Kernel Deep Dive II: IKBO Flash Attention
As recommendation models scale to capture richer user sequential behavior, sequential architectures – including attention – have emerged as a critical compute bottleneck, accounting for approximately 40% of inference latency at 1K sequence lengths. This motivates our focus on IKBO-aware Flash Attention, co-designed with RecSys’s unique batching semantics.
Inspired by Transformers and Set Transformers [7, 8], two fundamental user history interaction modules have been widely adopted in RecSys:
- Target attention (analogous to cross-attention) captures the relationship between the prediction candidate and the user’s historical interactions.
- Self-attention models sequential dependencies within the user history itself
Since user history is a RO feature while the target operates on a distinct candidate (non-RO) batch dimension, this architectural asymmetry presents an opportunity for IKBO to improve model scalability and computational efficiency. Target attention will be our main focus for optimization, while with minor co-design, self attention could also be fused into IKBO target attention in Section. 3.3. As our model is encoder-driven, full attention is applied without causal masking.
The ultimate optimized target attention version leveraging e2e co-design achieves 2.4×/6.4× the throughput of non-co-designed CuTeDSL FA4-Hopper (attn kernel only / attn kernel + broadcasting cost), reducing latency by 0.320ms / 1.232ms respectively (Table. 2).
3.1 IKBO flash attention solves the IO bound issues under RecSys boundary conditions
Fig. 10: Traditional SDPA with candidate-user broadcasting (left) vs. fused IKBO target attention (right).
IKBO fuses K/V broadcasting into the attention kernel, maintaining mathematical equivalence via a candidate-user mapping tensor from the inference runtime that handles non-uniform candidate-to-user ratios. Fig. 10 contrasts the two approaches: the traditional SDPA path broadcasts K and V to the full candidate batch size before attention, while the IKBO path eliminates this materialization entirely — each candidate indexes into its user’s K/V on the fly.
Shifting IO-Bound to Compute-Bound by IKBO co-design
In RecSys boundary conditions, target attention uses a relatively small number of candidate embeddings to represent the candidate attributes compared to the user’s browsing history. Roofline analysis of standard attention reveals an arithmetic intensity of ~60 FLOPs/Byte – well below the H100 (SXM5 HBM2e version) peak of ~495 FLOPs/Byte (Appendix 2)—making even standard flash attention heavily IO-bound. IKBO addresses this by amortizing K/V memory accesses across multiple candidates sharing the same user context, improving arithmetic intensity from ~60 FLOPs/Byte to ~833 FLOPs/Byte (at B_candidate : B_user = 70:1) and shifting the kernel firmly into compute-bound territory.
To maximize this benefit, our implementation reorders the threadblock launch grid so that batch_size_candidate comes before num_heads. This ensures threadblocks processing different candidates — but sharing the same user K/V — are scheduled concurrently, improving L2 cache reuse.
| Grid dimension | Flash attention (SDPA) | IKBO target attention |
| x | num_q_seq_block | num_q_seq_block |
| y | num_heads | batch_size_candidate |
| z | batch_size_candidate | num_heads |
Table 1: Launch grid configuration comparison. SDPA prioritizes GQA optimization by placing num_heads in grid.y. IKBO swaps head and candidate dimensions, placing batch_size_candidate in grid.y to enable efficient K/V sharing across candidates.
Table 2 compares our IKBO Triton implementation (FA2 logic + IKBO) against state-of-the-art Flash Attention implementations on Hopper (without IKBO co-design). Throughput and IO are measured on attention only; the broadcasting latency for Key and Value is even larger than the attention cost itself.
| Throughput (TFLOPs/s) | IO (GB/s) | Latency (ms) | |
| Triton IKBO FA2 | 425 | 487 | 0.321 (broadcast fused) |
| TLX FA3 | 245 | 2152 | 0.561 + 0.912 (broadcast K&V) |
| CuTeDSL FA4 Hopper | 250 | 2193 | 0.550 + 0.912 (broadcast K&V) |
| TLX IKBO FA3 persistence generalized | 594 | 681 | 0.230 (broadcast fused) |
Table 2: Attention kernel comparison under RecSys boundary conditions (B_candidate = 2048, B_u = 32, uniform candidate-to-user ratio). Without co-design, even cutting-edge Hopper implementations remain IO-bound.
3.2 Adopting Modern Kernel Techniques (FA3, FA4) with IKBO on TLX
With IKBO shifting the kernel from IO-bound to compute-bound, the natural next step was to adopt the state-of-the-art compute optimizations from Flash Attention 3 (FA3 [10]) and Flash Attention 4 (FA4 [11]) on Hopper – specifically warp specialization and pipelining. However, our boundary conditions on the number of query embeddings (q_seq = 32 or 64) make it difficult to directly adopt FA3’s ping-pong or cooperative warp specialization.
Warp specialization on Hopper requires asynchronous WGMMA instructions, which impose a minimum BLOCK_M ≥ 64. Two consumer warp groups are also necessary to minimize bubbles between them. To satisfy these constraints, we customized the kernel to launch both B_candidate = i and B_candidate = i + 1 within a single threadblock, sharing the same B_user. In the discussion below, we assume all users rank an even number of candidates with q_seq = 64; odd-candidate handling follows afterward.
Performance improvement for IKBO FA3 kernel
Starting from FA3’s recipe — intra-warp pipelining, warpgroup specialization, and ping-pong scheduling — the initial TLX IKBO FA3 kernel performed similarly to the FA2 baseline (Fig. 12, blue vs. red, Appendix 11), with on-par throughput.
To diagnose the bottleneck, we visualized intra-warp pipelining using the Proton tracer with GPU cycles as the latency unit (Fig. 10). Table 3 summarizes the key bottlenecks before and after persistence, measured in GPU cycles via the Proton tracer.
Fig. 11: Proton-based intra-warp profiling of the TLX IKBO FA3 kernel. Representative warps from each warp group are shown: warp 0 (producer), warp 4 (consumer 1), and warp 8 (consumer 2). The softmax_PV_overlap and pure softmax regions are marked separately to identify the tensor core bubbles. (A) Before persistence zoomed in view of B (B) Before persistence with 2 waves (C) After persistence with 2 waves
| Bottlenecks | Before | After | Key change |
| Tensor Core Bubbles (1st QKT per wave, Blue) | ~1,300 cycles (400 cycles from warp scheduler switching) | ~1,300 cycles | Unchanged |
| Tensor Core Bubbles (last PV per wave, Blue) | ~2,000 cycles | ~300 cycles | Async TMA store + reciprocal overlap with last PV |
| Cross-CTA Stalls (Orange) | ~14,000 cycles | Eliminated | Persistence removes CTA re-launch entirely |
| Init Buffers & Barriers (Green) | ~1,600 cycles/wave | ~1,600 cycles (1st wave only) | Persistence shared buffer and barrier amortized across waves |
| Wait 1st Q/K Load (Dark purple) | 2,100~4,000 cycles/wave (length varies depending on HBM bandwidth contention) | ～2,000 cycles (1st wave only) | Cross-wave pipelining; producer prefetches ~3K cycles ahead |
Table 3: Key bottlenecks before and after persistence + optimizations.
Key takeaway: cross-CTA stalls are the dominant bottleneck — not tensor core utilization – at these small query sequence lengths. Persistence is a must for this improvement. After persistence, the profiling results and its latency changes are presented in Fig. 11C and Table. 3.
HBM2e-Specific Optimizations
We further tuned the persistent kernel for the H100 SXM5’s HBM2e bandwidth constraints, trading shared memory capacity for reduced load/store blocking. (Table 4).
| Customized optimization/fix | Benefit |
| Decoupled SMEM buffer of O from Q/V with pipelined TMA async store | Decoupled O from Q/V SMEM sharing enable TMA async stores could overlap with next-wave compute, shortening store blocking time from 1,300 to 400 cycles/wave |
| Separate Q₀ and Q₁ buffers | Reduces per-Q loading time, allowing one consumer group starts earlier— beneficial when wave count greatly exceeds K/V sequence iterations (common in RecSys) |
| Instruction Cache Misses fix | Merges the peeled-out last-iteration code path back into the main loop, eliminating icache thrashing caused by excessive warp-specialized instructions (Appendix 12) |
Table 4: Customized optimizations for the HBM2e H100 SXM5. These still fit within the available SMEM budget under RecSys boundary conditions (Appendix 10).
We also implemented persistent V2, which iterates from the end of the K sequence to the front (matching FA3/FA4-Hopper’s approach) to simplify masking logic. Both persistent variants apply the Table 4 optimizations. As shown in Fig. 12, at low sequence lengths (512–4,096) the TLX FA3 persistent kernel outperforms all other candidates; beyond 8K the two persistent variants converge.
Fig. 12: IKBO implementation throughput vs. sequence length (B_candidate = 2,048; B_candidate : B_user = 64; num_head = 2; d_head = 128). Practical RecSys sequence lengths are under 4K [3]; longer lengths are included for comparison with LLM use cases. The generalized version handles non-even candidates per user with 50% odd-candidates per user probability
Generalizing IKBO FA3 for ranking Arbitrary Candidate Batch Sizes
Our IKBO FA3 kernel co-processes two candidate batches per CTA to meet WGMMA’s BLOCK_M ≥ 64 requirement. When a user has an odd number of candidates, one consumer warpgroup has no pairing partner. We handle this with idling logic (Fig. 13, left; Algorithm 1):
- The idle warpgroup drains K/V buffers via mbarrier signaling to prevent producer deadlock.
- The active warpgroup disables ping-pong synchronization (its partner no longer arrives at the named barriers).
At a ~70 : 1 candidate-to-user ratio, the idle path triggers less than 0.7% of the time with negligible overhead (Fig. 12, IKBO TLX FA3 generalized). This approach generalizes to q_seq_len = 32, where four candidate batches are bundled per CTA using analogous idling and masking logic.
Fig. 13: CTA assignment for generalized target attention (left) and self + target attention fusion (right). Each CTA assigns two consumer warp groups sharing the same user K/V. When the candidate count is odd, the 2nd consumer idles and drains barriers.
Algorithm 1: IKBO Attention Forward Pass with Odd Candidate Handling
3.3 Self + Target Attention Fusion via Model Co-Design
The previous sections focused on optimizing target (cross) attention. A natural question arises: can we fold self-attention into the same kernel?
The key insight is that both attention types share the same key-value source — the user sequence. The only difference is the query: self-attention queries come from the user side, while target-attention queries come from the candidate side. By sharing K/V projections between the two, we enable direct horizontal kernel fusion within a single launch. Fig. 13 (right) illustrates the fused CTA layout: the first CTAs handle self-attention query blocks, while the remaining CTAs handle target-attention candidate pairs — all reading from the same pipelined K/V stream.
Similar co-design ideas have been explored in XAI Phoenix, an open-source recommendation system from X [4].
We prototyped a fused kernel to quantify the fusion benefit, excluding K/V projection savings (Fig. 13, right):
- seq_len = 512: 6.6% improvement (514 vs. 482 TFLOPs/s)
- seq_len = 1,024: 4.1% improvement (581 vs. 558 TFLOPs/s)
- seq_len = 2,048: 0.3% improvement (612 vs. 610 TFLOPs/s) — self-attention saturates the SMs
The gains at short sequences stem from kernel fusion benefits: reduced launch overhead, shared buffer allocation savings, cross-kernel pipelining opportunities, and wave quantization mitigation — the same inefficiencies that megakernel techniques [12] target in LLM inference. In production, the shared K/V projections provide additional savings on linear projection cost, analogous to KV cache reuse.
4. Summary of Benchmarks and Results
We summarize the kernel-level benchmarks presented in this post alongside end-to-end deployment outcomes. All kernel benchmarks below are on H100 SXM5 (see details in Appendix 1).
- Linear Compression (Section 2). Four progressive co-design stages — matmul decomposition, memory alignment, broadcast fusion, and warp-specialized multi-stage fusion via TLX — yield a cumulative ~4× speedup (1.944 ms → 0.482 ms) at representative settings. Gains remain robust across batch sizes and candidate-to-user ratios (Fig. 9).
- Flash Attention (Section 3). IKBO shifts target attention from IO-bound (~60 FLOPs/Byte) to compute-bound (~833 FLOPs/Byte), achieving 2.4×/6.4× the throughput of non-co-designed CuTeDSL FA4-Hopper (kernel only / kernel + broadcasting) with 621 BF16 TFLOPs.
- End-to-end deployment. IKBO has been deployed broadly across Meta’s RecSys inference stack — from early-stage to late-stage ranking models, on both GPU and MTIA accelerators — delivering up to 2/3 reduction in compute-intensive net latency on co-designed models. IKBO has been validated across candidate-to-user broadcast ratios spanning from ~10,000 : 1 down to ~10 : 1, confirming both numerical stability and scalability across workloads.
5. Conclusion and Future Directions
IKBO demonstrates that broadcast — long treated as an unavoidable cost of user-candidate interaction — can be eliminated at the computational primitive layer through kernel-model-system co-design. By encoding broadcast semantics directly into kernels, no replicated tensors ever materialize, and savings scale naturally with the candidate-to-user ratio.
While the kernel implementations presented in this work target NVIDIA Hopper via Triton and TLX, the core idea — replacing materialized broadcasts with index-driven in-kernel lookups — is hardware-vendor independent. Adapting the IKBO kernels to CuTeDSL (for advanced NVIDIA backend support) and completing the AMD CK support are natural next steps.
Beyond the two-level user-candidate hierarchy presented here, some RecSys scenarios involve deeper hierarchies — for example, user → ads vendor → ads item, where each user sees multiple vendors and each vendor offers multiple items. This introduces two nested broadcast relationships with independent, non-uniform ratios. IKBO can handle this elegantly, and applying it to multi-level workloads is a natural direction for further reducing materialization overhead in production RecSys architectures.
Acknowledgements
We are grateful to Hongtao Yu, Yuanwei (Kevin) Fang, Daohang Shi, Yueming Hao, Srivatsan Ramesh and Manman Ren for their strong internal support of the Triton and TLX foundation, the powerful Triton profiling toolings, and for promptly resolving Triton-related issues throughout this work.
Thanks Chris Gottbrath for his insightful feedback, which significantly improved the clarity of this post. We also greatly appreciate his help in facilitating a smooth review process.
Thanks Santanu Kolay, Sandeep Pandey, Matt Steiner, GP Musumeci, Ashwin Kumar, Ian Barber, Aparna Ramani, CQ Tang for leadership support.
References
[1] Naumov, M., et al. “Deep Learning Recommendation Model for Personalization and Recommendation Systems,” arXiv:1906.00091, 2019.
[2] Wang, R., et al. “Deep & Cross Network for Ad Click Predictions,” ADKDD, 2017.
[3] Zhai, J., et al. “Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations,” ICML, 2024.
[4] xAI. “Phoenix: Recommendation System,” GitHub, 2026. https://github.com/xai-org/x-algorithm
[5] Guo, L., et al. “Request-Only Optimization for Recommendation Systems,” arXiv:2508.05640, 2025.
[6] Zhang, B., et al. “Wukong: Towards a Scaling Law for Large-Scale Recommendation,” ICML, 2024.
[7] Vaswani, A., et al. “Attention Is All You Need,” NeurIPS, 2017.
[8] Lee, J., et al. “Set Transformer: A Framework for Attention-based Permutation-Invariant Input,” ICML, 2019.
[9] Dao, T. “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,” ICLR, 2024.
[10] Shah, J., et al. “FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision,” NeurIPS, 2024.
[11] Zadouri, T., et al. “FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling,” arXiv:2603.05451, 2026.
[12] Spector, B., et al. “Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B,” Hazy Research Blog, 2025. https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles
Appendix
Appendix 1. Benchmark Setup
All experiments are conducted on a single NVIDIA H100 SXM5 GPU (700 W TDP, 96 GB HBM2e) with the following software stack:
- CUDA: 12.4
- PyTorch: 2.11.0a0+fb (internal build)
- Triton: facebookexperimental/triton@
4059e79bf(#831)
Appendix 2. Arithmetic Intensity Analysis
2.1 Machine Balance Point of H100 SXM5 (700 W TDP, 96 GB HBM2E)
2.2 Arithmetic Intensity of the Baseline LCE
For a batched matmul (M, K) @ (B, K, N) → (B, M, N) in FP16, with B=1024, M=433, K=2044, N=256:
Appendix 3. Detailed Result Analysis for Section 2.1
Setup: H100 SXM5 (Appendix 1), PyTorch eager mode (no kernel fusion), inference. Shapes from a representative configuration.
| Version | Total
(ms) |
Kernels | Latency
(ms) |
DRAM
(GB) |
L1/TEX Sectors
(M) |
Compute
(GFLOPs)* |
Bottleneck
† |
| Baseline | 1.944 | 1 CUTLASS GEMM | 1.944 | 1.31 | 798 | 460 | L1/TEX (89%) |
| Decomposition | 1.389 | 2 CUTLASS GEMM (user + candidate matmul) | 0.984 | 0.68 | 351 | 200 | L1/TEX (84%) |
| 1 ATen Gather + 1 ATen add | 0.405 | 0.87 | 36 | 0.11 | DRAM (92%) |
*Total FLOPs executed, not throughput.
†Bottleneck identified via NCU Speed of Light analysis; methodology in Appendix 4.
Deduplication eliminates >98% of user-side work (batch 1024 → ~15), cutting L1/TEX sectors from 798M to 351M and GEMM latency from 1.944 ms to 0.984 ms. The post-GEMM broadcast and addition costs 0.405 ms (DRAM-bound), yielding a net saving of 0.555 ms.
Precision note. The baseline accumulates all K products in a single FP32/TF32 reduction. Decomposition accumulates K_user and K_cand separately, then sums the partial results in BF16/FP16. Training uses the same decomposition, so numerics match end-to-end. For exact inference parity, a fused kernel (Section 2.4) can perform the final summation in FP32.
Appendix 4. Bottleneck Analysis Methodology
For a closer look after roofline analysis, we use NCU’s Speed of Light analysis to identify hardware subsystem bottlenecks. The bottleneck is the subsystem with the highest utilization relative to its peak sustained throughput. For the analysis in Section 2.1, we monitor three metrics:
Compute is the peak SM pipeline utilization, reported directly by NCU (Compute (SM) Throughput). It measures how busy the most active execution pipeline (tensor cores for GEMMs) is relative to its peak instruction rate.
L1/TEX utilization is derived from the total sectors the L1/TEX unit must process as below, where num_L1_tex_sectors is l1tex__t_sectors_pipe_lsu_mem_global_op_ld.sum and _st.sum counter, is SM_active_cycles sm__cycles_active.avg counter, num_SM is 132 and num_sustained_peak_sectors_per_sm_per_cycle is 2.0 on H100.
DRAM utilization is derived from total HBM bytes transferred as below, where dram_bytes_read_and_write is the dram__bytes_read.sum and dram__bytes_write.sum counter. peak_bandwidth is 2TB/s on the testing GPU server.
Appendix 5. Detailed Result Analysis for Section 2.2
Result. 1.389 ms → 0.798 ms (42.5% reduction).
| Version | Total Latency
(ms) |
Kernels | Latency
(ms) |
DRAM Traffic
(GB) |
Compute
(GFLOPs) *not speed |
L1/TEX Sectors
(M) |
Bottleneck
† |
| Decomposition
(unpadded) |
1.386 | 2 CUTLASS GEMM – user & candidate matmul | 0.984 | 0.68 | 200 | 351 | L1/TEX (84%) |
| 1 ATen Gather – broadcast
1 ATen Elementwise – add |
0.402 | 0.87 | 0.11 | 36 | DRAM (92%) | ||
| Decomposition
(padded K) |
0.798 | 2 CUTLASS GEMM – user & candidate matmul | 0.400 | 0.69 | 200 | 0 | Balanced |
| 1 ATen Gather – broadcast
1 ATen Elementwise – add |
0.398 | 0.87 | 0.11 | 36 | DRAM (92%) |
Two factors behind the large speedup.
- TMA. With aligned matrices, CUTLASS selects a TMA-based kernel, bypassing L1/TEX entirely (sectors → 0). The unpadded kernel also penalized matrix
Bunnecessarily: it applied 4-byte loads to both matrices, even thoughB(with alignedN) could have used 128-bit loads. - Bank conflicts. The unpadded kernel also uses sm80 MMA path whose swizzle pattern doesn’t protect against 4-byte cp.async writes, causing many shared memory bank conflicts. The padded kernel doesn’t have this issue.
Appendix 6. Detailed Result Analysis for Section 2.3
Result. Latency: 0.798 ms → 0.580 ms (27.4% reduction).
| Version | Total Latency
(ms) |
Kernels | Latency
(ms) |
DRAM Traffic
(GB) |
| Decomposition
(padded K) |
0.798 | 2 CUTLASS GEMM – user & candidate matmul | 0.400 | 0.68 |
| 1 ATen Gather – broadcast
1 ATen Elementwise – add |
0.398 | 0.87 | ||
| iKBO Fusion | 0.580 | user GEMM & candidate iKBO kernel | 0.580 | 0.68 |
The 0.87 GB of intermediate DRAM traffic is eliminated as expected. NCU profiling reveals further opportunity: occupancy is just 6.25% with 1 warp per scheduler, and PC sampling shows only 23% of cycles are productive:
| Stall Reason | Percentage | What it mainly refers in the kernel |
| Stall long scoreboard | 41.8% | Global memory loads |
| Selected (executing) | 23.1% | Productive work (good) – instructions actually issued |
| Stall wait | 20.1% | Wait WGMMA |
| Stall barrier | 5.7% | bar.sync between software-pipeline stages |
With 1 warp per scheduler, every stall is fully exposed: there is no other warp to switch to. Increasing occupancy by reducing pipeline depth would sacrifice K-loop latency hiding. This is a challenging situation for this kernel:
Building Fast & Accurate Agents with Prime-RL Post Training
Ramp Sheets leveraged reinforcement learning to build "Fast Ask," a specialized agent for quickly navigating spreadsheets and retrieving specific information, detailing the process as a case study for training specialized agents.
Summary
Original Article
ds4.c (GitHub Repo)
Antirez, the creator of Redis, has released `ds4.c`, a Metal-only (with future CUDA plans) native inference engine specifically optimized for DeepSeek V4 Flash, aiming to provide a "finished" end-to-end local inference experience on high-end personal machines like MacBooks with 128GB RAM.
Summary
Deep Dive
Decoder
Original Article
Natural Language Autoencoders
Anthropic has introduced Natural Language Autoencoders (NLAs), a new interpretability method that translates AI model activations into human-readable text, allowing researchers to uncover hidden motivations and safety concerns in models like Claude Opus 4.6 and Mythos Preview.
Summary
Deep Dive
Decoder
Original Article
Long AI Short AGI
The idea of Artificial General Intelligence (AGI) as a perpetually scarce resource is being rapidly challenged by the commoditization of AI models, which are increasingly following the same market trajectory as other fundamental tech resources like compute and bandwidth.
Summary
Original Article
Notes from inside China's AI labs
Chinese AI labs, unlike their American counterparts, foster an ecosystem of collaboration and humility, prioritizing meticulous model improvement over individual recognition or business monetization.
Summary
Original Article
Google DeepMind partners with EVE Online for AI model testing
Google DeepMind acquired a minority stake in EVE Online's developer, Fenris Creations, to use the complex sci-fi MMO as a unique testbed for AI systems requiring long-horizon planning and continual learning, following Fenris's $120 million buy-out from Pearl Abyss.
Summary
Original Article
Perplexity Brings Personal Computer to Mac
Perplexity made its "Personal Computer" AI agent available to all Mac users, allowing it to interact with local files, applications, and web resources directly through its desktop app.
Summary
Original Article
Trusted Contact for ChatGPT
OpenAI introduced "Trusted Contact" for ChatGPT, an optional feature that allows adult users to designate a contact who will be alerted if the AI detects severe self-harm risk in their conversations.
Summary
Original Article
Apple's Camera-Equipped AirPods Reach Late Testing in AI Device Push
Apple is reportedly in the late stages of developing new AirPods with integrated cameras, marking its first dedicated AI hardware, but the launch could be delayed by concerns over the quality of its visual AI capabilities.
Summary
Original Article
Google unveils screenless Fitbit Air and Google Health app to replace Fitbit
Google is re-entering the screenless wearable market with the $99.99 Fitbit Air, launching May 26, 2026, which funnels health data into a new Google Health app featuring an AI-powered coach built on Gemini to interpret user metrics.
Summary
Decoder
Original Article
Wearables have really come full circle. The early Fitbits didn’t have screens, but the move to smartwatches put a screen on everyone’s wrist. Now, devices like Whoop and Hume are designed as data trackers first and foremost without so much as a clock. Google’s newest wearable jumps on that trend: The Fitbit Air doesn’t have a screen, but it does have a suite of health sensors that pipe data into the new Google Health app. And if you want, Google has a new AI-powered health coach in the app ready to tell you what that data means (maybe).
The Fitbit Air itself is a small plastic puck about 1.4 inches long and 0.7 inches wide. It slots into various bands that hold the bottom-mounted sensors against your wrist. There’s no display pointing upward, so the entire device is covered by the fabric or plastic of the band. It’s a streamlined and potentially stylish look—in uncharacteristic fashion, Google has plenty of colors and style options available, including a special-edition Steph Curry version. You may have heard chatter about Curry being seen teasing a new screenless Fitbit, and this is it.
Smartwatches never quite became a must-have device—plenty of people have them, but we don’t all wear them all the time because they need to be charged often and aren’t always very comfortable. The screenless Fitbit Air doesn’t have those issues. Google says it lasts about a week on a charge, and it does that while collecting continuous health data. It can even store a day of data without being connected to your phone.
While the Pixel Watch is very comfortable for a smartwatch, Google still wants to make it easier for people to keep collecting data all day and night. The company says that product testers rated the Air as more comfortable than competing devices, so you may actually be willing to wear it to bed for sleep tracking. You don’t have to choose between these devices, either. You can keep a Pixel Watch and Fitbit Air paired with your phone and wear whichever one you want over time. This capability will come to more wearable devices in the near future, too.
The Fitbit Air will have all the standard wearable health sensors: heart rate, accelerometer/gyroscope, infrared SpO2, and skin temperature. Google notes that the heart rate monitor isn’t as advanced as the one in the latest Pixel Watches, so the Air might not be as accurate during vigorous activity. The Air also has a vibration motor that can be used for alarms, but it’s not going to buzz for phone notifications like a smartwatch.
The Fitbit Air launches on May 26 for $99.99 with the included Performance Loop band. There are also silicone Performance Loop and Elevated Modern Band options. Bands start at $34.99 and come in various colors. A Fitbit Air purchase also includes three months of Google Health Premium (replacing Fitbit Premium), which now features Google’s new AI Health Coach.
Goodbye, Fitbit… Hello, Google Health
The Fitbit app is getting a major makeover and a new name. An update in the coming weeks will transform that app into Google Health, featuring a new interface with a more extensive Material Expressive aesthetic and redesigned menus and tabs. You also won’t see Fitbit branding in as many places—the Fitbit Premium subscription will become Google Health Premium.
Without a subscription, the app still does all the basic things, like tracking your health stats, automatically logging workouts, and showing it all in a pretty dashboard. With the Premium subscription, you get all the features from Fitbit Premium plus the new AI Health Coach. It’s a chatbot, so you can ask it about any health or wellness topics, and the answers are grounded in your health data.
Google suggests asking the Health Coach for customized workout routines or exploring health concerns. The robot can theoretically use your accumulated health metrics, like workouts, nutrition, and sleep, to provide better suggestions. You can even upload a picture of food to Health Coach and have it automatically logged in the app.
This Health Coach AI was built on Gemini, but it has been tuned differently from the normal frontier model. According to Google, it used a panel of health experts and extensive user studies to validate the Health Coach model. Curry and his “performance team” also had input on how the Health Coach responds.
We won’t know how useful the coach is until it begins rolling out later this month, but the idea is that it will be more useful the more data is piped in from your wearable. Naturally, health data is extremely sensitive, and Google is asking you to dump a lot of it into a cloud-based AI model. Google says it will never use this data for advertising, which has been the case in all its previous health endeavors. In the AI era, it has further stipulated that it won’t use your health data for AI training unless you choose to do that. There will be an opt-in toggle in the settings to contribute data for training, but it’s unclear why anyone would do that.
Like the retired Fitbit Premium, the new Google Health Premium will be available for $10 per month or $100 per year. It’s also included if you’re already paying for AI Pro or AI Ultra. If you choose to skip the subscription, you can continue to use your Fitbit and Google wearables in the new app with the same basic stat-tracking features. And what of Fit, that other Google-branded health tracking app? Fit will shut down later this year, at which time users will have to migrate their data to Google Health.
What the hell is happening in China?
China's biotech industry, characterized by early-stage biotechs with over 10 development candidates and cheaper drug development, is poised to surpass Western rivals within years, driven by a "breadth-first" strategy and less concern for IP protection given global patent visibility.
Summary
Deep Dive
Decoder
Original Article
The AI Revival of the Three Mile Island Nuclear Plant
The increasing energy demand from US AI infrastructure buildout is pushing a reliance on older nuclear technology, like the potential restart of Three Mile Island, because advanced, safer nuclear reactor designs are still years away from contributing meaningfully to the energy supply.
Summary
Original Article
Behind the Scenes Hardening Firefox with Claude Mythos Preview
Mozilla significantly improved Firefox's security by using Claude Mythos Preview and other AI models to discover and fix an unprecedented number of latent security bugs, many of which would typically require combining with other exploits for a full attack.
Summary
Decoder
Original Article
OpenAI launches new realtime voice and translation AI models
OpenAI has launched three new real-time audio models via its API, including GPT-Realtime-2 for GPT-5-class reasoning in voice agents, GPT-Realtime-Translate for live multilingual conversations in over 70 languages, and GPT-Realtime-Whisper for streaming speech-to-text.
Summary
Original Article
Elon Musk tried to hire OpenAI founders to start AI unit inside Tesla
OpenAI claims Elon Musk attempted to hire its founding team, including Sam Altman, in 2018 to lead a Tesla AI unit, contradicting his lawsuit that accuses Altman of "stealing a charity" by commercializing OpenAI.
Summary
Original Article
Cloudflare to Slash 1,100 Jobs Due to AI-Driven Restructuring Plan
Cloudflare announced plans to cut 1,100 jobs as part of a restructuring to adopt an "agentic AI-first operating model," anticipating $140 million to $150 million in related charges.
Summary
Decoder
Original Article
The agent principal-agent problem
AI agents are exacerbating the "principal-agent problem" in code review by enabling "slop PRs" and increasing review load, making the traditional review-then-commit process unmanageable in low-trust large company environments.
Summary
Decoder
Original Article
Markets in everything?
The author, a proponent of markets, expresses concern over the "ever-increasing overt marketization of society" where everything, including personal sentiment, is being assigned a price, potentially leading to widespread dissatisfaction.
Summary
Decoder
Original Article
AI load breaks GitHub – why not other vendors?
GitHub's recent spate of data integrity incidents and outages, including a 85% uptime, is attributed by CTO Vlad Fedorov to an unexpected surge in AI agent-fueled load, which GitHub, unlike competitors, seemingly failed to adequately anticipate.
Summary
Deep Dive
Decoder
Original Article
Tokenmaxxing, Promomaxxing, and Misaligned Incentives in Tech
The pursuit of "tokenmaxxing" and "promomaxxing" in tech, driven by metrics that become targets, can lead to perverse incentives where engineers generate high output without corresponding positive outcomes, as exemplified by Meta engineers burning millions of AI tokens for no productivity.
Summary
Decoder
Original Article
Tokenmaxxing, Promomaxxing, and Misaligned Incentives in Tech
When a measure becomes a target, it ceases to be a good measure
Engineer’s Codex is a publication about real-world software engineering.
Coinbase did layoffs recently. They’re cutting a large percent of their employees, and along with the announcement, they mentioned wanting their employees to use AI more. Basically, they want people to tokenmaxx.
If you don’t know what tokenmaxxing is, its the idea of maximizing the amount of tokens you use when working with AI. Basically, use AI a lot! And it’s generally seen as a good thing. The more tokens you use should, in theory, mean the more productive you are.
This isn’t always true.
When a measure becomes a target, it ceases to be a good measure
Meta actually created an internal leaderboard that counted the amount of tokens people were consuming. You would think that the people consuming the most tokens are generally the most productive.
But here’s the problem: anything that is measured can and will be gamed (Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.”), especially by smart, pragmatic people, like Meta engineers. Smart engineers optimize for the best personal outcomes, which usually means a promotion, more money, more scope. If tokenmaxxing is the path to get there, they will do that (more visibility, higher number is good).
So people started setting up scripts to burn millions of tokens for literally zero productivity. Just burning tokens to do nothing. Meta eventually shut the leaderboard down because it created the wrong incentive.
Promomaxxing
Tokenmaxxing actually reminds me of a famous complaint about Google: specifically how hard it is to get promoted there and how that led to a lot of misaligned incentives. You could call it promomaxxing.
Googlers would make things more complex than needed, write way more docs than needed, and make those docs much longer than needed, all to manufacture the appearance of hard, complex work. Because at Google, if your project wasn’t technically complex enough or there weren’t enough of them, you weren’t getting promoted.
In theory, this makes sense. People who get promoted should be doing harder and harder things.
However, good software engineering should lean toward simplicity. But promotion rewards complexity. And as frameworks and developer infrastructure keep making engineering simpler (which is genuinely good), engineers run out of adequate complexity to justify their promotions. So you get these irrational decisions for the business that are completely rational decisions for the individual. That’s a textbook misaligned incentive.
The Cobra Effect of Perverse Incentives
The most famous example I can think of in history of perverse incentives is the Cobra Effect in India.
During British rule in India, the government was concerned about the number of venomous cobras. They offered a bounty for every dead cobra brought to them.
Enterprising citizens began breeding cobras specifically to kill them and collect the reward.
When the government realized this and scrapped the bounty, the breeders released their now-worthless snakes, leaving the cobra population higher than when the program started.
The Input → Output → Outcome Discrepancy
Tokenmaxxing has the same problem. Good intentions, wrong incentive structure.
Tokenmaxxing is built on the idea of:
input → output → outcome.
More input should produce more output, which should drive better outcomes. This framing actually comes from this article by Arnav Gupta on Twitter, and I think he expressed it well.
The issue is that input to output always has some loss. You put in 100% input and you might get anywhere from 50% to 150% output, because sometimes that input is thinking time, debugging, exploration. It’s not a clean conversion.
Outcome is even further removed. Output doesn’t necessarily even correlate to outcome. In fact, output can even lead to negative outcomes.
Just because you shipped a feature doesn’t mean you moved a metric positively. If your goal is to increase retention and you built a notifications feature, the outcome you want is higher retention. But more notifications doesn’t guarantee that. If users already have notifications and you add more, you might actually annoy them and hurt retention. There is no guarantee that more output even produces a positive outcome.
There are even worse examples. Say you, as a Google engineer, spend a ton of input to ship a system and that system has bugs that take down Google’s ad platform for two hours. Google just lost $5 million. Your output was a net negative. Was tokenmaxxing worth it in this case? Tokenmaxxing does also result in lower quality output, on average. Can we really guarantee better outcomes from more rapid input?
This is where input quality does matter. And generally, tokenmaxxing degrades input quality.
Slow coding was a feature, not a bug. It required clearer thinking and higher-quality inputs.
.When a CEO or PM had 10 ideas and the team could only do 2, you were forced to debate. You had to fight for your idea, kill the weak ones early, and actually pressure test what was worth building. The constraint created a filter. Now that code is basically free and fast, that filter is gone. Yes, an MVP is valuable data.
Another note is that tokenmaxxing does not just 5x your output. It can also 5x’d your noise. More features, more bugs, more teams building overlapping things in different ways, more meetings to align on stuff that should have been killed in a Slack thread. The alignment tax went up at the exact same time the coding cost went down.
These are really recent, really clear examples of misaligned incentives. And misaligned incentives are hard problems because they’re people problems. You’re trying to optimize for multiple things at once, and it’s leadership’s job to lay the dominoes in the right way.
Some startups like Anthropic have naturally aligned incentives. If an engineer tokenmaxxes using Claude, Anthropic just gets paid more. They don’t really care about your output or your outcome. They care about their own outcome, which just happens to be directly correlated to your input. To you, they sell the potential outcome and you purchase the guaranteed tokens.
SWE Quiz (Featured)
SWE Quiz is a structured crash course of everything you need to know for system design and modern AI engineering interviews. It contains thousands of questions that have been asked in interviews at DeepMind, OpenAI, Anthropic, and more.
Where Tokenmaxxing Excels
Now, I’m not against using AI at all. I use AI extensively. In fact, I would say probably 95% of the code generated by me is AI generated. So this is actually not an argument against using AI at all. It’s actually more of an argument towards misaligned incentives and understanding where token maxing as a behavior may have good intentions, but a wrong implementation. At the end of the day, we just do want better outcomes. And understanding how to token max in the right ways is important for better outcomes.
A great example from a friend at work that I heard recently was they had to run a bunch of experiments on reducing some latency for their system. Previously they would have to guess at these experiments and pick the top three. But with AI, he was actually able to add flags and set up experiments to test all seven of his ideas. And it turns out probably one of the experiments that he would not have tried earlier actually ended up being one of the better latency reduction wins.
Tokenmaxxing is great for cases like these, for rapid exploration and throwaway work in favor of an outcome, where the goal is knowledge. In this case, the outcome is guaranteed.
There is a cost to that outcome, which previously was human time, but now you have less human time needed, but token costs added on top. So the calculus has changed, and will continue changing over time.
My Takeaways
All this is to say that incentive alignment is really hard. It’s a constant struggle at all levels of leadership, but I have found that the best leaders and companies I’ve worked at excelled at aligning incentives as much as possible.
For example, promotions across the board in tech are focused on outcomes and seem to be more focused on outcomes nowadays. This may be due to the turmoil of tech too - outcomes are harder to achieve, and thus allows companies to promote less.
Tech is full of missionaries and mercenaries. Generally, there are more mercenaries than missionaries. Mercenaries will always optimize for the path to promotion and money. If that path is misaligned with the company’s health, the fault lies with the incentive structure, not the employees.
It’s worth framing things in your work with incentive alignment. I’ve found it an useful exercise for lining up things for promotions, getting cross-functional collaboration done, and, in general, getting “champions” across orgs for both me and my work.
Matt Mullenweg Assembles Trusted Group to Overhaul WordPress.org and Five for the Future
Matt Mullenweg, co-founder of WordPress, has granted a select group of trusted contributors direct authority to redesign WordPress.org and the "Five for the Future" program, bypassing traditional team and committee approvals.
Summary
Decoder
Original Article
Introducing HCP Terraform powered by Infragraph - now in public preview
HashiCorp has made HCP Terraform powered by Infragraph available in public preview, introducing an event-driven knowledge graph that unifies infrastructure data across hybrid and multi-cloud environments, paving the way for AI-driven automation.
Summary
Decoder
Original Article
Introducing the Datadog Code Security MCP
Datadog launched Code Security MCP, a new service that scans AI-generated code in real time for vulnerabilities, secrets, and risky dependencies directly within a developer's local workflow.
Summary
Decoder
Original Article
The AWS MCP Server is now generally available
AWS has made its MCP Server generally available, offering AI coding agents secure and authenticated access to over 15,000 AWS API operations and current documentation, solving the problem of AI agents relying on outdated training data and generating non-production-ready infrastructure.
Summary
Decoder
Original Article
The AWS MCP Server is now generally available
I have been building with AI agents and MCP tools for a while now, and one question kept coming up: how do you give an agent real, authenticated access to AWS without handing it the keys to the kingdom? Today, there is an answer.
I’m happy to announce the general availability of the AWS MCP Server, a managed remote Model Context Protocol (MCP) server that gives AI agents and coding assistants secure, authenticated access to all AWS services through a small, fixed set of tools.
The AWS MCP Server is part of the Agent Toolkit for AWS, a suite of tooling that includes the MCP Server, skills, and plugins that help coding agents build more effectively and efficiently on AWS.
AI coding agents are already useful for many tasks, but they run into real trouble when working with AWS at any meaningful depth. Without access to current AWS documentation, agents rely on training data that may be months out of date and may not know about services like Amazon S3 Vectors, Amazon Aurora DSQL, or Amazon Bedrock AgentCore. When asked to build infrastructure, they tend to reach for the AWS Command Line Interface (AWS CLI) rather than AWS Cloud Development Kit (AWS CDK) or AWS CloudFormation, and they produce AWS Identity and Access Management (IAM) policies that are far broader than necessary. The result is infrastructure that works in a demo but is not production-ready.
The AWS MCP Server addresses this through a compact set of tools that do not consume your model’s context window. The call_aws tool executes any of the 15,000+ AWS API operations using your existing IAM credentials. When we will launch new APIs, they will be supported within days. The search_documentation and read_documentation tools retrieve current AWS documentation and best practices at query time, so the agent always works from up-to-date information.
With general availability, we are introducing several new capabilities. The AWS MCP Server now supports IAM context keys, so you no longer need a separate IAM permission to use the server and can express fine-grained access in a standard IAM policy. Documentation retrieval no longer requires authentication. We have also reduced the number of tokens required per interaction, which matters for complex, multi-step workflows.
Also new, the run_script tool lets the agent write a short Python script that runs server-side in a sandboxed environment. The sandbox inherits your IAM permissions but has no network access, so you can give an agent the ability to process data without giving it access to your local file system or a shell. When an agent needs to call multiple APIs and combine the results, making them one at a time is slow and burns context. With run_script, the agent chains API calls, filters responses, and computes results in a single round-trip, which is both faster and more context-efficient.
The most significant addition is the transition from Agent SOPs to Skills. Skills provide curated guidance and best practices for the tasks where agents most commonly make mistakes. This helps agents complete work faster, using validated best practices, with fewer errors and fewer tokens — all of which saves you time and money. Skills are contributed and maintained by AWS service teams. This keeps the tool list short and predictable, which reduces hallucination and keeps the agent focused.
For enterprise customers, the AWS MCP Server provides a clear separation between human and agent permissions. You can use IAM policies or Service Control Policies to specify that a given user can perform mutating operations while the MCP server is restricted to read-only actions. Amazon CloudWatch metrics published under the AWS-MCP namespace let you observe MCP server calls separately from direct human calls, giving you the audit trail that compliance teams require. Amazon CloudTrail captures all API calls for a complete record.
Let’s see it in action For this demo, I chose to use Claude Code, but I can use the AWS MCP Server with any AI agent that supports MCP, which is basically all tools available today: Kiro CLI, Kiro, Cursor, Codex, and more. I configure Claude Code to use the Anthropic Opus 4.6 model.
Opus 4.6 has a knowledge cutoff date in May 2025. It means it doesn’t know anything that happened after May last year. I ask a question about an AWS service that was introduced recently: Amazon S3 Vectors, launched in preview in July 2025 and that went GA in December 2025.
The question is “how to store embedding on S3″. (embedding is a kind of vector)
It gives me five solutions, all correct, but none using S3 Vectors as I asked. Note that this answer comes from the Opus 4.6 model, not from Claude Code. Any AI tool using the same model will return similar answers because S3 Vectors wasn’t announced at the time the model was trained.
Let’s now try with the AWS MCP Server.
The AWS MCP Server uses AWS Identity and Access Management (IAM) and IAM SigV4 authentication. To use my local AWS credentials configuration over MCP, which only supports OAuth 2.1, I configure my AI coding agent to call the AWS MCP Server through a proxy. The MCP Proxy for AWS is an open source proxy that runs on my machine and bridges the world of IAM authentication to OAuth.
I add the MCP configuration with this command:
claude mcp add-json aws-mcp --scope user \
'{"command":"uvx","args":["mcp-proxy-for-aws@latest","https://aws-mcp.us-east-1.api.aws/mcp","--metadata","AWS_REGION=us-west-2"]}'
You’ll have to have uv installed before you can use the AWS MCP server. On Linux or Mac, you can run: curl -LsSf https://astral.sh/uv/install.sh | sh
Let’s analyze the JSON configuration:
- I use the user scope to make the server available to all my projects on my laptop.
uvx mcp-proxy-for-awsis the command to launch the proxy; the rest of the arguments are parameters passed to the proxy.https://aws-mcp.us-east-1.api.aws/mcpis one of the two regional endpoints for the AWS MCP Server. The proxy will forward Claude Code’s requests to that endpoint.--metadataare passed to the proxy target. Here, it tells the AWS MCP Server to use the US West (Oregon) Region.
I start Claude Code and I type /mcp to verify the AWS MCP Server is correctly installed and can use my credentials.
I ask the same question: “how can I store embedding on S3”.
This time, Claude Code knows it has a tool it can use to answer the question. It asks me permission to invoke the aws___search_documentation tool. After a few seconds, I receive a correct answer: “AWS now has a dedicated service for this: Amazon S3 Vectors …”
Pricing and availability The AWS MCP Server is available today in the US East (N. Virginia) and Europe (Frankfurt) AWS Regions and can make API calls to any Region. There is no additional charge for the AWS MCP server itself. You pay only for the AWS resources you create and any applicable data transfer costs.
The AWS MCP Server works with Claude Code, Kiro, Cursor, and any MCP-compatible client. To get started, see the AWS MCP Server User Guide.
I have been waiting for something like this since I started using MCP tools in my AI agents early last year. The combination of current documentation, authenticated API access, and sandboxed script execution in a single server changes what an agent can actually do on AWS. I am curious what you build with it. Let me know in the comments.
— sebUpdated on May 6th – Added uv installation script.
How we built a real-world evaluation platform for autonomous SRE agents at scale
Datadog engineered a replayable evaluation platform for its Bits AI SRE agent, utilizing production-derived labels and noisy simulated environments to continuously measure and enhance the agent's performance in investigating complex production incidents.
Summary
Decoder
Original Article
We shipped a feature that made perfect sense. It improved a specific type of investigation we had been testing against. Then other investigations started getting worse.
Nothing crashed. No tests failed. But the overall quality of the agent had shifted, and we had no reliable way to detect it.
Bits AI SRE is Datadog's autonomous agent for investigating production incidents. It reasons across metrics, logs, traces, infrastructure metadata, network telemetry, monitor configuration, and more to determine, triage, and remediate the root cause of an issue.
As we built Bits, we expected behavior to improve incrementally with each feature we added. Instead, we saw something more subtle. Improvements in one area could quietly introduce regressions in another. The problem wasn't just the model. We had no way to replay real production context, measure behavior consistently across diverse incidents, or track whether the agent was actually improving over time.
We needed infrastructure that could turn production issues into reproducible investigation environments. So we built a replayable evaluation platform from scratch.
In this post, we'll walk through how the Bits AI SRE team built that platform and what it took to make agent behavior observable, measurable, and repeatable.
When one improvement caused subtle regressions
Early in development, before exposing the system to customers, we added a feature that extracted the service name from the monitor under investigation into Bits AI SRE's initial context. On the surface, this made sense, and in a handful of internal test cases, it worked as expected.
What we could not see was the broader impact. Without a representative evaluation set, we had no way to measure how that change behaved across different environments. The feature pulled in a large amount of irrelevant signals, which degraded investigation quality in unrelated scenarios, often by subtly confusing the reasoning of the agent. This change introduced regressions that didn't become apparent until we began seeing widespread investigation misses internally.
This wasn't an isolated case. Features that improved Bits in one area could quietly degrade performance in another, and the relationships often weren't obvious. We had no standardized way to catch these regressions, no way to track quality across changes, and no confidence that the next feature wouldn't cause the same problem. We needed a way to catch these regressions before users reported them.
Why tool-level testing and live replay weren't enough
Beyond standard test suites, we first tried testing individual tools in isolation. This approach seemed reasonable. If each tool behaved correctly, the agent should behave correctly.
In practice, that assumption broke down. Bits' value comes from how it chains tools together and reasons across their outputs. Failures often emerged from interactions between steps, not from a single tool call. For example, the agent might retrieve valid signals from multiple tools but combine them incorrectly, leading it to attribute an issue to the wrong component.
We also experimented with rerunning live Bits investigations as a form of online evaluation. That did not scale. Results were not aggregated, environments changed underneath us, and investigations could not be replayed once the underlying signals expired.
We needed an offline system that could replay realistic scenarios across Datadog's signals and measure the agent's behavior in a controlled, repeatable way. Off-the-shelf eval frameworks assume clean inputs and static test sets, which breaks down when your agent reasons across live production telemetry.
We ended up building two components that work in tandem: a curated label set that defines representative investigations and an orchestration platform that executes and scores the agent against them.
Anatomy of a label
Each evaluation label represents a single investigation scenario Bits would encounter in production. The label has two parts. The first is the ground truth, which defines the root cause of the issue. The second is the world-snapshot, which captures the signals that were available at the time the issue occurred. For example, a label might define the root cause as a Kubernetes pod being OOM killed, with a world snapshot that preserves the telemetry queries the agent would need—such as where to find memory metrics, container logs, and deployment events—rather than raw data.
The agent never sees the root cause directly. It only has access to the signals that existed when the issue occurred. Our evaluation needs to reflect that constraint. Each label has to preserve the same signals the agent would have seen in production.
At the same time, the set of labels must be broad enough to reflect reality. From Kubernetes pod failures to Kafka lag, and from simple bad-code deployments to complex multi-service business logic side effects, the real world of SRE spans many technologies, failure modes, and levels of complexity. Our label set has to reflect that diversity. A narrow or overly clean dataset would inflate performance and hide weaknesses.
Orchestrating evaluations at scale
The evaluation platform is the system that runs Bits against our label set, scores the results, and tracks performance over time.
We needed to know whether an improvement for Kafka lag investigations had accidentally broken our Kubernetes investigations. Answering that meant running both at once, across different model and configuration variants, and comparing results across runs.
From there, the requirements became clear. We needed to segment the label set by relevant dimensions, run investigations at scale, track results over time, and make it easy to compare performance across versions.
At a high level, the system consists of a shared label set, an orchestration layer that runs investigations against those labels, and reporting infrastructure that tracks performance over time.
With this architecture in mind, we'll walk through how each piece came together, starting with the labels.
Starting with manual labels
Given the range of scenarios Bits handles, a small set of hand-crafted labels wasn't enough. We needed broad, representative coverage from the start. So we began with a manual internal labeling campaign, generating labels from Datadog's own alerts across a wide range of scenarios.
This got us started, but we were burning engineering hours faster than creating labels, and our label set was still nowhere near representative of the real world.
Embedding label creation into Bits AI SRE
To scale label creation, we turned to the one system that already understood every investigation: Bits itself. When customers provide feedback on a Bits AI investigation, we use that signal, along with the information from the investigation itself, to construct a ground truth root cause analysis and the queries that make up the world snapshot. Every user interaction becomes a potential evaluation label.
This turned label collection from a manual effort into a pipeline that grows with product usage. As adoption increases, so do the volume and diversity of our labels. Embedding label creation in the product increased our label creation rate by an order of magnitude.
From manual review to agentic validation
Before reconstructing the signals of a label's world snapshot, we required human review to ensure quality. Early on, this process was heavily manual, especially when customer feedback was ambiguous or the fidelity of a generated label was unclear.
As our label ingestion rate grew, manual review could not keep up. We were at risk of losing valuable feedback signals simply because we couldn't process them fast enough.
To address this, we used Bits itself to assist before human review. Grounded in customer feedback and investigation telemetry, Bits aggregates related signals, derives relevant relationships, and resolves ambiguous references in feedback. For example, it can turn "it was slow" into a more precise statement about the elevated latency in a specific service. Since Bits now knows the true root cause, it can build a full causal chain that starts with the problem statement (such as a monitor firing or a user initiating an investigation) and ends with the underlying root cause.
Just like diagnosing the root cause of an issue, this derivation of the root cause analysis was a high-precision, low-margin-of-error operation; however, we were confident our agent's quality had reached the level where this was possible. We also produced several alignment studies with human judges to ensure we were producing high-quality and causally accurate root causes.
The result is a proposed ground truth and signal set that holds up under review and supports a complete root cause analysis.
As this agentic flow improved, human involvement shifted up a level. Instead of manually assembling root cause analyses from raw signals, reviewers now validate and refine Bits' outputs.
The results were dramatic: Validation time per label dropped by more than 95% in a single week.
As confidence in the validation pipeline has grown, we have reduced the amount of human intervention required, without sacrificing label quality.
To ensure label quality, each generated label is assigned confidence scores, and anything below a defined threshold is flagged for human review. These scores evaluate the generated RCAs across several dimensions, including thoroughness, specificity, and accuracy.
We observed roughly a 30% increase in the quality of root causes in the generated labels—root causes that would hold up under a “5 Whys” analysis in a postmortem. These higher-quality labels also enabled more robust evaluation.
Instead of scoring only the final conclusion, we could evaluate the agent's trajectory. We looked at how close it got to the correct answer, whether it investigated deeply enough, and whether it was able to surface valuable telemetry. This allowed us to understand not just whether Bits got the correct answer, but how helpful its investigation was.
Bring the noise
The most counterintuitive thing we learned was that our simulated worlds need to be messy.
With a well-constructed label in hand, we have the ground truth and the signals that surrounded the issue. But telemetry has a limited time to live (TTL). To evaluate the agent later, we reconstruct the investigation context, capturing the structure and relationships across signals, abstracted from the underlying telemetry data, as a snapshot of the world at the moment of issue.
In effect, we build a simulated environment that mirrors the original investigation context, then run Bits inside it. Each environment is fully isolated at the data layer so that investigation context from one label can not affect another. This allows Bits to face the same constraints it would encounter in production, scoped to a single environment.
One key discovery was that these simulated worlds need to be noisy. Snapshotting only the signals directly tied to the root cause is not enough. In production, Bits operates in environments full of unrelated services, background errors, and tangential signals.
To reflect that reality, we capture more than the minimal signal needed to explain the issue. We expand the snapshot by discovering related components based on the root cause chain, even if those components are not directly involved in the failure itself. A component might be included because it belongs to the same platform, team, or monitor, or even just similarly named.
This approach provides a cost-effective mechanism of injecting real-world noise into the evaluation process, mirroring the way an SRE must sift through red herrings during an investigation.
Without that noise, evaluation results looked better than they should have. We were essentially giving the agent an open-book exam with only the relevant pages. No wonder it aced it. The agent appeared more accurate in these simplified environments than it did in real investigations.
Snapshotting telemetry is a one-way door. Once telemetry expires, its structure and signals cannot be reconstructed. When we realized our early labels were too narrow, we had to discard many of them and regenerate those labels with a broader signal reconstruction scope. In the short term, the numbers looked terrible. This reduced our pass rate by roughly 11% and decreased our label count by 35%. But in the long term, it made our evaluations predictive of production behavior.
The evaluation system evolved across three major components: label collection, label validation, and signal reconstruction. Early versions relied heavily on manual workflows, but as the platform matured, each of these stages became increasingly automated and integrated into the product. The following diagram summarizes this progression, from the initial manual system to the industrialized pipeline we run today.
Segmenting, scoring, and catching regressions
With labels collected, processed, and signals reconstructed, we needed a system to run evaluations and make the results actionable.
The platform lets the team segment the label set across multiple dimensions, including technology, problem type, monitor type, and investigation difficulty.
This segmentation lets us scale development across the team. Engineers can focus on the parts of the agent they are improving and evaluate changes against scenarios that matter most, without interfering with other workstreams.
On the reporting side, we store scores for every scenario across every run. We track these results in Datadog dashboards and Datadog LLM Observability so we can compare performance across agent versions. We also maintain an internal labeling application, allowing for centralized observability and metadata management of our labels.
Historical visibility is useful for spotting shifts in behavior. A previously failing scenario starting to pass is informative, as is a previously passing scenario starting to fail.
These kinds of historical score tracking, along with linking to agent metadata, help us understand agent success evolution over time, areas where the agent is strong or weak, and label attributes such as consistently passing, consistently failing, or metrics like pass@k (for a scenario, given k independent attempts, does the agent succeed on at least one of them).
In addition to more targeted runs, we run the full evaluation set weekly to catch regressions that may have slipped through. For example, recently we started internally dogfooding a new tool reasoning strategy. Results looked great on a small subset of evaluation cases; however, upon running the full set, we reported the regression immediately. Results from these runs flow into dashboards and Slack notifications, and we alert on significant deviations in overall performance.
What we'd do again (and what we'd do sooner)
Building this platform changed how we think about agent development. A few lessons stand out.
Invest in label collection and processing early
Manual collection doesn't scale. People scale linearly, but evaluation needs grow faster as the agent expands into new domains.
Using Bits itself to perform quality checks and fill gaps in labels—rather than requiring high-toil human review—removed the biggest blocker to scaling that system.
This shift required careful scoring and alignment work, but it paid off quickly. Label creation rates increased dramatically, and it pushed us to build better reporting around label quality so we could monitor the health of the label set over time.
Build the platform to be extensible from the start
Bits evolved faster than we expected, and so did the models powering it. If adding a new label type, integrating with new data sources, or modifying the underlying models requires significant rework, the evaluation system becomes a bottleneck.
For example, only weeks after releasing the Bits AI SRE Agent, we were able to develop a new agent architecture and capability set for a v2 release. That development speed was only possible because the evaluation platform was designed to evolve alongside the agent.
Use evaluation data to steer product direction
Segmenting results by domain shows where the agent performs well and where it struggles. When we identify a weak area, we expand the label set in that domain. We actively seek out the hardest scenarios, mining negative feedback and exploring frontier areas where the agent is least proven. The labels that matter most aren't the ones Bits passes. They're the ones it fails.
In some cases, we even create labels for capabilities the agent does not yet support. This lets us build evaluation suites alongside new features instead of retrofitting them later.
From single investigations to organizational learning
The feedback loop we built for Bits is now extending beyond a single agent.
We have extended this evaluation platform across other agents at Datadog, turning label collection from human signals into fuel for additional products. Additionally, in following our example, agents across Datadog are starting to personalize their reasoning loops based on evaluation information provided by users, allowing for high agentic precision and reliability across the organization.
In the process of expanding this platform, we've also widened the top of the evaluation funnel even further. Our agentic label collection now extends into the everyday workflows of software engineers at Datadog. Internal incidents, issues, and alerts can be transformed into coherent evaluation labels. This has allowed us to bootstrap other Datadog teams, such as APM and Database Monitoring, as they build and refine their own agentic features. Any team building an agent now has access to a large, representative label set and evaluation infrastructure from day one.
The evaluation platform also changes how we respond to new models. New models don't just offer incremental improvements. They can unlock new workflows and capabilities. When a new model becomes available, we run it against the full label set to measure its impact across domains and understand what it improves and what it breaks. Instead of discovering those shifts in production, we evaluate them upfront. When Claude Opus 4.5 became available, we ran it against our full label set within days and identified which investigation types improved, and more importantly, which ones regressed. That kind of rapid, systematic evaluation of a new model would not have been possible a year earlier.
Building a reliable AI agent is as much about evaluation infrastructure as it is about the agent itself. When we started, we had no standardized way to track quality, catch regressions, or understand how features generalized across real-world scenarios. By building an evaluation platform fueled by diverse, representative labels collected directly from the product, we created a feedback loop that scales with usage and keeps Bits improving.
Along the way, we learned that noise matters, that manual processes don't scale, and that the evaluation platform has to keep pace with the agent it supports. Every week, we run Bits against tens of thousands of scenarios drawn from real incidents. Every week, something surprises us. That's the point.
We didn't set out to build an evaluation platform. We set out to build an agent that could investigate production incidents. The evaluation platform is what it took to trust it.
If you're excited about building infrastructure that evaluates autonomous agents across complex, multi-signal production systems, we're hiring.
How to build CI/CD observability at scale
GitLab leverages Prometheus, Grafana, and custom pipeline exporters to achieve scalable CI/CD observability, which helps optimize pipeline performance, job efficiency, and infrastructure capacity planning for its enterprise self-managed environments.
Summary
Decoder
Original Article
How Cloudflare responded to the “Copy Fail” Linux vulnerability
Cloudflare rapidly mitigated the "Copy Fail" Linux kernel vulnerability (CVE-2026-31431) disclosed on April 29, 2026, by deploying a custom eBPF-based solution across its 330-city infrastructure within hours, preventing any customer impact even before a patched kernel could be fully rolled out.
Summary
Deep Dive
Decoder
Original Article
How lakebase architecture delivers 5x faster Postgres writes
Neon's "image generation pushdown" technique, implemented in its lakebase architecture, drastically improves Postgres write throughput by up to 5x and reduces WAL generation by 94% by offloading full-page write operations to the distributed storage layer.
Summary
Deep Dive
Decoder
Original Article
Kubernetes v1.36: Server-Side Sharded List and Watch
Kubernetes v1.36 introduces server-side sharded list and watch as an alpha feature, allowing API servers to filter events at the source and send only relevant resource slices to horizontally-scaled controller replicas, significantly reducing network, CPU, and memory overhead.
Summary
Decoder
Original Article
Azure DevOps MCP Server April Update
Azure DevOps MCP Servers received an April update adding WIQL-based work item querying, introducing tool annotations for safer LLM interactions, expanding repository tooling, and beginning a consolidation of existing tools for improved user and LLM performance.
Summary
Decoder
Original Article
Kubernetes v1.36: Declarative Validation Graduates to GA
Kubernetes v1.36 has moved Declarative Validation for native types to General Availability, replacing thousands of lines of handwritten Go validation code with +k8s: marker tags for more consistent, maintainable, and self-documenting API constraint enforcement.
Summary
Decoder
Original Article
Airbnb Co-founder Taps Peter Arnell as First US Chief Brand Architect
Airbnb co-founder Joe Gebbia has appointed veteran designer Peter Arnell as the first US Chief Brand Architect for the National Design Studio, aiming to overhaul 27,000 government websites for a unified and trustworthy user experience.
Summary
Decoder
Original Article
Sketchy iPhone 18 Pro Dynamic Island rumors continue with claimed CAD images
Unreliable rumors and easily faked CAD images continue to surface, suggesting the iPhone 18 Pro might feature a smaller Dynamic Island, despite a lack of convincing evidence.
Summary
Decoder
Original Article
Google unveils Whoop-like screenless Fitbit Air
Google has introduced the $100 Fitbit Air, a screenless, lightweight fitness wearable similar to Whoop, alongside a new Gemini-powered Google Health Coach for Premium subscribers.
Summary
Decoder
Original Article
St. Augustine and AI's false promise
AI's promise to optimize decisions is fundamentally flawed because it can only embody human-defined values, which are inherently partial and contested, rather than offering objective truth or solving moral dilemmas.
Summary
Deep Dive
Decoder
Original Article
The Future of Design—What's Next?
The design profession is evolving from aesthetics to strategic influence, pushing designers to adopt "humanity-centered design" to tackle complex societal issues through interdisciplinary collaboration and systems thinking.
Summary
Decoder
Original Article
Revive Your Design Superpowers
Designers possess innate "superpowers" as investigators, explainers, and negotiators of ideas, and should internalize this value to increase their influence rather than constantly seeking external validation.
Summary
Original Article
We built this. Now we own it
Emotionally engaging AI chatbots are an ethical consequence of tech's long-standing focus on engagement and individualistic design, making tech professionals directly responsible for systems that exploit human vulnerabilities.
Summary
Original Article
How universal appeal gets designers to hide their best skills
Designers seeking universal appeal by chasing every new tech skill, including AI, are overlooking a more valuable and future-proof advantage: deep domain expertise in specific industries like healthcare or B2B SaaS.
Summary
Decoder
Original Article
Color Memory Game (Website)
Dialed.gg has launched a free browser-based Color Memory Game where players recreate five colors from memory using hue, saturation, and brightness sliders, scored by the CIEDE2000 perceptual color distance model.
Summary
Decoder
Original Article
Digital Comics Platform (Website)
Panels Store offers a digital comics platform where users can buy and read a diverse range of comics, from cyberpunk noir like "Fluorescent Killers" to epic fantasy and indie comedy.
Summary
Original Article
AI Image Generator Built for Professionals (Website)
Higgsfield.ai has launched SOUL 2.0, a photorealistic AI image generator specifically engineered for creative professionals to convert text prompts into high-quality visuals.
Summary
Original Article
Rethinking the Experience of System Tools
Lead Product Designer Kyrylo Levashov argues that utility software, unlike physical products transformed by brands like Dyson and Method, remains an emotional chore because designers make fundamental assumptions that neglect the user experience.
Summary
Deep Dive
Original Article
The Psychology Behind Well Designed Websites People Actually Remember
Websites make a crucial first impression within 50 milliseconds based on visual cues alone, not content, with the domain name acting as the foundational memory device.
Summary
Deep Dive
Decoder
Original Article
When AI decides and human signs off
Many "decision support" AI systems inadvertently push human users to blindly trust AI outputs, creating an illusion of oversight while shifting actual responsibility onto the user.
Summary
Deep Dive
Original Article
This Website Takes the Cacophony of NYC's Subway and Turns it Into Jazz Music
Designer Joshua Wolk created "Train Jazz," an interactive website that transforms real-time NYC subway data into live jazz music, giving each train line a unique instrument.
Summary
Decoder
Original Article
Disney and Pixar Love this Latte Artist's Delightful Animations
London-based multidisciplinary artist Hazel Zakariya, known for her intricate smoothie bowl art, is now creating acclaimed animated latte art featuring characters from Disney, Pixar, Hello Kitty, and Snoopy, even earning praise from Disney and Pixar themselves.
Summary
Original Article
iOS 26.5 adds beautiful wallpapers for your iPhone, here's what's new
iOS 26.5 introduces a new Pride wallpaper collection for iPhone, featuring 11 preset designs and a custom builder allowing users to create unique wallpapers with up to 12 selectable colors.
Summary
Original Article
SEC's Constructive Stance Reopens Tokenization Buildout
Nasdaq President Tal Cohen stated at Consensus Miami 2026 that the SEC's "much more constructive" regulatory stance, contrasting with a previous "no-fly zone," is now allowing market operators like Nasdaq to experiment with tokenization and digital market infrastructure.
Summary
Original Article
Kraken to Buy Stablecoin Payments Firm Reap in $600 Million Deal
Kraken's parent company, Payward, agreed to acquire Hong Kong-based stablecoin payments firm Reap for $600 million in cash and stock, valuing Payward at $20 billion, marking Kraken's first infrastructure acquisition in Asia as it prepares for an IPO.
Summary
Original Article
Virtuals Protocol's ACF Mechanism
As crypto VC funding plummeted to $659 million in April and onchain perp DEX volume dropped to $699 billion in March, Virtuals Protocol’s Automated Capital Formation (ACF) mechanism offers a new method for projects to secure funding by distributing 25% of token supply to founding teams in USDC tranches based on climbing Fully Diluted Valuation (FDV) without market impact.
Summary
Decoder
Original Article
Introducing Amazon Bedrock AgentCore Payments
Coinbase announced its x402 discovery layer and wallet infrastructure are now integrated into Amazon Bedrock AgentCore Payments, allowing AWS developers to build AI agents capable of autonomous service discovery, micropayments, and task completion, settling in USDC on Base and Solana.
Summary
Decoder
Original Article
Tether-Circle Duopoly Hampers Stablecoin Product-Market Fit
Ben O'Neill, head of money movement at Bridge, argued at Consensus Miami that the combined $260 billion dominance of Tether and Circle in the $306 billion stablecoin market stifles the product diversity needed for specialized payment use cases, citing structural fee issues with both issuers.
Summary
Decoder
Original Article
Bermuda Expands USDC Airdrop
Bermuda's government is expanding its "onchain economy" initiative with a second USDC airdrop, offering up to $100 to residents who use Coinbase-supplied wallets at participating local merchants, as Premier David Burt positions the island as the first fully onchain national economy.
Summary
Decoder
Original Article
Bitcoin Overtakes Gold as Debasement Hedge
JPMorgan reports that Bitcoin has surpassed gold as the preferred debasement hedge among investors following the Iran conflict, with Bitcoin ETFs seeing three consecutive months of inflows while gold ETFs face outflows.
Summary
Decoder
Original Article
Crypto Is Only Hiring in NYC
Crypto startups in 2026 are increasingly enforcing NYC-only hiring mandates, mistakenly conflating proximity to capital with a need for entire teams to be based in Manhattan, leading to a 90% reduction in viable candidates and intense competition with top-tier financial firms.
Summary
Decoder
Original Article
Crypto Apps Are Payments + Auth Layers in Disguise
Georgios Konstantopoulos suggests that crypto applications are fundamentally advanced payment and authentication layers, enabling users to control their assets via biometrics and grant granular access to applications for always-on, cross-border ownership.
Summary
Original Article
Solv Protocol Drops LayerZero for Chainlink
Solv Protocol is moving over $700 million in tokenized bitcoin from LayerZero to Chainlink's CCIP, citing enhanced security assurances after the recent $292 million Kelp DAO bridge exploit highlighted vulnerabilities in cross-chain infrastructure.
Summary
Decoder
Original Article
NFTs may make a comeback as AI agents strain online identity
LinkedIn co-founder Reid Hoffman predicts a "rebirth" for NFTs as AI agents increasingly strain online identity and trust, arguing that crypto-based identity systems will be essential for secure transactions between agents on the open internet.
Summary
Decoder
Original Article
Coinbase Posts $394M Loss as It Pushes Beyond Spot Trading
Coinbase reported a significant $394 million loss in Q1 and a 31% revenue decrease, largely due to falling crypto prices impacting trading activity and its balance sheet assets, prompting its CEO to seek reduced dependence on volatile spot trading.
Summary
Original Article
Digest devoured!
May 8