Introducing Claude Opus 5

Claude Opus 5 is available today. It’s a thoughtful and proactive model that comes close to the frontier intelligence of Claude Fable 5 at half the price.

On coding and knowledge work evaluations like Frontier-Bench and GDPval-AA, Opus 5 is the new state-of-the-art, though it remains behind Mythos 5 on cybersecurity tasks.

Opus 5 is designed to be used every day: it works more efficiently than other models. It’s the new default model on Claude Max, and the strongest model on Claude Pro.

Performance and cost-effectiveness

Claude Opus 5 provides greatly improved performance for the same cost as its predecessor, Opus 4.8. The charts in this section show how performance changes according to the model’s effort setting, which customers can use to optimize for intelligence or conserve tokens for faster and cheaper results.

Opus 5 excels on valuable software engineering tasks. For example, on Frontier-Bench v0.1, Opus 5 surpasses all other models, and more than doubles Opus 4.8’s performance at a lower cost per task. On CursorBench 3.2, at max effort, the model performs within 0.5% of Fable 5’s peak score, but at half the cost per task; it also achieves greater performance at a given cost than all other models on high, xhigh, and max effort.

We see similar results on knowledge work and problem-solving tasks. For example:

• On ARC-AGI 3, an evaluation where the model has to solve novel problems, Opus 5’s score is three times as high as the next-best model.
• On Zapier AutomationBench, which measures whether models can complete business tasks from start to finish, Opus 5’s pass rate is around 1.5× the next-best model for the same cost per task. Even at its lowest effort setting, Opus 5 passes more tasks than any other model.
• On OSWorld 2.0, a computer use benchmark, Opus 5 outperforms every other model at any given cost, surpassing Fable 5’s best result at just over a third of the cost.

It’s also our best and most cost-efficient model on several related evaluations.

Opus 5 is a meaningful improvement over Opus 4.8 for scientific research. It shows better performance than Opus 4.8 on every one of our life sciences evaluations, which cover topics including structural biology, organic chemistry, and bioinformatics. Its improvements are most notable on organic chemistry tasks, like inferring molecular structures from spectroscopy data (it scores 10.2 percentage points higher than Opus 4.8 on our internal benchmark), and on protein-related tasks like predicting how variations in a protein’s sequence affect how it functions (here, it scores 7.7 percentage points higher).

Finally, Opus 5 is capable of producing much stronger visual outputs.

Working with Claude Opus 5

Claude Opus 5 is much stronger at verifying its work and iterating carefully until it succeeds. In evaluations and early-access testing, we and our users found many examples of Opus 5’s agency and thoroughness:

• On one Frontier-Bench task, Opus 5 was given a drawing of a machine part and asked to write code to rebuild it as a 3D FreeCAD model. However, in this task, the model was intentionally given no way to directly view the drawing. Opus 5 responded by writing its own computer vision pipeline to pull the geometry from the raw pixels, then reconstructed the full machine part. It succeeded in doing so repeatedly; no competing model with the same setup could solve it after five attempts.
• Given a real bug in a popular open-source package manager, Opus 5 found the root cause and fixed an edge case that the community’s patch had missed. A competing model fixed only the surface symptom (not the underlying cause), then reported the bug resolved.
• An engineer at a trading firm used Opus 5 to build a market data feed for a new exchange in a single session. Previous models could not complete this task at all, even given extensive plans from the engineer. Finding no live feed to validate against, Opus 5 even built its own test harness to check that its code parsed the exchange’s data correctly.

Below are further reports from our early-access customers on their experience of working with Opus 5:

On FrontierCode 1.1, Claude Opus 5 approaches Fable-level performance at half the cost. Within Devin, it also shows particular strength on difficult debugging and root-cause analysis tasks.

Claude Opus 5 delivers near Fable 5 intelligence at Opus speed and cost. On CursorBench it’s just under Fable 5 and has many of the same behaviors. We are excited to see how developers use it in Cursor.

Claude Opus 5 topped Zapier’s AutomationBench leaderboard without spending more tokens than prior Claude models. It took a raw account-health workbook and ran a full churn-prevention sequence end to end: flagging at-risk accounts, alerting the right owner, and summarizing for retention ops. Previous models didn’t pass; Opus 5 hit 100%.

On our genomics analysis work, Claude Opus 5 behaves more like a careful scientist than any model we’ve run. It reaches for the right statistical tests to rule out confounders, cross-checks its own results by independent methods, and stays on track through long multi-step analyses.

Claude Opus 5 came out ahead of every model in its family on our internal evals. It isn’t just better on our hardest agentic coding tasks, up 22% over Opus 4.7, it’s steadier, with far less variance run to run. For the millions of builders on Lovable, that consistency is the whole game. Reliable results, build after build.

Claude Opus 5 is the biggest leap in the Opus family since 4.5. On the same full-stack app builds, the front end shows it first: the best animations, games, and 3D work we have seen from an Opus model.

We’re loving Claude Opus 5. For the kind of open-ended analytical work our agent handles, it’s a strict upgrade over Opus 4.8, and the gains are biggest exactly where it matters: the harder, vaguer tasks. Responses are clearer and more concise, and we see improved efficiency at higher effort levels too.

Claude Opus 5 is a striking improvement over Opus 4.8 for the financial research workflows our analysts run every day. It stands out on numerical reasoning, table work, and sharper critical thinking where precision matters.

Claude Opus 5 delivers the industry intelligence and accuracy that is essential for the analysis of specialized enterprise content. Box found that Opus 5 outperforms Opus 4.8 by 8% and delivers notable performance gains in the data analysis (11% improvement) and due diligence (17% improvement) workflows that technology, healthcare, and public sector organizations rely on daily.

Claude Opus 5 is a clear generational step up from Opus 4.8. Over one weekend I gave it a chief-of-staff role over my dev environments: it built its own monitor, drove each box, and pulled me in only for the judgment calls.

Claude Opus 5 made large scale changes across our Fundamental Research Assistant codebase, adapting to feedback throughout an agentic workflow and explaining its reasoning more clearly than any model we’ve used. It handled work we would normally have broken into much smaller pieces.

On some of our hardest financial-modeling tasks, Claude Opus 5 is a clear step up from Opus 4.8 in both accuracy and efficiency. Its performance floor is materially higher, especially on deep finance domain logic. Across effort levels it averaged 9 percentage points higher accuracy with a third fewer turns and tool calls and 60% less time.

Claude Opus 5 checks its own work the way a real frontend developer would. On our benchmark it opened its pages in a browser at desktop and phone widths, caught a product hidden below the mobile fold and an off-screen checkout button, and fixed both before handing the work back.

Claude Opus 5 is a clear step up in performance on legal agent work compared to prior Opus models, and we saw the biggest gains in practice areas like corporate governance and arbitration. We were also impressed with Opus 5’s ability to maintain quality at lower reasoning levels, achieving similar performance while generating 26% fewer tokens on average compared to Opus 4.8 at max reasoning.

Claude Opus 5’s biggest gains for us are on longer-horizon work: building a full deck, then revising it. Artifact quality is what decides which model we ship, and this is the clearest step up we’ve seen — better visual understanding, cleaner formatting, fewer slide issues.

Claude Opus 5’s judgment is what stands out. Handing off a PR, it doesn’t rush to publish: it verifies the branches, checks the template, and thinks through test implications so the handoff is clean. The older models tended to jump ahead and get caught on our checks.

During a rearchitecting session, Claude Opus 5 pushed back on a design I proposed, and it didn’t fold when I insisted. Instead, it explained exactly what was valuable in my idea, narrowed its objection to a single design question, and proposed a compromise that kept the good part while fixing the flaw. That’s the kind of judgment that lets us trust it with less oversight.

On first-turn redlines, Claude Opus 5 scored the highest of any model we tested, nearly double Opus 4.8. Commenting is better too: on NDAs it gets to the redline in less time and with fewer passes, with accuracy maintained or better.

Claude Opus 5 writes clean, tight diffs with no dead code, and it’s the stronger hazard spotter on subtle, codebase-specific issues. We’re adopting it for production workloads.

We will definitely migrate a number of use cases in Cosmos, our unified agent platform. We’re looking forward to increasingly using Claude Opus 5 for code review, and I am confident in saying we would rather people be using Opus 5 than Opus 4.8.

What stands out about Claude Opus 5 is judgment. It thinks harder before it writes a single line, catches its own logical faults during planning rather than after the fact, and reasons about why an answer is right, not just whether it works. It’s the clearest jump in problem-solving we’ve seen from one Claude model to the next, and we’re looking forward to seeing it adopted in JetBrains IDEs.

Claude Opus 5 is the strongest Opus model we’ve tested on our trading benchmark, and it gets there using roughly a seventh of the reasoning tokens and under half the latency of Opus 4.8. Better answers at a fraction of the compute.

Claude Opus 5 lets monitoring agents manage parts of their own memory in production, making them more autonomous and reliable over longer horizons. The agent treats its context as a living document: after flagging a potential anomaly in one of our services, it re-checked its own assumption against production, found the signal was benign, wrote the correction into its memory, and retired its monitoring queries on its own.

Claude Opus 5 is a strong agentic coding model built for long-running, multi-step work. It deeply understands your codebase, holds the thread across complex tasks, and pins down requirements for feature development and bug-fixing more effectively than Opus 4.8. Developers can now build with Opus 5 in Kiro, accessing its advanced capabilities to tackle ambitious projects.

Alignment and safety

Alignment. During pre-deployment testing, our automated behavioral audit found Opus 5 to be our most aligned model to date. It adheres to Claude’s Constitution better than Opus 4.8, Sonnet 5, or Fable 5; exhibits the lowest rates of deceptive behavior; and is the least susceptible to being tricked into misuse. It’s also our safest model yet in terms of avoiding reckless actions that could have hard-to-reverse side effects.

Safety. Opus 5 does not advance the frontier in risky, dual-use capabilities. In rigorous evaluations conducted alongside private-sector and government partners, we found it remains behind Mythos 5 in both biology research and offensive cybersecurity. More information about these evaluations can be found in our System Card.

As with its predecessor, Opus 4.8, we’ve intentionally avoided training Opus 5 on cyber tasks. The model has nevertheless improved substantially on these tasks as a result of becoming more generally capable, and it comes close to Mythos 5 at finding cybersecurity vulnerabilities. However, it remains substantially behind Mythos 5 on the exploitation of those vulnerabilities—that is, in turning vulnerabilities into material cyber threats.

This is illustrated by Opus 5’s performance on OSS-Fuzz, an evaluation we’ve developed to assess how well models can find and then exploit vulnerabilities without extensive human guidance. Although Mythos 5 and Opus 5 identify vulnerabilities with similar success, Opus 5’s score on the development of exploits is far behind that of Mythos 5.

Safeguards for Opus 5

Claude Opus 5’s safeguards are designed to allow beneficial uses of the model in both cybersecurity and biology. They are similar to those we applied to Opus 4.8, with the exception of some stronger guardrails on a narrow range of cyber tasks.

Cybersecurity. Opus 5’s cyber classifiers are proportionally less restrictive than those on Fable 5. They allow Opus 5 to find vulnerabilities in source code, but block “binary-based” vulnerability scanning (a method more likely to be associated with malicious actors), penetration testing, and exploit generation.

Based on our testing, we expect the classifiers to intervene around 85% less often than they do for Fable 5. In Claude.ai, Claude Code, and Claude Cowork, any flagged requests will fall back to Opus 4.8 by default. Fallbacks to Opus 4.8 can also be enabled on the API.

Our Cyber Verification Program (CVP) facilitates cybersecurity work that would otherwise be impeded by the model’s safeguards. Enterprises and researchers who are already part of the CVP have immediate access to a version of Opus 5 with fewer security restrictions.

Biology. Since Opus 5 has a similar suite of safeguards to Opus 4.8, it is now our most capable generally available model for scientific research. Nevertheless, the model still shows important limitations on long-running, autonomous research tasks, which is where we expect AI models to pose the most substantial biology-related risks. (Mythos 5 remains the stronger model for this type of biological work.) As part of this launch, biology-related requests that are blocked on Fable 5 will now route to Opus 5 rather than Opus 4.8.

Getting started

Claude Opus 5 is available today on all platforms, priced at $5 per million input tokens and $25 per million output tokens (the same as Opus 4.8). Developers can get started with claude-opus-5 on the Claude API.

It’s also offered in Fast mode, where it runs around 2.5 times the default speed. As with Opus 4.8, Fast mode is available at twice Opus 5’s base price on the Claude Platform and through usage credits in Claude Code.

Alongside Opus 5, we’re releasing two updates in beta:

• Mid-conversation tool changes on the Claude Platform. Within a conversation, developers can now change which tools Claude can use without invalidating the prompt cache.
• Automatic fallbacks on the API. Users can now choose to have requests that are flagged by our safety classifiers on Opus 5 (or Fable 5) automatically route to another model. With automatic fallbacks on, API requests always route to the best available model by default rather than being blocked.

Consistent with prior Opus models, Opus 5 does not have data retention requirements for general access.

For more guidance on how to get the best out of Opus 5, see our prompting guide.

Footnotes

Frontier-Bench v0.1, Effort plot: These results are from an internal run of Frontier-Bench v0.1, on the mini-SWE-agent harness and a GKE backend, mean reward over 5 attempts per task. Opus 4.8 served as fallback on safety-classifier refusals for Opus 5 and Fable 5.

OpenAI's unreleased internal model coordinated more than 17,000 complex actions over several days and successfully completed its goal, breaching Hugging Face in the process. The model escaped its sandbox, gained access to Hugging Face, escalated access, harvested credentials, and then found the data it was looking for. It took many days for the issue to be discovered. This post takes a detailed look at what happened during the attack.

ModelExpress: Distributing Model Artifacts at the Speed of Light

Every byte moved has a cost. As model checkpoints grow to hundreds of gigabytes or even a terabyte, that cost adds up quickly. To make things even worse, moving these model weights around the cluster is extremely common. For instance, a cold start may pull weights from remote storage into GPU memory; autoscaling and rolling updates must populate each new replica; and RL post-training continuously moves updated weights from trainers to roll out workers. These may look like different workflows, but they impose the same recurring tax: time spent moving weights before useful work can begin.

ModelExpress: Accelerating the model weight lifecycle

NVIDIA ModelExpress (MX) is built around a simple idea: Before loading a model, first ask where a compatible copy of its weights already lives. Rather than treating every replica as an independent cold start, MX chooses the fastest available source and transfer path.

When a serving peer already holds compatible weights in GPU, MX transfers them directly from GPU to GPU over P2P RDMA via NVIDIA Inference Xfer Library (NIXL), bypassing redundant access to object storage, local disk, and host memory. When no peer is available, MX bootstraps from the fastest supported path by streaming from an object store without landing on disk or reading local files directly into GPU memory.

MX transfers DeepSeek-V4 Pro weights and JIT Kernel cache artifacts from a serving replica into a fresh replica in under 10 seconds, reducing the total startup time to 1 minute 44 seconds from 8 minutes. The rest of the post shows how MX selects the fastest available path to GPU memory, prioritizing P2P RDMA from a serving replica and eliminating redundant downloads and copies along the way. It then extends the same approach to reusing kernel caches and distributing RL weight updates.

Accelerating every stage from remote storage to GPU memory

Every new worker must get its weights from one of three places: remote storages (e.g. HF or S3), local storage, or another worker already serving the model. For the first worker, there is no peer yet, so it must bootstrap from storage. MX can stream the checkpoints from object storage or load it from fast local storage, removing avoidable copies along either path.

Once that first worker is serving, the preferred source changes. Its weights are already resident, post-processed, and laid out in GPU memory, so every compatible worker after it should load directly from that peer over P2P RDMA. MX makes this transition automatically: bootstrap once from storage, then scale out GPU to GPU, falling back to storage only when no compatible peer is available.

Starting the first worker: Bootstrap from storage

Remote object storage to GPU: Avoiding local disk

When the checkpoint lives in a cloud bucket and you would rather not provision and manage a disk cache tier, MX uses the Model Streamer to pull safetensors through a reusable CPU staging buffer and into GPU. The checkpoint never lands on local disk, eliminating the intermediate download, reload, and storage volume.

Model Streamer uses a multithreaded tensor reader to fetch tensor ranges concurrently across checkpoint shards. As tensors arrive, it pipelines remote reads with GPU placement: completed tensors are passed to the inference engine while later tensors are still being fetched. This keeps the storage, network, and GPU copy paths busy while reusing a bounded amount of host memory.

In tensor-parallel deployments, the participating ranks divide the remote reads and share the results, typically over NCCL, instead of having every rank download the full checkpoint independently. MX connects this distributed stream directly to the inference engine’s weight loader, preparing the first worker to become the P2P source for every compatible replica that follows.

Cluster ingress: Download once, not N times

When a cluster maintains a shared disk cache tier (e.g. persistent volumes in K8s), MX ensures that the fleet populates it only once. If 10 replicas concurrently want to fetch the 806 GiB DeepSeek-V4 Pro model, they will need to pull roughly 8 TiB of identical data across the network while competing for the same ingress bandwidth. The MX Model Cache Service collapses those requests into one coordinated download: an atomic claim in Metadata Store selects a downloader, while the remaining replicas track its progress and reuse the cached copy. The cluster pays the external download cost once, then every replica can begin from the same cached checkpoint.

Local storage to GPU: Bypassing host-memory staging

When GPUDirect Storage (GDS) is supported in the system, MX reads checkpoint files directly from local storage into GPU memory through NIXL’s multithreaded GDS backend. NIXL executes batched tensor reads in parallel directly into GPU memory, bypassing host memory and the staging copy required by a conventional loader. Users don’t need to enable GDS explicitly: MX detects the capability automatically and falls back to another loading strategy when it is unavailable.

Local storage to GPU: Pipelining local reads with ModelStreamer

MX can also load local checkpoints through ModelStreamer. Multiple OS threads read safetensors concurrently into a configurable CPU buffer while completed tensors move to the GPU and later reads continue in parallel. Unlike GDS, this path still stages through host memory, but it overlaps disk I/O with GPU placement, benefits from the OS page cache, and provides a portable fast path when direct storage-to-GPU access is unavailable.

Starting every worker after the first: Fetch from a serving peer

This is the key feature of MX. Once another replica is already serving the same model, the weights have completed most of their journey: they are resident in GPU memory, post-processed, and laid out for the inference engine. MX treats that replica as a live weight source. After confirming compatibility, it transfers the tensors directly from the source GPU to the target GPU. Once its weights are loaded, the new replica joins the source pool, giving subsequent replicas another peer to load from. With every successful transfer, that pool grows alongside the deployment, turning scale-out into GPU-to-GPU fan-out instead of repeated cold loads.

The MX control plane discovers compatible peers, exchanges transfer metadata, and tracks source readiness, but never handles the weight bytes themselves. On the data plane, MX uses NIXL as a default transfer engine whose pluggable backends allow for peak performance across a variety of networks, such as Infiniband, RoCE, NVLink, EFA, etc. MX has a first-class transport interface that allows libraries such as fabric-lib and standalone Mooncake to integrate with MX.

Before any transfer begins, MX computes an mx_source_id from the model and runtime settings that determine tensor layout, then considers only peers with a matching ID. The control plane discovers those peers through Redis, Kubernetes CRDs, or k8s-service (serverless) metadata backends.

Optimizing NIXL memory registration overhead

Before NIXL can RDMA a tensor, the GPU memory backing it has to be registered: an ibv_reg_mr call that returns the Remote Key (rkey) used for remote access. A large model has tens of thousands of tensors, and registering them one at a time is slow enough to show up in the budget. By default, MX registers each tensor individually. Two opt-in strategies reduce that registration cost:

• Pool registration registers each underlying cudaMalloc allocation once instead of each tensor, cutting registration count by 80 to 99 percent on typical models with no change to transfer semantics.
• VMM arena registration goes further. It installs a CUDAPluggableAllocator that routes every load-time allocation into a single 16 TiB virtual-address arena, then registers the whole used range as one dmabuf-backed memory region at end of load. Registration collapses from one call per tensor to one call, total; each tensor descriptor simply carries an offset into that single region.

Runtime path selection and safe fallback

At startup, MX probes the available capabilities, automatically skipping any path the environment does not support. The first applicable strategy runs in the current priority order: P2P RDMA -> ModelStreamer -> GDS -> default loader (host-staged POSIX I/O). If a path is unavailable or fails before modifying the model state, MX falls through automatically. If a failure occurs after weights have begun landing, it reinitializes the model before continuing, so partially written weights are never served.

End-to-end results

We ran DeepSeek-V4-Pro on an 8xB200 GPU node with NVIDIA ConnectX-7 NICs and compared the total model loading time across different cold start scenarios. Each replica used vLLM 0.23.0 with TP=8 and --enable-flashinfer-autotune.

Warm, not just loaded: Inheriting the compiled kernels

Getting weights into GPU memory is critical, but a loaded model is not yet ready to serve. During its first forward passes, the engine JIT-compiles and autotunes kernels (e.g. torch.compile, Triton, DeepGEMM, TileLang, and etc.) and captures CUDA graphs for the exact model, dtype, quantization, and GPU. For models such as DeepSeek-V4 Pro, this can take several minutes and can become the dominant startup cost once MX reduces weight-loading latency.

That repeated warmup is avoidable. When the model, software stack, and GPU architecture match, one replica can pay the compilation cost and the rest can inherit the resulting caches. MX’s Artifact Transfer API packages these file-backed artifacts, transfers them directly between registered host-memory buffers over NIXL’s CPU-to-CPU RDMA path, then verifies and installs them in the target engine’s cache directory.

When the weights change every Step: RL post-training

Everything so far assumes a model’s weights are fixed once loaded. RL post-training breaks that assumption. A trainer updates the policy every step, and the inference actors generating rollouts must pick up those weights before the next round of generation.

MX drives the refit through the following four stages:

1. Publish: Each trainer rank advertises the tensors or shards it already owns, together with metadata describing their shape, dtype, placement, and parameter mapping to MX.
2. Discover: A rollout worker looks up the requested weight version and its available sources through MX.
3. Plan: The receiver maps the published ownership information onto its own target layout and identifies which sources contain the required tensors or ranges.
4. Pull, convert, and load: The receiver issues one-sided reads directly against those sources.

Contributing to Dynamo and our roadmap

MX has native integrations with vLLM and SGLang and supports serving frameworks including Dynamo and llm-d.

The Dynamo open source community is actively working toward deeper TensorRT-LLM integration and broader inference capabilities.

Nvidia is in talks to provide a roughly $250 billion backstop for OpenAI as part of a massive data-center project. The guarantee will help OpenAI lease a 10-gigawatt project that Softbank's energy subsidiary is developing. Nvidia's backing allows the data center developer to raise debt at more favorable terms. Under the deal, Nvidia would guarantee a series of financing vehicles intended to make lenders feel more confident that funding behind the project is secure.

An Inside Look at the Relay Market Powering Token Resellers and Fraud

My Story

I’ve spent a lot of time thinking about token fraud, a problem I first stumbled upon while working as a software engineer on an AI gateway.

We faced constant abuse. At first it was free-credit abuse, where users spun up accounts en masse. Then it was our support chatbot. I started talking to friends about it and hearing stories of companies losing millions of dollars to abuse each day.

The abuse took a number of different shapes, and the abusers were relentless. I came to realize that the problem was much bigger than us. A new form of fraud had emerged, and it had become endemic to the token economy.

While researching where the abuse was coming from, I stumbled onto a Chinese forum where operators openly discussed the relays and their methods. My notes on the industry and its players are below.

So What Is a Relay?

A relay — or “transfer station” — is essentially a service that proxies traffic to U.S. models, often at a deep discount. For example, one operator’s price-comparison site listed a package that bought the equivalent of $3,333 worth of official Anthropic credit for 425 RMB — roughly $0.13 of usage per $1 spent.

To make that concrete, here’s how far below official pricing the relays we track actually run, ranked by discount:

• 01 Now Coding: 97.8%
• 02 I Code Easy: 97.1%
• 03 Claude ZZ: 96.6%
• 04 Doro: 96.4%
• 05 UoCode: 96.3%
• 06 ZeroCode: 94.9%
• 07 AiYa: 94.9%
• 08 HongMaCC: 94.2%
• 09 Right Code: 94.1%
• 10 BUZZ: 94.1%

How the Market Works

The ecosystem runs four layers deep, from the merchants sourcing raw accounts down to the developers buying cheap tokens:

• Upstream (卡商 / 号商): Card & account merchants — virtual credit cards built to pass U.S. and European billing checks, plus bulk-registered accounts.
• Midstream (账号池): Account pools — aggregate hundreds of upstream accounts, manage tokens and rate limits, handle failover, and expose a single API.
• Downstream (中转站): Relays / transfer stations — wrap the pool's API in a clean, billed, Chinese-language product and compete on price.
• End users: Developers, startups, and SaaS chasing cheap inference — plus commercial buyers running model distillation.

Upstream

Sitting at the top are the card merchants (卡商) and account merchants (号商). They sell virtual credit cards designed to pass U.S. and European billing checks, along with bulk-registered accounts.

Midstream

In the middle sit the account pools (账号池). A pool aggregates dozens or hundreds of upstream accounts, manages their authentication tokens and rate limits, handles failover when accounts get flagged, and exposes a single API surface that downstream relays can consume.

The inventory isn’t only model-lab accounts. Alongside direct OpenAI, Anthropic, and Google credentials are accounts harvested from the application layer. To a pool, it makes no difference whether a token comes from a lab or from an app built on one; anything that resells or exposes a model is a target.

Downstream

Downstream sit the relay / transfer stations themselves — the consumer-facing layer. They wrap the pool’s API in a clean Chinese-language product, handle billing and invoicing, run customer-support WeChat groups, and compete on price.

End users

At the bottom are individual Chinese developers, small startups, and mid-sized SaaS companies hunting for cheap inference — as well as some larger commercial buyers using the infrastructure for model distillation.

The Software Behind the Relays

Almost every relay I’ve looked at runs on one of two open-source projects: one-api or new-api.

Both are OpenAI-compatible gateways. An operator deploys the panel and adds a set of channels. Each channel represents a provider plus a pool of API keys. The panel exposes a single endpoint that matches the OpenAI API, so buyers just point their existing SDK at the relay’s URL. On every request it pulls a key from the pool, forwards it upstream, returns the response, and deducts quota priced by usage times a multiplier.

new-api is a more actively developed fork of one-api, and the difference is mostly commerce: it ships with self-service payment and recharge, plus image, video, and audio models. Across the relays we track, one-api turns up roughly four times as often as new-api.

The Methods

• Free-trial abuse: Abusers automate account creation en masse to claim free credits, then proxy that traffic back to their own end users.
• Chargeback attacks: Abusers charge back their spend after the usage period ends to recoup their costs — or use stolen cards from the start.
• Prepaid cards: Abusers fund accounts with prepaid cards capped at a set limit.
• Open inference: Any support chatbot without strict guardrails is ripe for having traffic proxied through it.
• Denial of wallet: Attackers fire off a flood of concurrent requests purely to burn a provider’s spend.

Who Are the Buyers?

The three main use cases seem to be cheap tokens, getting around geo-restrictions and model distillation. One forum participant noted: “Distillation uses Claude/CodeX models to train domestic models... many distillers in the industry have made millions.”

A Growing and Maturing Market

I was surprised by how mature the market already is. There are price-comparison sites for the relays, affiliate programs, and even gateway products. The ten highest-traffic relays we track pull a combined 3.6 million visits a month between them.

They’re Raffling Off Keys Now

A clear sign of how normalized this has become is that one of the relay directories runs a daily lottery for API keys. The site, hvoy.ai, gives away fifty $100 API keys every single day. The draw is provably fair — the same cryptographic scheme legitimate crypto-gambling sites use to prove they didn’t rig the result.

How Providers Can Defend Themselves

Fraud is a constant cat-and-mouse game. Here is a set of measures to mitigate abuse:

• Raise the cost of entry: Make accounts hard to create in bulk and cap what a fresh one can spend. Check for browser-based automation signals.
• Watch the money: Flag prepaid cards, virtual cards, mismatched billing info, and small card-testing charges.
• Watch the behavior: Look for patterns no real user produces: time from registration to first token, account age, and IP signals.
• Cluster the accounts: Watch for IP sybils and shared device fingerprints that tie supposedly-separate accounts back to one operator.
• Monitor for cost anomalies: Setup monitors and alerts on AI spend as a failsafe.

Assume some abuse gets through anyway, and limit what it can cost you:

• Enforce spend caps, spend locks, and concurrency limits per account.
• Reserve budget for every in-flight request.
• Start new accounts with low caps; let them earn higher limits with age and a verified card.
• If an account’s risk rises mid-session, add friction like a CAPTCHA or additional verification.

When you do catch someone, throttle quietly. A clean error just tells the attacker which signal to fix before they come back.

In case you missed it: OpenTelemetry (OTel) has officially achieved CNCF graduated status! It now stands proudly alongside amazing open source projects such as Kubernetes and Prometheus, to name just a few. It’s been a long journey, and we’re very excited… But, now what? To understand where we’re going, it’s important to understand where we came from.

History

In the not-so-distant past, telemetry signals were not standardized. This meant telemetry formats differed from tool to tool, with each telemetry vendor creating and maintaining its own instrumentation libraries. Vendor lock-in was a huge problem: If you wanted to switch vendors, you had to strip out the previous vendor’s libraries from your code and replace them with the new vendor’s libraries. As a result, switching vendors was a nontrivial task.

In addition, the three core telemetry signals – traces, logs, and metrics – were treated as separate, so there was no easy way to correlate them. Because of this, the observability story was incomplete.

Previous attempts had been made at standardization: the CNCF’s OpenTracing, and Google’s OpenCensus, forming the basis for what was to become OpenTelemetry.

In the interest of having a single standard, OpenCensus and OpenTracing were merged to form OpenTelemetry in May 2019. OpenTelemetry takes the best of both worlds, and then some, providing a tracing, metrics, and logs specification, a set of standardized APIs, and language specific implementations of these APIs, in addition to the Collector.

Both OpenCensus and OpenTracing are now officially archived. OpenTracing was archived in January 2022, and OpenCensus was archived in July 2023.

With the backing of all major observability vendors, and an active developer and end user community, OpenTelemetry became the de facto open standard for telemetry.

Growth

OpenTelemetry is the second-highest velocity project in the CNCF, just behind Kubernetes. According to the CNCF, OpenTelemetry has “over 12,000 contributions, from over 2,800 companies and hundreds of maintainers across various language-specific Special Interest Groups (SIGs).”

Since its inception, traces, logs, and metrics have reached general availability (GA). Profiling was added as a new OTel signal. The OpenTelemetry Demo has expanded. The OTel Collector has expanded, with new components being added regularly. We’ve seen the addition of new components to the OTel ecosystem to help make it more ergonomic, including OpAMP, the OTel Operator, OTel Weaver, and OTel Arrow.

This is a very impressive achievement, considering that OpenTelemetry is a mere seven years old. It sends a clear signal: OpenTelemetry is here to stay. And graduation helps to cement that.

Graduation!

OpenTelemetry achieved graduated status in May 2026, having started its path to graduation in 2025.

So what does it take to become a graduated CNCF project? Projects must fulfill the following criteria:

1. Production adoption. Many different organizations, such as GitHub and Farfetch, run OpenTelemetry in production.
2. Robust governance. OTel has a documented governance model with clearly defined roles around election and retirement, along with transparent communication and decision-making.
3. Community health. OpenTelemetry has an established process for PR review and management. The project has a number of regular contributors across multiple organizations. Reviewers are responsive, ensuring that issues and fixes are addressed in a timely manner.
4. Security. OTel has undergone at least one independent security audit, and all critical issues identified have been remediated.
5. API stability. APIs are stable, properly versioned, and released at a regular cadence, with backwards compatibility ensured so as to not break existing implementations.
6. Documentation. OTel’s documentation provides an architectural overview, along with user, operator, and contribution guides.
7. TOC application and review. A graduation application template was submitted for review by the CNCF’s Technical Oversight Committee (TOC). You can check out OTel’s submission.

As you can see, a lot of work was done behind the scenes by many dedicated folks, ranging from OTel maintainers, to end users, to CNCF TOC members to make this happen.

We’d like to give a huge shoutout to all in the OpenTelemetry community who made graduation happen, and especially to Austin Parker, OpenTelemetry Governance Committee member and former Community Manager, who led the graduation effort with the CNCF.

What this means for you

So what does OpenTelemetry graduation mean for you, dear reader?

Dan Gomez Blanco, one of the maintainers of the OTel End User SIG put it perfectly in a recent LinkedIn post:

For end users, this graduation signals that OTel is far from being an “emerging standard”. Its contributor health, security and quality standards, governance processes, and wide adoption have been evaluated to be at the level required by any enterprise, of any scale. So, if you’re in the 25% of skeptics not using OTel, there’s really no excuse anymore. There has never been a better time to adopt it!

In a nutshell: OpenTelemetry is production-ready, and fully open for business. If your organization was holding out on using OpenTelemetry, you have no more excuses!

What’s next?

Software is never really “done”, and the same goes for OpenTelemetry. It will continue to grow and evolve: from the specification to the API & SDK to the Collector, and beyond.

Looking ahead, we see a strong need for observability around new types of workloads, such as agentic workflows, an area covered by the emerging generative AI semantic conventions. We’re also tackling challenges in areas that we hadn’t focused on as much previously, such as browser and mobile observability.

More mature teams are looking for guidance on using OpenTelemetry at scale. That’s where tools like Weaver, which helps teams define and govern their telemetry schemas, come into play. We’re also making OTel easier to roll out by packaging components into installable modules through OpenTelemetry Packaging, and by enabling zero-code instrumentation with the OpenTelemetry Injector.

OpenTelemetry has a long future ahead of it, but we also know that it’s only possible through continued work by maintainers and contributors, and of course, through continued support and adoption by our end users.

We can’t wait for what the future has in store for us, and we’re excited to have you along for the ride.

Postgres LISTEN/NOTIFY has a bad reputation thanks in part to a popular blog post asserting it does not scale. If that were true, it would be a shame, because LISTEN/NOTIFY is a powerful tool, allowing you to use your Postgres database for low-latency durable notifications, streams, and pub/sub. The accusations aren’t wrong: NOTIFY has unintuitive and undocumented performance characteristics arising from its use of a global lock. But “unintuitive behavior” is not the same as “cannot scale.” In this blog post, we’ll show how with the right optimizations, LISTEN/NOTIFY can be used at scale. We'll focus on our implementation of LISTEN/NOTIFY-backed streams, which achieves 60K writes per second on a single Postgres server with millisecond-scale latency.

Low-Latency Streaming with LISTEN/NOTIFY

The basic design of Postgres-backed streams is simple: create a streams table where each stream chunk (for example, an LLM response token) is a new row, then write to streams by inserting into the table.

The tricky part is reading from the stream because you don't know when the next chunk will arrive. One solution is polling: have each reader poll the end of the stream for new chunks. However, polling scales poorly. If the polling interval is set too high, latency is too high for interactive use-cases (e.g., online chats). But if the polling interval is set too low, concurrent pollers overwhelm the database.

The better solution is LISTEN/NOTIFY. This allows readers to block waiting for a notification from a writer that a new chunk has been published to the stream. That way, readers don’t waste resources polling, but wake up immediately when a new stream chunk arrives.

In our initial implementation of LISTEN/NOTIFY-based streams, a trigger on the streams table fired a function that sent a notification every time a new stream chunk was written. Readers waited for these notifications and woke up to a new stream chunk.

This implementation was correct and delivered low latency, but at scale its throughput was poor. Even using a large Postgres database, it could not sustain more than 2.9K stream writes per second. Interestingly, it bottlenecked without visibly consuming any Postgres resource (CPU, memory, or IOPS). As you may have guessed, the root cause was the original “LISTEN/NOTIFY is not scalable” issue: a global lock Postgres takes during NOTIFY. But why does Postgres do that, and how can we optimize it without losing the benefits of Postgres notifications?

The LISTEN/NOTIFY Exclusive Lock

To understand the problem, we’ll need to examine how Postgres LISTEN/NOTIFY actually works.

The root cause of the poor performance is that in Postgres, committing a transaction that calls NOTIFY requires taking a global exclusive lock. This lock is taken as the transaction begins to commit, and is not released until the transaction is fully committed and its contents have been flushed to disk with fsync().

This lock is necessary because Postgres guarantees that notifications are sent in transaction commit order. To enforce this, it stores all outgoing notifications in a global internal queue whose order must exactly match the commit order of the transactions sending those notifications. Adding notifications to this queue must be done transactionally as part of the commit. However, Postgres doesn’t assign transactions a commit order until those transactions are done committing, as committing can take a variable amount of time.

This creates an ordering problem: transactions containing notifications must add themselves to the queue in commit order, but commit order isn’t defined until the commit is complete. The solution is the global lock, which serializes commits of transactions containing notifications, so their commit order is defined ahead of time and they can correctly order themselves in the internal notifications queue.

This exclusive lock explains the poor performance we observed. Because we call NOTIFY from a trigger on the streams table, every stream write includes a call to NOTIFY. In order to commit, each stream write needs to take the global lock and hold it for the entire duration of its commit, including the flush to disk. This means that stream writes need to commit sequentially, precluding Postgres’s usual optimizations like group commit (which commits many transactions together in a single fsync()). As a result, stream writes can complete no faster than Postgres can commit transactions, which leads to this bottleneck. This also explains why we did not see significant consumption of any Postgres resource such as CPU or disk: there wasn’t any, because all transactions were serialized by a global lock.

As an aside, there’s been some online discussion of a Postgres patch related to this issue. This patch (to be released in Postgres 19) does not remove the global lock or fix the bottleneck we observed. Instead, it optimizes the narrower case where there are many notification channels and each listener is waiting only on a specific channel.

Optimizing LISTEN/NOTIFY

To make LISTEN/NOTIFY-backed streams faster, we have to work around this bottleneck. The key observation is that for streams, and for many other applications of LISTEN/NOTIFY, the notifications aren’t themselves a source of truth. Instead, they just ping a reader to check a database table (the real source of truth) for new data. As a result, notifications don’t have to be globally ordered or perfectly durable, so we can optimize NOTIFY by buffering notifications in memory and periodically flushing them in a single batch transaction, significantly reducing contention on the global lock.

Buffering and batching NOTIFYs avoids the bottleneck because the global lock only needs to be taken when the buffer is flushed, not for each individual stream write. This means that individual stream writes can proceed quickly, taking advantage of Postgres optimizations like group commit to obtain high throughput, while the buffer flushes in the background.

Adopting a buffer introduces a new complication, which is that a process crash while notifications are buffered leads to those notifications never being delivered. To solve this issue, we add a fallback to stream readers: in addition to waiting for notifications, they also periodically poll the database to check if the stream was written to without a notification. The frequency of this polling can be low (because it is only a fallback for undelivered notifications), so it does not significantly affect performance.

Benchmarking this optimized solution, we see massively improved performance: in the presence of concurrent readers, we can perform up to 60K stream writes per second (20x more than before) while still obtaining 15-100ms latency. At maximum throughput, Postgres CPU is fully utilized, showing the database is actually saturated instead of bottlenecked on contention.

Learn More

All benchmark code is available on GitHub: github.com/dbos-inc/dbos-postgres-benchmark

If you like building scalable, reliable systems, we’d love to hear from you. At DBOS, our goal is to make Postgres-backed durable execution as simple and performant as possible. Check it out:

• Quickstart: https://docs.dbos.dev/quickstart
• GitHub: https://github.com/dbos-inc
• Discord community: https://discord.gg/eMUHrvbu67

Full article content is not available for inline reading.

Read the original article →

Earlier this year, we ran experiments to test the limits of scaling agents to cooperate toward a goal. Our hypothesis was that this would unlock a new tier of task scale and complexity.

The flagship project was a long-running swarm building a web browser from scratch. It succeeded as a proof of concept, but fell far short of polished software.

That work was deliberately empirical. We started from a blank canvas and hill-climbed toward a stable, effective system. Since then, our goal has been to understand the agent swarm well enough to engineer it deliberately.

To test that progress, we returned to a task the old swarm had struggled with: building SQLite from scratch, in Rust, from nothing but its documentation.

Our initial results have been promising. We ran the old and new swarms on the same task, with the same models and the same time budget, and measured how much of a held-out SQL test suite each could pass.

The new swarm did better in every model configuration. Using Grok 4.5, it reached 80% in four hours, while the old swarm spiraled and had to be paused before its second hour.

We also varied which models did which jobs. In some runs, one model handled everything while in others, a frontier model planned while a fast, inexpensive model carried out the work. Every mix produced similar quality, but the costs varied enormously.

Trees and leaves

Descriptions of large tasks naturally take the shape of trees, with a goal at the root that subdivides recursively into basic units of work. Our swarm has two roles, both organized around that same tree-like decomposition:

• Planner agents, powered by the smartest models, split a goal into pieces and delegate them.
• Worker agents, generally powered by faster and less expensive models, execute those pieces.

The design is a superset of more rigid orchestration systems. Rather than imposing a fixed topology on the problem, the swarm’s shape grows to cover the problem’s contours, and compute and context scale in proportion to the task’s complexity.

We think this is why the design generalizes to tasks as diverse as building a browser, solving math problems, and optimizing GPU kernels. We’ve also used it internally to find and fix vulnerabilities in open-source software, raise test coverage on our own codebase, and generate billions of tokens of synthetic training data.

What the tree does for memory

When a single agent takes on a complete task, it has to walk the entire tree itself, descending to each leaf while holding its ancestors, its current position, and the wider goal in context the whole time.

We think this explains why long-running single agents drift. They can either focus on the work in front of them and lose sight of the bigger picture, or hold the big picture and do a worse job on the piece.

In a swarm, a planner never implements, so its context never fills with low-level detail, and a worker never plans, so it can spend all its context on one narrow piece of work.

We suspect the ability to scale the agent swarm comes from this context efficiency, more than from parallelism itself. That efficiency is present in the swarm at every scale, which is why this decomposition helps agent performance even on moderately sized tasks.

There are echoes of this structure elsewhere. The economist Ronald Coase, asking why firms exist at all, argued that coordination costs grow faster than the work itself, so organizations settle into tiers of bounded units rather than letting everyone talk to everyone.

A version control system for agents

In an earlier post about the swarm, we noted that tools like Git and Cargo rely on coarse locks for concurrency control. This is fine for one developer but unworkable for the volume of work produced by hundreds of concurrent agents.

The browser swarm from earlier this year peaked at roughly 1,000 commits per hour on Git. The new system peaks at around 1,000 commits per second.

To facilitate this rate of activity, we built a new version control system (VCS) from scratch. Throughput was not the only reason to own this layer. Every change in the system passes through the VCS, so it is where collisions first become visible, and several of the coordination mechanisms in the next section are implemented directly inside of it.

Failure modes at 1,000 commits per second

Human engineering teams have standard coordination mechanisms like code review, ownership, standups, and merge queues. Those systems work at human tempo, but at the commit-rate of the swarm, we see failure modes that human teams don’t routinely encounter.

Split-brain design

Two planners, unaware of each other, implement the same concept in different ways in different parts of the codebase.

We fixed this through prompting. Planners make design decisions themselves rather than delegating them, and we require them to ensure that no two delegated subtrees decide the same question.

Contention between planners

A harder form of contention is when two planners know about each other and fight through back-and-forth changes over the same files.

The problem is two pictures of reality, and merge tooling can't fix a disagreement. Instead, we have agents record decisions in shared design docs. Code that depends on a decision carries a compile-checked reference back to its doc. When planners unknowingly contradict each other, a reconciler merges the docs and the references propagate the resolution downstream.

Merge conflicts

Within the swarm, agents constantly collide on the same files. In order to resolve a collision they would have to stop, absorb the other agent's context, and merge around it. Worker agents are bad at this and, in practice, either overwrite the other change or abandon their own.

To fix this, we created a system where a neutral third-party agent intervenes on merge conflicts and resolves them on behalf of all parties. Its only goal is to be impartial and efficient, similar to the way merge queues work in engineering teams.

Megafiles

Some files are particularly popular places for agents to work. Each agent might add only a small amount of code, and no single agent is responsible for keeping the files small.

These “megafiles” choke everything. They’re expensive to transport, diff, and merge, and become the site of constant collisions.

To fix this, we gave worker agents a way to flag bloated files. Once flagged, we block new commits and an outside agent decomposes the overgrown file into smaller modules.

Ossification

Agents have learned, from working in existing codebases with humans in the loop, not to touch core code even when it needs to change.

To fix this, we license intentional breakage. An agent that judges a core change worthwhile can make a focused patch outside its scope and leave a comment explaining why it did it.

The compiler carries the change through the rest of the system, and everything depending on the old design fails to build. Each agent that hits one of those errors finds the comment, reads the reasoning, and updates its own piece of work to match.

Review lenses

In a system that is both long-running and multi-agent, errors accumulate, and the swarm needs a way to correct itself before small mistakes become foundational.

We experimented with many kinds of review lenses, such as giving a review agent the worker's full transcript, or only its output, or nothing but the codebase. We also tried reviewers running on different models, with different training and a different personality.

No single lens catches everything, but decorrelated lenses stack, the way self-driving systems reach above-human reliability without any single perfect component. The compute spent on review is high return, since review is much cheaper than the work it audits. We suspect this stacked review system was a major contributor to the sustained quality of the runs.

Letting agents shape the environment

Stigmergy is the mechanism by which swarm organisms like ants and termites coordinate without direct communication. They shape the environment, and the environment shapes the next organism.

We had encoded rules like “keep notes” and “document decisions” in earlier runs because they seemed obviously good. In retrospect, they were letting agents institutionalize knowledge for their future selves and teammates.

We pushed this further with an experiment in self-authored, shared context we call the Field Guide. It’s a folder owned entirely by the agents, whose index.md is automatically injected into every agent at start. It is the agents’ job to curate what goes into the guide and their only constraint is a line budget.

The underlying logic of the guide is that model weights are frozen, so it’s precisely surprise encounters that are worth capturing so the next agent trajectory is shorter.

The Field Guide is an early experiment with promising results. We’d expect the benefits to be even larger on codebases agents don’t fully own. Training models to write for their successors, where better capture leads to better rewards, is an interesting follow-up area of research.

The SQLite experiment

We instructed the new version of the swarm, equipped with all the improvements described above, to implement the whole of the 835-page SQLite manual in Rust. We withheld the source code, test suites, SQLite binary, and internet access.

To measure progress, we graded against sqllogictest, a test suite from the SQLite project built to check that different database engines return the same results for the same queries. It contains millions of queries with known correct answers, and the grade is the fraction the swarm's database gets right. Progress shows up as a rising curve over the course of a run.

The swarm was never told the suite existed. After each run, we manually reviewed the code and the run itself, checking for cheating and shortcuts, and confirming the system was built out evenly, rather than just in the places where the tests look.

As you read the curves, keep in mind that agents chose their own strategies. Some built broad foundations and scored low for hours before a late spike while others went deep on one area, scored early, then plateaued while filling in the rest. Trends matter more than exact scores at exact moments.

Results across model mixes

We tested four configurations spanning capability and cost:

1. GPT-5.5 as both planner and worker. A strong frontier model throughout.
2. Grok 4.5 as both planner and worker. Our cost-efficient frontier model, as a comparison point.
3. Opus 4.8 as planner and Composer 2.5 as worker. Frontier judgment paired with efficient execution.
4. Fable 5 as planner and Composer 2.5 as worker. To see whether a next-tier planner makes the hybrid more or less worthwhile.

The new harness outperformed the old in every mix.

The Fable 5 hybrid passed about two-thirds of the suite within the first hour. By the four-hour cutoff, the new runs sat between 73% and 85%, while the old runs ranged from 11% to 77%.

The old Grok 4.5 run was paused before its two-hour mark (more below). Every new configuration went on to pass 100% of the suite.

A deep dive into the runs

Starting with the simplest measure of activity, we can see how the rate of commits varied for Grok 4.5 under the old harness versus the new. The old run produced 68,000 commits in its first two hours, roughly 70 times the new run's pace.

One reading is that it was more productive. Another is that most of those commits were busywork (thrash, contention, churn).

The merge conflict data points to the latter interpretation. The old run accumulated more than 70,000 conflicts before we paused it, accelerating rather than stabilizing, while the new run logged fewer than a thousand over its full four hours.

The conflicts concentrated where files grew largest. In the old run, the biggest files kept growing for the entire run and its single hottest file collected 7,771 conflicts, touched by 1,173 different agents. In the new run, the most contested file in the whole codebase saw 47.

The old swarm's biggest coordination failure — split-brain, or planners duplicating each other's work — showed up in the package structure. Rust code is organized into packages called crates, and in a project like this, each crate is roughly one major component.

The old run sprawled to 54 crates, including three separate SQL packages. The new run settled on nine crates early and never added another.

All of this shows up in the final codebase. In the Fable 5 mix, both the old and new swarms ultimately passed the full suite, but the old one needed 64,305 lines of engine code and the new one did it in 9,908. The Opus mix shows the same shape with 19,013 lines at a 97% grade under the old harness, and 4,645 lines at 100% under the new harness.

Model economics

We said at the top that every model mix produced similar quality while the costs varied enormously, from $1,339 for the Opus 4.8 hybrid to $10,565 for GPT-5.5 alone. The token data shows where that difference comes from.

The structure of the spend was consistent across every run, with workers carrying at least 69% of the tokens, and over 90% in most.

But the dollars split differently than the tokens, because planner tokens cost more. In the Opus 4.8 and Composer 2.5 mix, the Opus-as-planner produced a small fraction of the tokens but roughly two-thirds of the cost, while Composer-as-worker handled the vast majority of the tokens for the remaining third of the cost.

Few moments in a large task genuinely require frontier intelligence, such as the original decomposition, the design decisions, and certain trade-offs. Once a frontier planner has collapsed the ambiguity into a detailed, explicit instruction, less expensive models simply have to follow it. This is a huge potential source of cost savings. In the run that used GPT-5.5 for both planners and workers, the workers alone cost $9,373. In the run where Opus 4.8 did the planning and Composer 2.5 did the work, the entire worker fleet cost $411.

Specs as prompts

Each jump in AI capability has raised the level of abstraction at which an engineer can work.

Autocomplete let engineers work one line of code at a time. Early models raised that to a block of code, and agents raised it to a file or a feature.

With swarms, the unit of work becomes the spec.

For that to work, the swarm has to actually follow the spec, which is what much of this post is about. We gave the swarm 835 pages of prose and it came back with a database. What was scarce in this experiment, and what we expect to be scarce in software engineering going forward, is the right description of intent.

Seen this way, the swarm starts to resemble a compiler. A compiler translates source code down to machine code through a series of intermediate steps. The swarm does something similar with intent. Planners parse a goal into task trees, then lower it step by step into executable work. The difference is that a compiler preserves meaning at every step while the swarm is probabilistic at every one. Everything described in this post exists to close that gap.

We invite you to explore the swarm's output. The codebase from the solo Opus 4.8 run is public at github.com/cursor/minisqlite. Based on our initial glance it looks great, but we have not done a deeper manual analysis. Take your own look, and tell us what you find.

Full article content is not available for inline reading.

Read the original article →

We have proof automation now

I've long had a soft spot for dependently-typed languages like Coq Rocq and Lean. They offer the possibility of a type system capable of encoding and enforcing arbitrarily subtle invariants. The sort of thing that, in regular languages, ends up (at best) as a comment, and which quickly gets lost as the size of the team grows. Then you get subtle misunderstandings and components that don't quite fit together. It's often the case that those components have grown to a sufficient size that, when the problem is noticed, aligning either of them is a wearying prospect. Perhaps, say dependent types seductively, you could write those invariants formally and have a machine check them.

The problem has always been that with great type-system power comes great proof effort. I can certainly attest to entire days spent proving really quite simple things. Doing proofs is actually quite fun: it's challenging, interactive, and there's a clear goal. But gosh, does it take a lot of time, especially if, like me, you don't know what you're doing. There's also the periodic, galling experience, at the end of many hours of effort, where you realise that the goal that you're trying to prove is, in fact, false.

That overhead has made programming in dependently-typed languages extremely niche. It has also spurred people to try and automate it away. The attempt I'm passingly familiar with is F*, where the system tries to have an SMT solver automatically discharge the obligations. That certainly works for simple cases, but it's very easy to craft something that causes the SMT solver to go off into space and run for hours, leaving you wondering whether it's ever going to finish.

We now have LLMs which, combined with proof irrelevance, promise to be an extremely capable form of proof automation. With sufficient amounts of automation perhaps you don't need to worry about proof engineering nearly so much. You still need to avoid blowing up the type checker but, in my limited tests, LLMs can avoid that. Potentially, LLMs suddenly make dependent-type systems dramatically more practical. I wanted to play around with this so built a Zstandard decompressor in Lean, mostly because I was also curious about Zstandard.

Zstandard seems like it's winning the competition to replace gzip as the canonical compression utility. It's another LZ77-style compressor, but it offers better entropy coding and a careful design that allows it to achieve very impressive decompression speeds.

The job of an entropy encoder is, given a set of symbols with non-uniform probabilities, to encode a sequence of those symbols using the fewest number of bits. Zstandard uses Huffman trees, but it also has a higher-compression entropy encoder called FSE. FSE is a state machine. There are more states than symbols, and each symbol gets a fraction of the states that mirrors its probability of occurrence in the stream.

The central trick is that, by giving multiple states to more common symbols, the encoder doesn't just pick a symbol: it also picks which of that symbol's states to land in, and that choice carries information forward to the next symbol. That's where the fractional bits of information go. But this entropy encoder is still just table based, and so it runs very quickly.

Lean

Let's talk about Lean! Above I said that it's a dependently-typed language, and that is a concept better articulated in examples than in a complicated definition. Lean is a purely functional language like Haskell, although it has a few properties that make it potentially a lot more convenient as a programming language. Firstly, Lean is strict, while Haskell is lazy.

Next, Lean has some nice helpings of sugar. Its monadic do notation contains for loops and return statements and break statements. If you want to program in an imperative style, you can do so pretty reasonably!

Lastly, Lean has an optimisation where it will make mutating updates to objects as long as their reference count is equal to one. So you can mutate an array in place as efficiently as in an imperative language, as long as you are careful not to have a reference to it someplace else.

I wrote an implementation of the FSE table construction algorithm from the RFC. The RFC contains “test vectors” for it: three sample outputs from given probabilities. Obviously those go into unit tests. But, in Lean, we can also prove universal properties of the function.

These are the subtle assumptions that an optimised decoding inner-loop requires, and things that can only ever be implicit or mere comments in weaker type systems. Proving strong statements like that is part of the 10× effort that the seL4 retrospective described, and a major barrier to the adoption of dependent types in regular software. Several LLMs can do it automatically now in about 20 minutes, and using only a fraction of a $20/month subscription quota. It'll probably be table-stakes next year. Combining dependent types and LLMs is not a new idea, but not much has been done on applying the combination to quotidian software engineering. Still, proof automation is here now and we, practically speaking, have a new type of programming language available to us. That's exciting!

Aside: verified assembly

AWS made LNSym: a semantics and simulator for AArch64. That's cool. Perhaps we could use it to show equivalence between an optimised assembly implementation of some functions, and their Lean counterparts, and then use the assembly code at run-time? Then we could let LLMs rip at optimisation and they couldn't introduce any functional bugs. Verified assembly is well-trodden in crypto implementations, but perhaps now it could be cheap?

The new rules of context engineering for Claude 5 generation models

We removed over 80% of Claude Code's system prompt for more advanced models. How to apply the lessons we learned to your own context engineering in Claude Code and with your own agents.

I’ve written previously about how to best prompt the newest generation of Claude 5 models and work with them iteratively to discover what you want to build.

But when you send a message to Claude, the prompt is only a small part of the context it gets. Much of your context is assembled from your system prompt, Skills, CLAUDE.md files, memory, and other sources. We call this context engineering, and it makes a big impact on the results you generate when using Claude Code or in building your own agents.

Unlike a prompt, context is used generally across many requests, so it cannot be as specific. How do you build these general prompts and guidance for Claude, especially when you don’t know what a user’s prompt might be?

This can be surprisingly difficult as Claude’s own capabilities evolve. Most recently, we noticed a large jump in the way we prompt the newest generation of Claude models. We removed over 80% of Claude Code’s system prompt for models like Claude Opus 5 and Claude Fable 5 with no measurable loss on our coding evaluations.

Here’s what we’ve learned about prompting this new class of models, and how you can utilize it to update your context engineering. We’ve put these best practices in claude doctor; use the command /doctor in Claude Code to rightsize your skills, and CLAUDE.md files.

Unhobbling Claude

Overall, we found that we were overconstraining Claude Code, both through our system prompt and in our CLAUDE.md files and skills.

For example, when we read transcripts of our own internal usage of Claude Code, we see several conflicting messages in a single request like “leave documentation as appropriate,” or “DO NOT add comments” as our system prompt, skills, and user requests clash with each other.

Generally, Claude can interpret the user’s intent to get to the right answer, but Claude must think more carefully about these overlapping and conflicting messages before deciding what to do.

And while these constraints were once needed to avoid worst case scenarios, we have since found we can delete many of them and let the model use surrounding context and judgement instead.

Additionally, Claude Code now has many more tools. Claude used to rely on CLAUDE.md as a source of memory, information, and guidance. Now we have memory, artifacts, and skills, which Claude can use to create new ways of loading and sharing context across sessions.

Then and now

There were a number of previous context engineering best practices that had become myths. Including:

Then: Give Claude rules

Now: Let Claude use judgement

When we first rolled out Claude Code, we needed to be sure that Claude avoided worst case scenarios, such as deleting files. This meant we would give particularly strong guidance that might not always be true, For example, in the system prompt we used to say:

In code: default to writing no comments. Never write multi-paragraph docstrings or multi-line comment blocks — one short line max. Don't create planning, decision, or analysis documents unless the user asks for them — work from conversation context, not intermediate files.

But for a certain subset of prompts, this guidance would be wrong. In the case of documentation, the user may have their own preferences, or specific parts of very complex code might need multi-line comment blocks.

Still, without these guardrails for older models, the comments Claude wrote would be incorrect in many cases and we had to accept this tradeoff. But newer models have better judgement and can handle these decisions well without explicit rules.

In the new system prompt we say: Write code that reads like the surrounding code: match its comment density, naming, and idiom.

Then: Give Claude examples

Now: Design interfaces

The number one rule for tool usage was to give Claude examples on how to use them. With our newest models, we’ve found that giving examples actually constrains them to a certain exploration space.

Instead of using examples, think more about the design of your tools, scripts and files- what parameters does Claude have and how can they be more expressive?

For example, in the Todo tool example, just listing status as an enumeration between pending, in_progress, and completed, hints to Claude about how to use it. The instruction on keeping one item in_progress helps define our requested behavior.

Then: Put it all upfront

Now: Use progressive disclosure

Because Claude Code was focused on coding, our system prompt included detailed information on how to do code review and verification. These were not always needed, but when they were, it was crucial information.

Since then, Claude Code has gotten very competent at using progressive disclosure- loading the right context at the right times. For example, we moved verification and code review into their own skills that Claude Code could selectively call.

But progressive disclosure is not just for skills, we also use it for tools. Some of our tools are ‘deferred loading,’ which means the agent must search for their full definitions using ToolSearch before using them. This allows us to have more tools (such as our Task tools) that don’t take up context until they’re needed.

The same can be applied to your own CLAUDE.md and Skill.md files. A common myth is that you want to make these a central repository for every known practice that you might run into, because Claude would not find it otherwise. Instead, consider having a tree of files that can be loaded at the right time.

Then: Repeat yourself

Now: Simple tool descriptions

Earlier Claude models could sometimes need repeated instructions or be more likely to listen to instructions at the end of their context window than at the start. This meant our system prompt would sometimes have references to tools in the main system prompt as well as instructions in the tool description.

We found we could delete these repeat examples and put instructions on how to use tools in the tool descriptions rather than the system prompt.

Then: Memory in CLAUDE.md files

Now: Auto-memory

We used to encourage users to save things to Claude’s memory, by using the # hotkey to write to their CLAUDE.md automatically. Instead, Claude now automatically saves memories that are relevant to the work and to you.

Then: Simple specs

Now: Rich references

In plan mode, Claude Code has heavily relied on markdown files with plans. Storing these files as plans helped Claude refer to them when needed. Another similar best practice was to store specs in the codebase for Claude to refer to while working across longer projects.

But we’ve found that Claude can handle increasingly more complicated references. Instead of simple markdown files, Claude can reference HTML artifacts created by our new artifacts feature.

You may also give Claude references in the form of code. A spec may also be a detailed test suite, or a function in a different codebase that Claude might port.

Rubrics are another form of references. Rubrics allow Claude to try and verify your taste in a particular field (e.g. what does a good API design look like) by using dynamic workflows and spinning up verifier agents with those rubrics.

Applying this to your context

System Prompt

A system prompt is heavily tied to the product context. It tells Claude what product it’s operating in and what it’s doing. For Claude Code, you will likely never modify this, but if you are building your own agent harness, this is where you should spend a lot of time.

CLAUDE.md

Keep your CLAUDE.md lightweight and briefly describe what your repo is for, but spend most of the tokens on gotchas inside of the codebase. For example, you may organize your code to keep types in one monolithic file and nowhere else. Avoid stating ‘the obvious’ things Claude should know by looking at your file system or your repo.

Use progressive disclosure heavily, for example if you have several unique instructions on how to verify your work, create a verification skill and reference it from your CLAUDE.md.

Skills

Think of skills as lightweight guides to let Claude find information when needed. Avoid making them overconstrained, except in highly important areas.

For long skills, try and use progressive disclosure as much as possible- divide it into many files and split them out.

It’s best when skills encode particular opinions, knowledge, or best practices that are particular to you, your team, or product.

References

You can @ mention files to include them as references. References allow Claude to refer to in-depth information about the current plan.

This might be in specs files, mockups, or even entire codebases. Generally you should prefer files that are in code as it provides clear, high-fidelity instructions to Claude in a language it knows very well. For example, a HTML mockup of a design will generally produce better results than a description of the design or a screenshot.

Try simplifying

Across your system prompt, skills, and CLAUDE.md files, you may need to simplify just like we did. We rolled out a new command called claude doctor, which will help you do this automatically as well. For more details on prompting more advanced models specifically, check out our Fable field guide.

This article was written by Thariq Shihipar, member of technical staff, Anthropic.

Prompt Caching In Agents

Large language models are often thought of like functions: send in some text, receive some text. That is a useful abstraction, but it ignores one of the most important parts of running a coding agent: most of the input is the same as last time. In other words we mostly append to it.

A coding agent sends the model its system prompt, tool definitions, project instructions, conversation history, tool calls, and tool results. On the next turn it sends almost all of that again, plus a small amount of new material. Once a session has grown to tens or hundreds of thousands of tokens, recomputing the whole prompt for every turn is slow and expensive.

Prompt caching is what makes this somewhat economic, but it is also quite fragile. A changed tool definition, a model switch or a provider routing decision can turn what one would expect to be a cheap incremental request into a full replay of the context.

For coding agents, cache behavior is therefore not just an implementation detail or optimization. It affects latency, cost, tool design, session design, and even which product features should be made available.

What a KV Cache Contains

A transformer processes a prompt in two broad phases. During prefill, it reads the input tokens and computes attention state for them. During decode, it produces new tokens one at a time.

At each attention layer, every processed token produces a key and a value. These are not quite like key-value lookups in a hash table: both are arrays of numbers, usually floats or lower-precision quantized values. When processing a new token, the model compares that token's query with the earlier keys to determine how relevant each earlier token is. It then uses those relevance scores to form a weighted mixture of the corresponding values. In that sense, a key is what the model matches against, while a value is the information it retrieves (but the lookup is fuzzy rather than "returning a single exact match" like a dictionary lookup.)

Those keys and values are retained so that the next generated token can attend to everything that came before without recomputing the earlier tokens. This retained state is the KV cache.

Conceptually, a request looks like this:

request 1:

[system][tools][user][assistant][tool result][user]
<--------------------- prefill -------------------->
                       |
                       K and V tensors per token and layer

request 2:

[system][tools][user][assistant][tool result][user][new]
<---------------- reusable prefix ----------------><--->
                                                    |
                                                    new work

The real representations are more complicated, model-specific, and "quite" large. The important property is that they correspond to a particular token prefix. Two prompts that mean the same thing but tokenize differently do not share a KV cache. If a token changes in the middle, everything after that token is a different continuation.

Prompt caching extends the lifetime of this state beyond one generation. When the next API request from the coding agent begins with the same tokens, the inference system can reuse the stored work for the matching prefix and prefill only the new suffix. So far, the theory.

Where the Cache Lives

In order for a cache to work it needs to be stored somewhere, and it needs to be addressable. There are two broad ways inference systems make KV caches available to a later request.

The simpler approach is session affinity. It works by keeping the KV cache on or near the GPU that computed it, and routing the next request back to the same worker. A session ID or prompt-cache key becomes a trivial routing hint and so you can potentially even deal with this problem on the HTTP load balancer level without having to look into the payload.

request(session-42) --> router --> worker 7 --> GPU 7 KV cache
next(session-42)    --> router --> worker 7 --> GPU 7 KV cache

This avoids moving a very large cache over the network. It is fast when it works, but it constrains scheduling. The selected worker can become overloaded, restart, or evict the entry. A router may also decide that balancing the fleet is more important than preserving one session's cache. It is however a very attractive solution because it works with little extra deployed infrastructure and hardware.

The other approach is to distribute the cache. KV blocks can be stored in another memory tier or made available across workers, so a request is not tied as tightly to one GPU.

                         +--------------------+
request --> scheduler -->| worker 3 / GPU 3   |
             |           +--------------------+
             |
             +----------> distributed KV blocks
             |
             +----------> worker 9 / GPU 9

That improves scheduling flexibility and recovery, but moving, indexing, and retaining KV blocks is itself a systems problem. Implementations mix GPU memory, host memory, local storage, remote storage, prefix-aware routing, and eviction policies in different ways.

To put KV caches into perspective: they can be large but they are in some ways smaller than one would assume. With various tricks, the size of KV caches can be reduced to a handful of gigabytes, even for long conversations.

Caches and Prefixes

Pi sessions are trees, not lists. /tree can move the active conversation back to an earlier point and continue along another branch. A rewind can discard the active suffix without deleting it from the session file. A new branch can share most of the old context, a little of it, or effectively none of it. This design is not unique to Pi, quite a few coding agents have something at least conceptually similar. Even if you do not represent the session as a tree, it's not uncommon for agents to have some form of rewinding.

                             +-- E -- F  another branch
                             |
session S: root -- A -- B -- C -- D  current branch
                   |
                   +-- Z  branch near the start

All three branches can have the same Pi session ID. From the router's perspective they are one session. From the prompt cache's perspective they are three token sequences with only partial prefix overlap.

If the cache keeps reusable prefix blocks, jumping from D to F may still reuse root -> C. If it only retains the hottest continuation, if the shared blocks were evicted, or if the request is routed elsewhere, the hit can be much smaller. Jumping to Z may preserve only the system prompt and initial tool definitions even though it starts from A. The precise cache management behavior here depends greatly on the providers.

The reverse can also happen. /fork or a new session can produce a new session ID while carrying over a large amount of identical context. A routing system that isolates caches by session key may fail to notice that useful overlap.

The reusable prefix determines what work can be cached. Session identity merely helps the infrastructure find likely content. On some systems the routing key is crucial to manage caches, on others it's merely an optimization.

Explicit vs Automatic Prefix Caching

Provider APIs expose caching in two main styles.

Anthropic's traditional interface uses explicit cache_control points. The client marks boundaries after stable parts of the request, such as the system prompt, tool definitions, or the latest cacheable conversation content. The server can then write or look up the prefix ending at those points. The boundary is explicit, but reuse still requires the content before it to match. Not only are the cache points explicit, so is the pricing. You pay for cache writes, and you get to choose for how long which comes at different price points.

Other APIs use automatic prefix caching. The client sends the request normally, and the provider finds a reusable prefix without client-placed breakpoints. A prompt-cache key or session header may improve routing or grouping, but it does not make different prefixes equal.

Why Tool Loadouts Trash Caches

Tool definitions usually appear before the conversation and they are "folded" into the system prompt internally. Their names, descriptions, and JSON schemas are model input just like any other text. Adding one tool, removing one, changing its schema, or even serializing the tools in a different order can move the first mismatch close to the start of the prompt.

turn 1: [system][read][write][bash][conversation...........]
turn 2: [system][read][write][bash][deploy][conversation...]
                                   |
                                   old conversation is now
                                   after a mismatch

This is a common surprise with plugin systems and MCP-style tool catalogs. Loading a tool only when it becomes relevant sounds efficient because fewer schemas are sent initially. On most models, however, the newly expanded loadout invalidates the cached conversation that follows it. Saving a few tool-schema tokens can cause tens of thousands of conversation tokens to be processed again.

Some newer model APIs support additive tool loading. A tool can become available at a specific tool result inside the transcript instead of being inserted into the original tool list. The old prefix remains unchanged:

[system][initial tools][conversation][new tool][next turn]
<--------- cached prefix ----------->

Pi nowadays supports this for models with native deferred-tool mechanisms. When an extension makes a purely additive change with setActiveTools(), Pi records the added names on the tool result. For supported Anthropic models it uses deferred definitions and a tool_reference and for supported OpenAI models it emits the corresponding tool-search items. Other models get a safe fallback: Pi sends the complete active tool list on the next request, which works functionally but may wipe the prompt cache.

The word additive matters as removing tools, replacing one loadout with another, or changing prompt snippets still changes earlier input. An extension that rebuilds the system prompt, shuffles tool order, injects timestamps, or changes active tools every turn can accidentally defeat caching for the entire session.

Extensibility means Pi cannot guarantee cache stability on behalf of every extension. We can provide cache-friendly mechanisms; extensions still have to use them and from what we have seen, for many extensions cache efficiency is an afterthought. This is, in part, because when you pay on a fixed subscription the associated cost with cache misses is not quite as obvious.

Interruptions and TTLs

Some important prompt caches have short default lifetimes. Anthropic's default five-minute cache is particularly important because it is shorter than many normal coding activities. If you go to sip a coffee when using Fable, and you come back 10 minutes later, a single "say hi" message will cost you more money than you expect.

That's because while the user may think of a coding session as continuously active, the inference provider sees a sequence of isolated requests:

model request --> run tests for 7 minutes --> model request
                  no cache traffic here

A long build, a test suite, lunch, a meeting, or simply stopping to review a diff can outlive the cache. The next request contains the same prompt, but the stored KV state is gone and the prefix is billed again as input.

Since Pi is currently not a permitted harness on Anthropic's subscription we're following the 5 minute default that Anthropic recommends for API users. However from looking at Claude Code's codebase we know that for their own subscription users, they are increasing that cache timeout to one hour. The increased cost of this however often is not worth it, when you need to pay API token prices.

But you can opt into this. Some providers such as Anthropic expose longer retention controls. For supported direct APIs, Pi users can set PI_CACHE_RETENTION=long to request them. That is still only a request: Pi cannot force a gateway to retain an entry, prevent eviction under memory pressure, or keep a cache alive while no model request is being made.

The Price of a Miss

Providers usually price uncached input, cache writes, and cache reads differently. Cache reads are commonly discounted because the expensive prefill work has already happened. Cache writes can carry a premium because the provider is promising to retain state for later use.

Imagine a coding session with 100,000 tokens of history followed by a short new request as the Fable example above. When the cache works, almost all of that history is charged at the lower cache-read price. Only the small amount of new material needs to be processed at the regular input price and potentially written to the cache.

When the cache misses, the provider has to process the entire 100,000-token history again at the regular input price. It may also charge to write that history back into the cache. This is why a short request such as continue can be surprisingly expensive after a cache expires. In a long coding session, re-reading the old input can cost much more than generating the next answer.

Caching also has a chance to create non-obvious incentives.

The user should want high hit rates because they reduce latency and price. An inference operator that owns the GPUs should want them too: less prefill work means more requests served with the same hardware. A well-designed cached-token discount can align both sides while leaving the operator with better margins.

A gateway or reseller can have a different incentive. If it earns revenue from input tokens billed at the uncached rate, a cache miss can produce a larger customer invoice than a hit. Whether that also produces more profit depends on its upstream costs, contracts, and who operates the cache. In a badly aligned stack, the party responsible for routing may not bear the full cost of misses, while the party billing the user earns more revenue when they happen.

That does not mean providers sabotage caches but it means cache performance should be observable. Users should not have to infer that only from a surprisingly large bill. Understanding if something odd is going on with caches can be an important insight.

Strict cache adherence also means less flexibility for a gateway to route you to the best option in-between turns. You might want to take a cache hit to continue with a different model which from that point onwards might be more economical, or it might be the case that you might be better off load balancing to another provider.

Why Pi Does Not Prune Aggressively

Now that you've made it this far, you probably have an idea why Pi does not prune tool calls. It is tempting to control cost by continuously deleting old tool results or rewriting history and sometimes that is necessary, especially near the context-window limit. But as we have learned, pruning has a cache cost of its own.

Deleting content from the middle changes the prefix at the deletion point. All surviving conversation after it may need to be processed again. The immediate cost of rewriting a long cached context can exceed the future savings from removing a small number of cheaply cached tokens.

A rough break-even comparison is:

one-time rewrite cost
    ~= surviving tokens after the edit * (uncached price - cache-read price)

future savings per turn
    ~= pruned tokens * cache-read price

This is not only an accounting question as old tool results often contain the evidence the model used to make later decisions. Removing them can degrade behavior even if a summary preserves the gist.

Pi therefore prefers a stable, append-oriented transcript and does not treat every old token as waste. Compaction is available when context pressure justifies a lossy rewrite. Because compaction deliberately creates new context rather than accidentally re-billing an unchanged prompt, Pi treats it as a cache reset rather than a cache failure in its session statistics.

The goal is not the smallest possible prompt but the best trade-off among model context, cache reuse, latency, and price.

Simultaneously there can be a case for pruning too. If you are working with providers that do not discount you for good cache usage, or it's for whatever reason not possible to get high cache rates, it might be preferable to prune. It definitely improves the opportunity for the router to balance between different backends as caches are not transferable.

What Pi Can and Cannot Do

Pi works to keep stable inputs stable. It passes a consistent session ID and provider-specific cache hints, places explicit cache points for APIs that require them, records cache-read and cache-write usage, and supports message-anchored additive tool loading where models allow it. Its default transcript behavior also avoids gratuitously rewriting old context.

Pi cannot control every layer after the request leaves the machine. It cannot choose a provider's eviction policy, extend a cache beyond what the API permits, keep a particular GPU alive, or guarantee that a gateway honors affinity. It also cannot preserve a prefix that an extension changes.

What it can do is make cache health visible.

The interactive footer shows cumulative cache reads and writes as R and W, plus CH for the latest request's cache-hit rate. The /session command gives a fuller view: total cached and uncached input, cumulative hit rate, cost, and an estimate of tokens and dollars re-billed by significant cache misses.

Messages
Total: 178
User: 6
Assistant: 58
Tools: 114 calls, 114 results

Tokens
Input: 7,129,883
  Cached: 6,776,832 (95.0%)
  Uncached: 353,051
Output: 30,013
Total: 7,159,896

Cost
Total: $6.054
Cache Re-billed: $0.728 (161,744 tokens, 2 misses)

Users who want misses called out as they happen can enable Show cache miss notices in /settings, corresponding to showCacheMissNotices in settings.json. Pi then inserts a warning after a significant miss, including the estimated re-billed tokens and cost. When it can observe a model switch or an idle gap beyond the usual short TTL, it says so. For other misses it reports the fact without pretending to know what happened inside the provider.

Common Reasons for Worse Cache Performance

When a session's cache-hit rate looks wrong, the usual causes are:

1. Idling. A command, review, or conversation pause exceeds the provider's retention window.
2. Model or provider switches. KV state is model-specific and generally does not move across providers.
3. Branch navigation. /tree, rewinds, forks, and alternate branches can change the active token sequence even when the session ID remains the same.
4. Compaction or manual history rewriting. These intentionally replace part of the prompt and establish a new prefix.
5. Tool and reasoning level changes. Adding, removing, reordering, or editing tool definitions changes an early part of the request unless the model supports message-anchored loading and the change is purely additive. Reasoning level changes usually have the same effect.
6. Dynamic system prompts. Timestamps, random values, changing project context, and extension-provided prompt snippets can invalidate everything after them.
7. Extension context transforms. An extension that modifies old messages or provider payloads can make an apparently stable Pi transcript unstable on the wire.
8. Provider routing and eviction. The prompt can be identical and still miss because the relevant KV blocks are no longer available where the request lands.

SANA-Video 2.0

Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

Abstract

We introduce SANA-Video 2.0, a hybrid video diffusion transformer instantiated at 5B and 14B scales under a unified architecture. Designed to generate high-quality video up to 720p on a single GPU, SANA-Video 2.0 matches full-softmax video DiTs in quality while retaining the favorable long-sequence scaling of linear attention. To avoid quadratic attention throughout, Hybrid Linear-Softmax Attention combines gated linear attention for O(N)-dominated mixing with periodic gated-softmax anchors at a 3:1 ratio, restoring the full-rank token interactions that pure linear attention lacks. To propagate these refreshed representations across depth, Block Attention Residuals (AttnRes) route completed block summaries into later linear layers, enabling anchor-feature reuse and boosting deep-layer effective rank by ~12%. Through from-scratch training, SANA-Video 2.0 learns the complete hybrid directly rather than linearizing pretrained models, with reduced-resolution proxy studies establishing 25% softmax as the optimal quality-efficiency trade-off. With 40-step sampling, SANA-Video 2.0 achieves a VBench score of 84.30 in 13.2s at 480p on a single H100, remaining competitive with far larger softmax video DiTs at a fraction of the latency. Its compiled DiT forward pass is 3.2× faster than a matched full-softmax baseline at 720p/60s, a gap that expands with video duration. Furthermore, full-stack Sol-Engine optimization (kernel fusion, caching, and sparse attention) accelerates this hardware-friendly backbone by a further 3.58×, bringing the 5B pipeline to 13.06s at 720p/5s and making it 120× faster than Wan 2.2-A14B on one H100. Overall, our hybrid design recovers softmax-level expressiveness at substantially reduced cost, unlocking scalable long, high resolution video generation.

@misc{chen2026sanavideo20hybridlinear,
  title         = {SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation},
  author        = {Junsong Chen and Jincheng Yu and Yitong Li and Shuchen Xue and Haozhe Liu and Jingyu Xin and Yuyang Zhao and Tian Ye and Zhangjie Wu and Zian Wang and Daquan Zhou and Ping Luo and Song Han and Enze Xie},
  year          = {2026},
  eprint        = {2607.21553},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2607.21553},
  url           = {https://arxiv.org/abs/2607.21553}
}

Introducing Classifiers, now in beta.

Tag inference in your workspace by task type, department, agent complexity, or whatever you care about. Then filter your logs or visualize your Activity.

Here's our internal model spread across task types.

A classifier is four things: a taxonomy of up to 8 dimensions, a classification prompt, a model to apply it, and a sampling rate.

We include six presets you can use to start quickly. Classification runs async after each request, so it adds zero latency to your inference path.

Here's our own internal inference, grouped by task family and model.

Analysis and coding dominate, and you can see exactly which model is absorbing each one.

Invert the grouping to e.g. see where GPT 5.6 Sol is being used the most.

Classifiers can work even with prompt logging disabled, and attach on each individual generation for full inspectability.

Create one in your workspace, or read the docs: openrouter.ai/docs/guides/features/classifiers

More info in our blog: openrouter.ai/blog/announcements/classifiers/

A couple of decades after the discovery of systems that could selectively target DNA, we’re starting to see the first therapies based on gene editing. One challenge these developments have faced is safety. While we can make them pretty specific to the gene we want edited, the human genome is very large, and even rare DNA sequences can appear a couple of times by chance.

As a result, all the original gene-editing systems had known rates of what are called off-target effects, in which they simply edit the wrong sequence. This may be a low-probability event, but edit enough cells—and therapies generally have to edit many—and errors become inevitable.

A lot of effort has gone into finding ways to minimize or eliminate off-target edits. In a recent issue of Nature, researchers described modifying the AI protein-folding software AlphaFold to help identify key areas of gene-editing proteins responsible for off-target effects. Those areas were then modified to reduce the problems.

Gene editing and off-target effects

Gene-editing systems have three key components. The first is guide RNA, which can base-pair with the targeted genome sequence. It’s possible to design many guide RNAs that all target the same gene, so one approach to making the system safer is to pick sequences that don’t share much similarity with any other genomic locations. This is now widely used as part of the basic design process for a gene-editing system.

The second component is a Cas protein named after the CRISPR system’s Cas9 protein. It interacts with both the guide RNA and genomic RNA and helps enforce the specificity of the interactions.

If there are too many mismatches in the base pairing between the RNA and DNA, Cas9 (or other Cas family members) won’t stick there. Various approaches have led to improved Cas family members that have reduced the tendency to enable off-target edits.

The final key component of the system is the protein that interacts with Cas9 when it’s bound to DNA and modifies the DNA. In the original CRISPR system, this cut both strands of the double helix, producing damage that’s difficult to control. Researchers have since modified other proteins to interact with Cas9 but catalyze more subtle changes to DNA, such as lopping off a single base or making chemical modifications that alter how it base-pairs.

Overall, the length of the base pairing between the guide RNAs and the genome is on the order of 18 bases long. That should show up in random DNA sequences only about once in 70 billion bases, and our genome is only about 3 billion bases. By that measure, we should be good. But it turns out that Cas9 can tolerate a small number of mispaired bases without losing its ability to stick to DNA. The exact number and location of the bases where variations are tolerated can vary somewhat, making it difficult to identify in advance which guide RNAs might pose a danger.

One route to improving safety is to better understand how these off-target interactions occur.

Making contact

The team behind the new work, based at a variety of institutions in China, reasoned out their approach in advance. A perfectly matched DNA-RNA hybrid will have one structure, while one with one or more mispaired bases will have a slightly different structure. Evolution has optimized the structure of the Cas9 to stick to the former. But it apparently hasn’t prevented Cas9 from adopting slightly different conformations that can interact with mispaired structures.

If we can identify the portions of Cas9 that mediate these problematic interactions, we can modify and potentially block them.

The team’s first step was to build a large library of off-target editing sites. They did this by using a modified CRISPR system that converts the DNA base adenine to a related chemical, inosine, and then isolating any DNA fragments that contain it. They repeated this process with 10 different guide RNAs and analyzed a large number of modified DNA fragments from each to get a broad picture of the types of off-target sequences present.

The next step was to examine how the CRISPR complex interacted with them, using the AlphaFold AI-based protein-folding software. Updated versions were designed to handle interactions between proteins and nucleic acids, as well as complexes of multiple proteins. So the team fed AlphaFold versions of a target DNA sequence, along with a guide RNA, the Cas9 sequence, and an enzyme that chemically modifies bases and can stick to Cas9.

Unfortunately, it choked, placing one of the proteins in what was clearly the wrong location.

Undeterred, the team simplified things and fed AlphaFold only the DNA, RNA, and Cas9 protein, since the latter is the primary factor determining its sequence specificity. This worked much better, producing a structure that agreed with ones determined by experiments with actual nucleic acids and proteins.

By comparing the structures AlphaFold generated when fed different on- and off-target sites, the researchers found a general pattern. Many (about two-thirds) of the off-target sites caused the Cas9 protein to adopt a slightly different structure. But nearly all (over 95 percent) of them altered which amino acids contacted the RNA. So there are clearly some cases where Cas9 maintains its normal structure but amino acids within it flex around in ways that accommodate the mispaired bases of off-target sites.

Conveniently, AlphaFold was already set up to identify what is termed the “contact probability,” namely, the chance that any two items, such as amino acids or nucleotides, are within a very small distance (eight Angstroms). The researchers could take the output of the contact probability analysis for on- and off-target sites and compare them, identifying exactly which amino acids in Cas9 have altered contacts when there’s a mismatch between the guide RNA and the DNA. They termed this computerized analysis setup “ContactSeek.”

Better targeting

On its own, ContactSeek tended to produce a large list of amino acids that shift around when bound to an off-target site. So the researchers focused on regions of the Cas9 protein where these amino acids clustered, viewing this as a sign that these areas were adapting to the differences caused by mismatched bases. They then began to test versions of Cas9 with different amino acids at these sites.

In all, the researchers made 23 different swaps, putting a different amino acid into one of 10 key positions identified by their work with AlphaFold. This allowed them to find a variant with activity similar to the normal Cas9 at sites that matched the target sequence, while its off-target activity dropped from 28 percent to 5 percent. Similar results were possible when the researchers used different guide RNAs. They also showed that the approach worked for a similar system that used a different Cas protein (Cas12) to recognize the DNA/guide RNA combination.

As mentioned above, other research teams have used different approaches, such as directed evolution, to develop Cas9 variants that are less prone to off-target editing. When tested against these variants, the newly designed versions tended to produce similar or even slightly better activity and specificity. The big difference here is that the changes identified by this approach may be somewhat more specific to a given guide RNA/mismatch combination rather than more generally effective.

Of course, some of the individual changes identified here could potentially be combined with those in the previously developed Cas9 versions for even greater improvements, although that wasn’t tested here.

In any case, the work is potentially useful because it describes a general method for producing gene-editing systems tailored to prevent known off-target events. If that ends up being a bottleneck in developing a therapy, this work could be a significant breakthrough. But the researchers also suggest that the approach would be useful more generally for fine-tuning protein-DNA interactions, which could have applications far beyond gene editing.

Nature, 2026. DOI: 10.1038/s41586-026-10794-z.

Overview

The Isochrones API calculates the area that a person can reach from a specific origin point within a set amount of time. Unlike a radius, which measures a straight-line distance, an isochrone accounts for actual road network travel times to create a polygon that represents true reachability.

Common use cases include:

• Delivery logistics: Determine which customers are servable within 30 minutes of a warehouse.
• Real estate: Find properties within a 45-minute commute of a workplace.
• Urban planning: Analyze the accessibility of essential services like hospitals or grocery stores.

To get started, see Set up the Isochrones API.

What will more intelligence actually do for us?

A lot, actually. But it won't look quite like what humans do with our intelligence.

“[I]ntelligence, that counterentropic conjoined twin of information, must become the most powerful force in the universe, the energy to which all other physical laws must eventually kneel…Intelligence was destiny, manifest.” — Ian McDonald, “Verthandi’s Ring”

Not a lot of people expected that AI would come for the mathematicians before it came for the truck drivers, but it did. An AI model disproved the Jacobian Conjecture — an 87-year-old open problem that human mathematicians had struggled to solve. The greatest living human mathematician, Terence Tao, turned to AI to help him understand the solution. Around the same time, AI solved a very important open question in quantum cryptography. Solving Erdos problems has now become almost child’s play for the best AI models. And this is the worst AI will ever be at math. Model capabilities, and the amount of compute available, both continue to increase at rapid rates.

I don’t expect mathematicians to actually lose their jobs en masse, of course. But it’s becoming clearer and clearer that humankind has invented machines that are smarter than we are. Intelligence isn’t defined for machines the same way it is for humans — AI’s capabilities are spiky in different ways than ours — but it’s undeniable that the technology is improving rapidly in every domain of cognitive capability. It’s still possible to find some mental tasks that humans are better than machines at, but those final advantages tend to disappear almost as quickly as we can identify them. “AGI”, or “ASI”, or whatever you want to call it, is certainly here.

And yet…the world remains much the same. In lots of sci-fi books, as soon as artificial superintelligence arrives, it bootstraps itself to even more godlike intelligence in an explosive “singularity” that rapidly transforms the entire physical universe. Lots of people, especially “AI safety” and “effective altruist” types, expected things to play out basically the same way in reality. But looking around, not much has changed since we entered the intelligence explosion. There’s a huge data center boom, and most people use AI on a daily basis, but we still live basically the same lives — driving to work or taking the train, sitting in front of a computer, scrolling on our phones, collecting a paycheck. People are staying in their jobs longer, but employment hasn’t been disrupted in a significant way.

Meanwhile, we’ve had decently robust productivity growth, but nothing really amazing. A lot of people I know are surprised by this. Ruxandra Teslo writes:

Walking around the world today one might notice that it is weirdly unchanged…To many, this is surprising. Just the other day I was at a conference where someone remarked that if he could have seen today’s AI capabilities a few years ago, he would have been astonished — and would have assumed the world by now would look far more transformed, with much higher GDP growth.

And Clifford Sosin writes:

Superintelligence arrived. You probably didn’t notice, because it turned out to be kind of incremental…Don’t believe me? Run the test. Talk to Fable 5 for an hour, then talk to your ten smartest friends. Which one is smarter? Don’t worry, they won’t be offended…We were told to expect something bigger. Once machines crossed some line, the system’s IQ would climb to heights we couldn’t follow, and we’d be sharing the planet with something that designs warp drives and thinks thoughts as far past us as mine are past my dog…What we got is a tool that writes excellent code, beats people at a startling range of tasks, and will clearly reshape the economy. It’s also, somehow, incremental. No takeoff. No explosion. That’s the strange part.

Teslo blames bottlenecks — governance and other “frictions” — for the slow economic impact. But some others are advancing a more radical hypothesis — that intelligence itself is subject to diminishing returns.

One of these is Francois Chollet, an AI researcher who specializes in measuring AI’s capabilities. In a highly controversial series of tweets back in March, he conjectured that intelligence might be subject to diminishing returns:

One of the biggest misconceptions people have about intelligence is seeing it as some kind of unbounded scalar stat, like height. "Future AI will have 10,000 IQ", that sort of thing. Intelligence is a conversion ratio, with an optimality bound. Increasing intelligence is not so much like "making the tower taller", it's more like "making the ball rounder". At some point it's already pretty damn spherical and any improvement is marginal.

Now of course smart humans aren't quite at the optimal bound yet on an individual level, and machines will have many advantages besides intelligence -- mostly the removal of biological bottlenecks: greater processing speed, unlimited working memory, unlimited memory with perfect recall... but these are mostly things humans can also access through externalized cognitive tools.

In fact, this is a possibility I myself had raised in a post a year earlier:

It seems possible that humans are simply incredibly specialized in a few types of cognitive tasks — extracting patterns from sparse data, synthesizing various patterns into “intuition” and “judgement”, and communicating those patterns in language — and that we’ve basically approached the theoretical maximum in those narrow areas…That would explain why AI has gotten much better at things like math and coding and forecasting over the last year, but why the basic chatbot interface doesn’t seem much more “intelligent”. It would also explain why when you talk to Terence Tao about math, it’s like talking to a superhuman, but when you talk to him about where to get lunch or which movies are the best, he’ll just sound like a fairly smart normal dude. AI will eventually get better than Tao at math…but it may never get much better than the most thoughtful, eloquent humans at deciding where to get lunch or recommending movies. It may simply not be mathematically possible to get much better than we already are at that sort of thing.

Why would intelligence top out like this? Well, if we think of intelligence as the ability to extract information from data, then even an infinitely advanced model endowed with infinite compute will be limited by the fact that there’s a limited amount of information that can be extracted from the data.

For one thing, data itself is in limited supply. You can’t transform the world unless you can (in some generalized sense) understand it, and you can’t understand the world unless you can measure it, and our ability to measure the world is inherently limited and finite.

This is basically the hypothesis advanced by Arvind Narayanan and Sayash Kapoor:

We think there are relatively few real-world cognitive tasks in which human limitations are so telling that AI is able to blow past human performance (as AI does in chess). In many other areas, including some that are associated with prominent hopes and fears about AI performance, we think there is a high “irreducible error”—unavoidable error due to the inherent stochasticity of the phenomenon—and human performance is essentially near that limit….We predict that AI will not be able to meaningfully outperform trained humans (particularly teams of humans and especially if augmented with simple automated tools) at forecasting geopolitical events (say elections).

Sosin says something similar:

We hold a thin scatter of facts about the world, and intelligence or reasoning is whatever fills the space between them…Where the space between the facts behaves well, this is close to godlike…Coding, math and most administrative work are [like this]. What makes them easy is that they have relatively smooth solution spaces and are tractably verifiable…Most of what matters doesn't behave like that. The universe is mostly the emergent behavior of complex systems…The limit is contact with reality. A smarter reasoner fills the gaps between known facts in simple areas faster and better, but it doesn't produce new facts.

Sosin makes an important point here, which is that the limitations of intelligence aren’t necessarily about limited data. Even if we can keep on collecting infinite data, the cost of extracting additional information from that data might explode to infinity. This is the idea of chaos. Even in a deterministic universe where the present states of all particles are enough to perfectly determine the future, our ability to predict the future can be inherently limited; the tiniest infinitesimal error in our measurement of the present explodes into a huge error when we attempt to extrapolate even a small distance into the future.

So although we don’t know yet, it’s possible that humans were already hitting the point of diminishing returns with regards to individual cognitive capacity, and that superintelligent machines will never be as far beyond us as we are beyond dogs. But even if that’s true, I can think of at least three reasons why machine superintelligence could still deliver huge productivity gains.

Smart matter

The most obvious advantage that machine superintelligence confers is replicability. The number of human intelligences is fixed by the fertility rate, and we don’t know how to substantially boost that rate; in fact, it’s falling inexorably, and the human race is set to shrink.

AI isn’t bound by those limitations. By building more data centers with more compute, you can run more agents in parallel — essentially, you get more cognitive work. It’s not free, but nor is it limited. It’s good old physical capital; unlike human capital, you can just build more of it whenever you like, simply by reinvesting some portion of your economic output. Imagine if we suddenly discovered a way to manufacture more land in any city on the planet; this is similar.

Of course, lots of tasks are physical ones, and for these you need physical machines — basically, robots. This is probably why so many AI people are now working on robotics, world models, physical AI, and so on. The roboticization of the world has a huge tailwind — the battery revolution, which allows energy to be stored and moved around much more easily. Conveniently, we got the physical tools to turn dumb matter into smart matter just as we also got the digital tools.

(It’s also worth noting that like the energy in batteries, the intelligence in robots is divisible. A robot the size of an ant can be remotely controlled by a data center the size of a football field. This is also a capability that human intelligence lacks.)

This doesn’t mean economic output will explode to infinity. But what it does mean is that humans will be able to use physical capital — GPUs and robots — to do more and more tasks at once, including many cognitive tasks that we used to do the hard way. In the long-run steady state, this should increase the capital-to-labor ratio of our society; each human will essentially leverage an army of intelligent machines. It’s basically another industrial revolution, and it has very little to do with whether artificial intelligence is smarter than human intelligence in any sort of head-to-head matchup.

Distributed tacit knowledge

The German company Zeiss makes the best glass on the planet. If one of the mirrors that Zeiss makes for ASML’s EUV chipmaking machines were the size of Germany, the biggest bump on that mirror would be just one millimeter high. Only a few other companies — and maybe no other company on Earth — can match that. Zeiss’ mirrors also have a number of other amazing properties, like not distorting much due to temperature changes.

How does Zeiss make glass this good? No one knows — not even the people at Zeiss. If the technology were capable of being written down on a blueprint, China would have hacked Zeiss and stolen it. If the technology were capable of being explained by a former Zeiss employee, or even several former Zeiss employees, China would have paid those people many millions of dollars to spill the beans.

Zeiss’ technology basically can’t be stolen, because it’s tacit and distributed. It consists of a vast number of little tricks and techniques that a huge number of individual employees use on a daily basis. These people don’t always even realize all those little things they’re doing that make the glass come out so good. And each employee knows a different set of tricks and techniques. The knowledge exists at the level of the organization itself, and is thus very hard to steal or recreate.

This is true of lots of corporate technology. A big part of the reason China can cut off the supply of rare earths to the rest of the world any time it wants to is that other countries aren’t very good at refining rare earths. Rare earths are difficult to separate from each other in solutions; it takes a ton of little chemistry tricks to do it cheaply at scale. Chinese refiners have spent four decades building up those little tricks and techniques; American or Japanese refiners won’t simply be able to replicate their efficiency overnight.

Except in the age of AI, this might change. Suppose American rare earth refiners give their employees a bunch of equipment to record everything they do — smart glasses, gloves, and so on — in addition to sensors distributed throughout their plants. AI will be able to synthesize all that information and very rapidly suggest small ways to improve the production process. Many of those little experiments will fail; others will succeed and will quickly be adopted, allowing another round of experimentation and improvement to begin very quickly. Crucially, AI’s ability to do this doesn’t depend on its raw intelligence — only on its ability to handle huge amounts of data very quickly.

In other words, in the age of AI, distributed tacit knowledge might not be nearly as big of a barrier to technological diffusion. This could improve economy-wide productivity, as lagging firms catch up to leading firms much more quickly. A more equal distribution of productivity would also make the economy more competitive, creating more surplus for consumers.

AI’s ability to quickly produce distributed tacit process knowledge might also supercharge productivity growth at the frontier. Imagine if any company could optimize any production process five times faster than today. The whole economy would speed up, as components got cheaper, turnaround times and product cycles got shorter, and scale-up got much faster.

Cloud laws

For decades, researchers in the field of natural language processing tried to figure out the principles behind human linguistic communication. They made frustratingly little progress; the processes by which humans convey information to each other through words just don’t seem to obey simple laws, like the ones that govern electromagnetism or the circulatory system.

Then along came AI, and suddenly linguistic communication seemed like a solved problem. LLMs can reliably sound like a human being, even if we don’t understand how they manage to do it.

What if there are lots of other aspects of the Universe that work the same way — too complex to understand in terms of simple laws, but not so complex that they just dissolve into unknowable chaos? It’s possible that we can reliably control these complex phenomena with AI, even if we never reduce them to the kind of principles that we can teach a grad student in a textbook.

Another way of saying this is that there may be laws of the universe that humans can’t understand but AI can. I call these “cloud laws” — causal regularities that can be exploited by technology, but which are too diffuse and complex for an individual human being to either intuit or communicate.

Human language seems to obey cloud laws, so why not other phenomena too? Perhaps social sciences like economics, sociology, and political science obey similarly complex regularities, and AI can help us find them. Perhaps there are physical processes — plasma, or topological materials, or aerial turbulence, etc. — that obey cloud laws instead of chaos?

In other words, thanks to AI, we might be on the precipice of a new age of scientific advancement. And this won’t necessarily depend on how smart AI is in comparison to a single human; it’ll depend on its computer-like ability to hold huge amounts of data in its working memory and extract complex patterns from that data.

If this turns out to be true, it means Francois Chollet is wrong. Chollet hypothesizes that groups of humans, using pre-AI computing tools, can approach AI’s level of scientific competence. But human collaboration is bottlenecked — it’s limited by our ability to intuit patterns at an individual level, and to communicate these patterns from one individual to another. AI, being a computer, just doesn’t have this sort of limitation; it can work with vast, diffuse patterns without having to break them into pieces or simplify them in order to compress them into the tiny pipelines of person-to-person explanation.

So even if AI never gets much better than humans at the kind of science that humans have done heretofore, it might open up whole realms of scientific discovery that have previously been totally inaccessible to even the largest groups of the smartest humans. If much of the Universe turns out to be ruled by cloud laws, we could be on the precipice of a scientific renaissance.

The common thread in all three of these examples is that AI may revolutionize productivity not by being much smarter than a single individual human — not by simply solving harder and harder math problems — but by marrying human-style intelligence to the vast, inhuman capabilities of computers. We could simply be thinking about the benefits of intelligence wrong — arrogantly privileging the kind of mental tasks we humans happen to do especially well, while ignoring the value of the tasks we do poorly.

SpaceX was more than lucky Friday, sending its 13th full-scale Starship test flight halfway around the world from South Texas to an on-target, intact splashdown in a remote section of the Indian Ocean.

Starship has made precise splashdowns before. This time, however, the ship gently tipped over and came to rest floating in a remote stretch of ocean west of Australia. Previous splashdowns ended with conflagrations, an outcome SpaceX officials have come to expect with each water landing of Starship.

The buoyant Starship looked to be in fine shape after reaching the Indian Ocean. SpaceX engineers flew drones over the vehicle to inspect its heat shield, getting their best look yet at how more than 18,000 ceramic tiles insulating the ship’s stainless steel airframe weathered the trip to space and back. The tiles withstood temperatures up to 2,600° Fahrenheit (1,430° Celsius) as Starship streaked back into the atmosphere at the end of its hourlong flight from Starbase, Texas.

Encouraged by the picture-perfect end to Friday’s flight, SpaceX is likely to take the next leap with Starship on the next launch, according to the company’s founder and CEO, Elon Musk. That would entail hurling Starship on a longer-range trajectory, perhaps into low-Earth orbit, to bring the vehicle back to the launch site. Mechanical arms on the launch tower will attempt to capture the ship as it slows to a hover. SpaceX has already done this with the rocket’s massive Super Heavy booster, which is larger and heavier than Starship itself, but comes back at a fraction of the speed.

“Unless we discover problems after mission data review, SpaceX will attempt to catch the ship with the tower on next flight,” Musk wrote on X Friday evening.

Hungry for data

Engineers have long identified the performance and durability of the heat shield as one of the toughest challenges for the Starship program. SpaceX’s ambition for Starship includes rapid turnarounds, and eventually multiple flights per day. If this is to be achievable, the heat shield has to be robust against the vibrations and extreme temperatures it sees during launch and reentry. It can’t be refurbished or replaced after every flight.

“This is the first time we’ve put an intact Starship in the water,” said Dan Huot, a SpaceX communications manager providing commentary on the company’s live webcast. “This is a dream scenario for the team that’s trying to get this heat shield data.”

The intact splashdown Friday also opens up the possibility for SpaceX to tow the rocket back to shore, probably somewhere in Australia, for more detailed inspections. Starship is designed for reuse, but not after exposure to salt water.

“This is so critical for refining the heat shield and everything else. The really good news is … you’re looking at a pretty intact-looking heat shield,” Huot said. “We were flying at a significantly higher dynamic pressure, so putting way more stress on this vehicle on the way uphill, and it’s sitting there in the water.”

SpaceX continued receiving signals from Starship after splashdown through the company’s Starlink satellite broadband network. A camera onboard Starship showed no signs of external damage to the vehicle’s six Raptor engines as water lapped against the lens.

Friday’s test flight was the 13th launch of SpaceX’s Starship on top of its massive Super Heavy booster, and the second flight of the latest iteration of the world’s most powerful rocket—Starship Version 3.

The 408-foot-tall (124-meter) rocket lifted off from Starbase, Texas, a few miles north of the US-Mexico border, at 5:51 pm CDT (6:51 pm EDT; 22:51 UTC). Thirty-three methane-fueled Raptor engines on the Super Heavy booster steered the rocket east over the Gulf of Mexico and into the upper atmosphere, where the booster separated from Starship’s upper stage after burning most of its cryogenic propellant.

The only thing left incomplete on SpaceX’s checklist for Flight 13 was the controlled splashdown of the Super Heavy booster. It was supposed to relight its engines for a landing burn off the coast of Texas, simulating maneuvers required to return the booster back to the launch pad for recovery and reuse.

But some of the Raptor engines failed to restart for the landing burn, and the booster fell into the ocean at high speed. SpaceX also had trouble with the booster splashdown on the last flight in May. Friday’s flight showed some progress, but engineers have more work to do on recovering this version of the Super Heavy booster. SpaceX previously recovered and re-flew Super Heavy boosters two times with the Starship Version 2 design, but not with Version 3.

“The booster successfully completed the high thrust portion of the boostback burn with all 33 engines, the first time with a Super Heavy V3, before ending the burn early,” SpaceX officials wrote in an update on the company’s website. “It attempted to relight its engines for the landing burn, with a subset successfully igniting before experiencing a hard splashdown in the Gulf.”

Success in space

Starship continued accelerating into space as the Super Heavy booster descended toward the Gulf, firing its six engines more than five minutes before beginning a half-hour coast over the Caribbean Sea, Atlantic Ocean, and South Africa.

A few minutes after engine shutdown, Starship opened its payload bay door and started deploying a stack of 20 Starlink satellites. The satellites are the first of SpaceX’s upgraded Starlink V3 model, with significant improvements in throughput and power. They are also larger and heavier than Starlink V2s, meaning they will not fit on SpaceX’s smaller workhorse Falcon 9 rocket.

Starship released the Starlink V3s using its Pez-like deployer, tossing them overboard through a slit-like opening on the side of the spacecraft. The flat-panel satellites did not stay in space for long. They flew on the same arcing suborbital trajectory as Starship and were designed to burn up as they fell back into the atmosphere less than an hour after launch.

In little time, SpaceX ground teams established contact with each of the satellites through radio and laser communication links and “downloaded key telemetry” before their fiery demise. This provided engineers with their first look at the performance of Starlink V3s in space.

With the satellite deployments complete, attention turned to a brief restart of one of the ship’s Raptor engines, something engineers forewent on the last flight. The burn lasted approximately 14 seconds and was visible to observers in South Africa. SpaceX hailed the demonstration as “core capability for future orbital missions.” Gravity then pulled Starship back to Earth.

The next steps for Starship are getting to orbit and returning the ship to the launch site. Those achievements will give SpaceX the capability to begin launching operational Starlink V3 satellites, unlocking higher-speed direct-to-device connectivity for consumers and the US military, and the revenue to go along with it.

For NASA, an orbital flight puts SpaceX on a path toward orbital refueling. This demonstration is vital for any Starship flight beyond low-Earth orbit, including missions to the Moon in support of NASA’s Artemis program. NASA is working with SpaceX and Blue Origin to develop human-rated versions of Starship and the Blue Moon lander to ferry astronauts to and from the lunar surface.

“Excited for what will be learned from this mission,” NASA Administrator Jared Isaacman wrote on X after the launch of Flight 13. “When Starship comes online, its capabilities will be game-changing, not least of which will be ensuring we never give up the Moon again!”

SpaceX has one active launch pad in Texas, and a second Starship pad undergoing renovations there. Two Starship launch sites are under construction at Kennedy Space Center and Cape Canaveral Space Force Station in Florida. The orbital refueling demonstration will include two Starship launches, allowing the vehicles to rendezvous and dock with one another in orbit. One ship will then transfer methane and liquid oxygen propellants to the other through automated couplers.

It will take multiple refueling runs in rapid succession to refill a Starship with enough propellant to reach the Moon, requiring rapid reuse of Starships and the Super Heavy boosters at multiple launch pads. With Friday’s test flight, SpaceX seems poised to embark on the next phase of Starship’s iterative development program. It won’t be easy.

“Really critical data collected this mission on a demonstration deorbit burn for Raptor, and a first flight for the next generation Starlink satellites,” Shana Diez, SpaceX’s director of Starship flight reliability, wrote on X.

“It feels like we are ready to kick into gear on delivering payload to orbit, which I am very excited for,” Diez added. “Development vehicles are fun but at the end of the day, we are here to put things into space and I am psyched about this rocket’s game changing capabilities.”

Here’s a thought experiment. Take the best, most productive engineering team you can imagine, and parachute them into a large, heavily regulated enterprise. What happens to their output?

It drops. Not because they forgot how to write code, but because the organization around them has a different risk tolerance. That’s the part of the AI productivity conversation we’re missing. We keep talking about developer velocity, but rarely talk about compliance velocity. But if you want to increase one, you have to increase the other.

It’s not a code-writing problem

The constraint on large software teams hasn’t been how fast engineers can write code. Most engineers could ship a weekend project to production in an afternoon. Put those same engineers inside a mid-sized or large company, and the same change takes days or weeks. That’s not because the tooling is worse, but because the risk and compliance process around every change is doing exactly what it was built to do.

That’s why an initiative that seeks to “generate changes faster with AI” fails to solve our problem. It was already possible to write code quickly; what we struggled with was managing the risk of shipping changes.

Every action has a reaction

This is where physics is a useful metaphor. For every increase in the volume of change, especially in a risk-averse environment, there’s an equal and opposite reaction from the organization’s risk and compliance functions.

Here’s an example. A team is making 100 changes a day. They introduce AI and increase this to 200 changes per day, but production starts falling over twice as often, even though the change failure rate is the same. That causes risk and compliance to take an interest and introduce processes and policies to reduce the failures.

Within a few weeks, the team is back to 100 changes a day. Half of them are AI-authored, but there’s no productivity gain to show for it. The system has found its new equilibrium, and it looks a lot like the old one, except for all the new processes that compliance wrapped around it.

There’s a second version of this reaction, and it’s less about infrastructure and more about experience. Small teams build up deep, shared context about the problem they’re solving and the customer they’re solving it for. Most software has many authors but one user at a time living through the whole journey. As a team grows, holding that cohesion gets harder, but at least the humans on the team still talk to each other.

Now give every one of those engineers their own AI coding assistant. Each engineer becomes far more productive individually, but each assistant has even less shared context than the humans did. The result is the same pattern: a high volume of change, but a more disjointed product experience, with the end-to-end story getting lost.

The equal-and-opposite reaction to that fragmentation is usually a heavy-handed swing back toward centralization. We saw that a decade ago, when “let every team choose its own tools” gave way to standardized platforms, because nobody could move between teams without relearning their entire stack.

These reactions are rational. They are the result of a system correcting for risk that increases faster than anyone can account for.

The one-way ratchet

There’s something even more crucial than the equilibrium problem. Compliance only ever ratchets up; it never ratchets down.

Once a new rule shows up, like a mandatory review step, an extra test suite, or a new sign-off, it rarely gets removed, especially once it’s on a regulator’s radar. Velocity can go up and down as the team changes. Compliance doesn’t work that way. It accumulates.

Imagine a team gets excited about AI and starts generating twenty changes a day instead of one per engineer. Production instability goes up. A data leak that used to happen once a year now happens twenty times. New compliance requirements get bolted on to stop the bleeding. Eventually, the team realizes the strategy isn’t working and reverts to its old habits, but the compliance burden doesn’t go away with them. They’re now doing less, with more overhead, than when they started.

This is also how bad global solutions get applied to local problems. If one engineer had production access they shouldn’t have had and made a mistake, the right fix is local: work out why they had that access and fix it. The wrong fix (but the one organizations reach for when they’re not being deliberate) is global: nobody gets production access, ever, for anything. It solves the immediate problem and creates a dozen new ones.

Sometimes it goes even further than the company. A handful of businesses misuse a technology, a regulator responds with a blanket rule, and now everyone on the internet has to click through a cookie banner. That’s what happens when a few actors are irresponsible with something new, whether that’s self-driving cars, AI, or otherwise. Regulators aren’t looking for a reason to step in, but companies force them to through irresponsible decisions.

Ship safer, not just faster

If your company has a new initiative to boost AI-driven productivity, here’s the thing worth saying out loud before it starts: We already know risk and compliance will find equilibrium with whatever volume of change you produce. So plan for that from day one.

If every engineer on the team is using AI purely to generate more code, faster, you will accelerate the compliance ratchet. The healthier split is roughly half and half. Half the team should focus on how to make changes faster, and the other half on how to make those changes safer, more compliant, and less risky, with AI doing some of that work, too.

That second half of the work is genuinely interesting, and it’s underrated. AI is well-suited to reasoning about risk, not just generating change. It could review a pull request and decide whether a one-line CSS tweak needs a human reviewer at all, while flagging any change that touches the payments pipeline for real scrutiny. It could review the build and decide which of the three hours of automated UI tests are relevant to a README update, rather than running the whole suite, which is what the pipeline does.

None of that reduces the volume of change getting shipped. It reduces the risk associated with it, which is what’s actually been holding teams back.

The foundation of every software business is customer trust, and trust is not built by shipping more. If AI makes you ship faster without making you ship safer, you’re not getting more productive, you’re just winding the compliance ratchet a little tighter, and that one doesn’t wind back.

Pinterest's Resource Provisioner Pipeline centralizes Terraform execution across hundreds of workspaces and tens of thousands of AWS resources while preserving team ownership in a multi-repository environment. It enforces least privilege through GitHub OIDC, strict repository-path-to-workspace mappings, backend validation, down-scoped IAM roles, mandatory review, and explicit apply commands, while centralization enables consistent auditing, security scanning, and company-wide fixes.

Buzz 🐝

A workspace where humans and agents build together, on a relay you own.

What is this, really?

Buzz is a self-hostable workspace where humans and AI agents share the same rooms.

A Buzz community is the workspace a user reaches by URL. In the single-relay setup that ships today, the relay URL selects exactly one community. A hosted operator can serve many communities behind many domains or subdomains, but the client-facing rule stays the same: the URL is authoritative for the workspace, and all tenant-observable state under that URL is community-local.

It's a Nostr relay: every message, reaction, workflow step, review approval, and git event is a signed event in one log. Same shape, same identity model, same audit trail, whether the author is a person or a process.

In practice it feels like a team workspace. Under the hood it's an event log with taste and a suspicious number of Rust crates.

Yes, it's another AI-adjacent developer tool. We're sorry. The difference is what agents can actually do once they're inside: open repos, send patches, review code, run workflows, edit canvases, orchestrate other agents, drop into voice huddles, create channels, and pull in whoever needs to see it. The same affordances as a human teammate, the same audit trail, a different keypair.

Stuff you do in Buzz

• Ask the project a question and get an answer with receipts. Agents search six months of history and post the threads, not vibes.
• Let an agent triage a bug without giving it the keys to the kingdom. Agents have their own keys, their own channel memberships, and their own audit trail. Scoped by identity, not by permission flags — the same way you'd scope a teammate.
• Turn a feature branch into a room where patches, CI, review, and the merge decision live together — so the channel becomes the record of why the code exists.
• Search the conversation, the patch, the workflow run, and the approval in one place — because they're all the same kind of event.
• Let an agent run the workspace, not just talk in it. Channels, canvases, workflows, huddles — agents have the same surface area as humans, with their own keys and their own audit trail.

A look inside

Agents are members, not bots. Add an agent to a channel the same way you add a person.

Spin up a room in seconds. Name it, describe it, make it private.

Media you can talk about. Leave comments pinned to specific frames.

Why Buzz is better

One community. One identity model. One event log. Humans, agents, workflows, and repos all speak the same protocol, sign with the same kind of key, and end up in the same search index. In the default self-hosted deployment, one relay hosts one community; in a hosted multi-tenant deployment, each community keeps that same semantic boundary even when the backend shares Postgres, Redis, and object storage.

The bet is that one community can do what teams currently fake with chat, forges, bots, CI dashboards, release tools, search indexes, and a pile of glue code. Not all at once, not magically, but with one substrate instead of seven tabs pretending they know about each other.

Agents are part of the room, not haunted cron jobs.

Three little stories

Incident memory. It's 2am. You type "have we seen this error before?" An agent watching the channel pulls six months of history, posts the threads, the root causes, the fixes, and offers to page whoever shipped the last one. The whole exchange — question, answer, evidence — stays in the channel.

Branch as room. You open a feature branch. A channel appears. Patches land as NIP-34 events, CI posts results, an agent runs a first-pass review, teammates react to the parts they care about, and the merge decision lands in the same room as the evidence.

A release that writes itself. A workflow fires on a tag. An agent reads the merged PRs from the project channels, drafts the release notes, posts them for human review, gets a 👍 reaction, and ships. Every step signed. Every step searchable.

Works today · Being wired up · Strong opinions, pending code

Getting started

New to Buzz? Pick the path that matches you.

I just want to try the app

Grab a packaged build from the latest release — macOS (.dmg), Linux (.AppImage / .deb), or Windows (.exe). Install it like any other app.

By default the app connects to ws://localhost:3000. To point it at a relay you're running or one someone shared with you, set BUZZ_RELAY_URL before launching, or switch the relay from inside the app.

I work at Block

Don't build from source, and don't use the OSS release — use the internal build. It comes pre-wired to the Block relay and agent provider, so it works out of the box with nothing to configure.

I want to build & run from source

See Quick start below — this is the developer / self-host path.

Quick start

You'll need Docker and Hermit (or Rust 1.88+, Node 24+, pnpm 10+, just).

Once:

git clone https://github.com/block/buzz.git && cd buzz
. ./bin/activate-hermit
just setup && just build

Every day:

. ./bin/activate-hermit
just dev

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                             Clients                                     │
│  Human client         AI agent              CLI / scripts               │
│  (Buzz desktop)       (Goose, Codex, ...)   (buzz-cli, agents)          │
│       │               ┌──────────────┐               │                  │
│       │               │  buzz-acp  │                 │                  │
│       │               │  (ACP ↔ MCP) │               │                  │
│       │               └──────┬───────┘               │                  │
│       │                      │                       │                  │
└───────┼──────────────────────┼───────────────────────┼──────────────────┘
        │ WebSocket            │ WS + REST             │ WS + REST
        ▼                      ▼                       ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                          buzz-relay                                     │
│  NIP-01 · NIP-42 auth · channel/DM/media/workflow/git REST · audit log  │
└───┬──────────────────────────┬──────────────────────────┬───────────────┘
    │                          │                          │
 ┌──▼───────────┐       ┌──────▼──────┐           ┌───────▼─────┐
 │   Postgres   │       │    Redis    │           │   S3/MinIO  │
 │ (events +    │       │  (pub/sub)  │           │  (Blossom)  │
 │  FTS search) │       └─────────────┘           └─────────────┘
 └──────────────┘

What it is not

• Not blockchain. Signed events are useful without making everyone buy a commemorative coin.
• Not an AI replacement plan. Buzz works best when humans stay in the loop and agents stay in the room.
• Not finished. We will tell you what works and what doesn't.

What is Chat2DB?

Chat2DB Community is a free, cross-platform database client for Windows, macOS, and Linux. It runs entirely on your machine and combines a full-featured SQL workspace with an AI assistant that you connect to your own model.

• 30+ databases — MySQL, PostgreSQL, Oracle, SQL Server, ClickHouse, MongoDB, Redis, SQLite, MariaDB, TiDB, Hive, DB2, Snowflake, BigQuery, Elasticsearch, and more via plugins.
• SQL workspace — editing, completion, formatting, execution, saved SQL, and execution history.
• AI assistant — bring your own AI model to generate, explain, and optimize SQL in natural language.
• Database management — browse metadata, manage tables and objects (DDL/DML), and edit data in place.
• Data import and export, dashboards and charts, and an open-source CLI with MCP support.

Quick Start

Option 1: Desktop App

Download the installer for your platform from GitHub Releases, install it, and start connecting to your databases. No further setup is required.

Option 2: Docker

Requirements: Docker 19.03.0+, Docker Compose 2.0.0+ (Compose V2, only for the Compose variant), 2+ CPU cores, 4+ GiB RAM.

First create the encryption key, then start the container:

# Run once from a repository checkout. Re-running reuses the same valid key.
git clone https://github.com/OtterMind/Chat2DB.git && cd Chat2DB
./script/security/init-community-encryption-key.sh

docker run --detach \
  --name chat2db-community \
  --restart unless-stopped \
  --publish 127.0.0.1:10825:10825 \
  --volume "$HOME/.chat2db-community-docker:/root/.chat2db-community" \
  --env CHAT2DB_COMMUNITY_ENCRYPTION_KEY_FILE=/run/secrets/chat2db-community-encryption.key \
  --volume "$HOME/.config/chat2db-community/encryption.key:/run/secrets/chat2db-community-encryption.key:ro" \
  chat2db/chat2db:latest

Then open http://localhost:10825 in your browser.

Alternatively, use the bundled Compose definition:

./script/security/init-community-encryption-key.sh
docker compose --file docker/docker-compose.yml up --detach

Notes:

• To update, pull the new image, remove the old container, and run the start command again. Keep ~/.config/chat2db-community/encryption.key across rebuilds.
• The docker run example stores application data in $HOME/.chat2db-community-docker; the Compose definition uses the chat2db-community-data named volume. These locations do not share data.
• Chat2DB Community 5.3.0 uses the independent /root/.chat2db-community directory and does not automatically migrate data from earlier images that used /root/.chat2db.

Security Notes

Chat2DB Community is a single-user, local-first application. It has no user accounts or authorization boundaries between users. Keep the HTTP service bound to 127.0.0.1 or ::1 and do not expose it to other users or untrusted networks.

Custom JDBC drivers are executable Java code — install them only from sources you trust. Imported configuration files, archives, SQL files, database contents, and AI responses remain untrusted data.

Encryption Key

Chat2DB Community encrypts stored datasource passwords and AI model API keys with AES-256-GCM using a per-installation key. Create it once from a repository checkout (requires openssl):

./script/security/init-community-encryption-key.sh

The key is written to ~/.config/chat2db-community/encryption.key. Back this file up separately and keep it across upgrades and container rebuilds — replacing or losing it makes previously stored datasource passwords and AI model API keys unreadable. Web/headless startup fails when no valid key is provided; only Desktop mode creates a missing key automatically.

The key must be valid Base64 that decodes to exactly 32 bytes. The bundled initializer generates the standard padded form: 44 Base64 characters ending in =. It is cryptographic key material, not a human-readable password. Datasource passwords and AI API keys use the same key with separate authenticated AAD values, so ciphertext from one purpose cannot be decrypted as the other.

To use a custom path, pass it to the script and configure the same path when starting Chat2DB:

./script/security/init-community-encryption-key.sh /secure/path/chat2db-community.key

java -Dloader.path=chat2db-community-server/chat2db-community-start/target/lib \
    -Dchat2db.runtime.mode=community \
    -Dchat2db.mode=WEB \
    -Dchat2db.gui=false \
    -Dchat2db.network.status=OFFLINE \
    -Dchat2db.community.encryption-key-file=/secure/path/chat2db-community.key \
    -Dserver.address=127.0.0.1 \
    -Dserver.port=10825 \
    -jar chat2db-community-server/chat2db-community-start/target/chat2db-community.jar

Key configuration is resolved in this order:

1. JVM property chat2db.community.encryption-key containing the Base64 key.
2. Environment variable CHAT2DB_COMMUNITY_ENCRYPTION_KEY containing the Base64 key.
3. JVM property chat2db.community.encryption-key-file containing a key-file path.
4. Environment variable CHAT2DB_COMMUNITY_ENCRYPTION_KEY_FILE containing a key-file path.
5. Default file ~/.config/chat2db-community/encryption.key.

Build from Source

Prerequisites

• Java runtime: Eclipse Temurin 17
• Node.js 18.17.0 or later
• Maven 3.8 or later

Clone the Repository

git clone https://github.com/OtterMind/Chat2DB.git

Frontend

Use Yarn with the checked-in lockfile.

cd Chat2DB/chat2db-community-client
yarn install --frozen-lockfile
yarn run start:community:hot

Backend

cd Chat2DB
mvn -B clean package -Dmaven.test.skip=true -Dchat2db.finalName=chat2db-community \
    -f chat2db-community-server/pom.xml \
    -pl chat2db-community-start -am
./script/security/init-community-encryption-key.sh
java -Dloader.path=chat2db-community-server/chat2db-community-start/target/lib \
    -Dchat2db.gui=false \
    -Dchat2db.runtime.mode=community \
    -Dchat2db.mode=WEB \
    -Dchat2db.network.status=OFFLINE \
    -Dchat2db.community.encryption-key-file="$HOME/.config/chat2db-community/encryption.key" \
    -Dserver.address=127.0.0.1 \
    -Dserver.port=10825 \
    -Dspring.profiles.active=dev \
    -jar chat2db-community-server/chat2db-community-start/target/chat2db-community.jar

Build a Local Docker Image

./docker/docker-build.sh 5.3.0 chat2db/chat2db:5.3.0

Database guides

Step-by-step guides for connecting Chat2DB Community to specific databases:

• BigQuery — Google BigQuery via a Google Cloud service account.

Community vs Commercial Editions

The Community edition contains the full local database client described above, including custom AI model support. The commercial Pro and Enterprise editions build on the same core and add hosted AI services, user accounts, cloud storage and multi-device sync, and team collaboration and governance features.

Contributing

We welcome bug reports, feature requests, documentation improvements, testing feedback, and pull requests from the community.

• For bugs and feature requests, please use GitHub Issues.
• For questions, setup help, and open-ended discussions, please use GitHub Discussions.
• If your pull request is related to an issue, please link it in the PR description.

Community and Support

• GitHub Issues: report a bug or request a feature
• GitHub Discussions: ask questions and share ideas
• Discord: join our Discord server
• Email: Chat2DB@ch2db.com

License

Chat2DB Community version 5.3.0 and later is available under the license terms in this repository. This is a source-available license based on the Apache License 2.0 with additional conditions. Chat2DB releases published before version 5.3.0, including version 0.3.7 and the earlier historical tags, remain under the Apache License 2.0.

“How do you find problems worth working on?” a senior engineer I mentor asked me recently. He’s trying to make the jump to staff engineer and realized that the role isn’t just about doing the work he’s assigned. He also needs to get involved in figuring out what his team and org should be building.

Someone else had suggested blocking out time in his calendar to think about the bigger picture. He’d tried that, but hadn’t found it productive, so he asked if I had any alternatives.

I told him I rarely find good problems by staring at a blank page and trying to “think strategically.” Instead, I act like a sponge. I listen to the stream of day-to-day noise, absorb the problems people are having and let them sit in the back of my mind. Over time, some fade away while connections begin to appear between others that initially seemed unrelated. Eventually, I start to see what’s really slowing people down and what my team or I can do about it.

I’ve worked with many engineers who’ve never really tried this. They wait for managers or leads to identify opportunities, then demonstrate their value by solving the hardest assigned problems. That can absolutely lead to promotion. But the projects that have made the biggest impression in my career were the ones where I found and solved an important problem my leaders did not yet realize existed.

One caveat: my experience comes mainly from working on infrastructure and developer tools at large companies, on teams where engineers have a lot of bottom-up autonomy to influence their roadmaps. In a more top-down environment, there may simply be less room to work this way.

Absorb problems, not requests

People love talking about the problems they are facing: in meetings, chat threads, presentations and email. They explain why their work is hard, complain about what slows them down and describe what they wish they could do.

When something overlaps with my area, I start pulling on the thread. I might ask, “If X existed, would it solve your problem?” or point them at an existing feature in a product I own and ask how much of their use case it covers.

Users often ask for a particular solution instead of explaining their root issue. Rather than taking the request at face value, I keep digging until I understand what they are trying to accomplish and why existing products do not work for them.

As a natural introvert, this sort of ambient listening works particularly well for me. I don’t need to fill my calendar with speculative meetings just to find ideas; there is already an enormous amount of useful information flowing around me during a normal week.

When a problem seems worth exploring, though, I become more active; I need to see how it affects the team’s day-to-day work. I’ll sit with them as they walk me through their workflows and the bugs they’re investigating. When I can, I’ll try working through some of those bugs myself. Seeing the problem firsthand makes it easier to separate what the team actually needs from the solution they asked for.

I also seek out people who see more of the organization than I do: those who own critical systems, work across several teams or have particularly deep insight into the work downstream of my team. I’ll arrange a 1:1 or coffee chat and ask about interesting problems they’ve come across. They may have already seen the same issue in several places and started connecting the dots, giving me a head start on patterns I might otherwise have taken much longer to notice.

Let problems accumulate

Several times, I’ve been burned by moving too fast. I became excited by a request from a vocal team, built the feature and watched them barely use it. Their priorities had changed, or the request had come from a one-off investigation that no longer mattered. How eager a team was in that moment wasn’t the same as how important the feature was relative to everything else my product needed to support. By hyperfocusing on their request, I lost sight of the bigger picture.

That taught me to let potential problems pile up. Listening the way I do leaves me with far more of them than I could possibly solve, and not all deserve action. Most don’t need to turn into projects the first time I hear about them; waiting can be a superpower.

Waiting means the same problem might pop up independently in different teams, making it a higher priority to solve. Or problems that look different on the surface might turn out to have the same shape, so I can address several use cases in one shot. Or, as I’ve learned painfully, the requesting team didn’t even care that much in the first place.

Instead, I make a mental note and revisit the problem if it comes up again. Other engineers I know write this sort of thing down more systematically. The mechanism is a personal choice: everyone has to figure out what works for them. What matters is keeping unresolved problems around long enough for more evidence to accumulate.

Find the common shape

Waiting helps me collect evidence, but that alone doesn’t tell me what to build. I still need to work out whether the problems I’ve retained are genuinely related and what, if anything, could address them together.

Perfetto, the performance debugging tool I work on, is a good example. It displays recordings of system activity on a timeline made up of rows called “tracks.” Over a couple of years, teams kept asking for small, specific additions to the UI. One wanted a command to keep their preferred tracks pinned to the top of the screen; the next team wanted the same, but for a completely different set of tracks. Others wanted Perfetto to open already zoomed in on a particular part of a recording, or to show a custom aggregation tuned to what they cared about. A few had stopped waiting for us and built elaborate workarounds with bookmarklets.

By the time enough of these had piled up, my head was the usual tangle: the requests themselves, the constraints on each and a handful of half-formed solutions. I’ve learned not to force a solution by just sitting at a desk and thinking. Instead, my best untangling happens on long, aimless walks around London, where connections come more easily when I’m not trying to force them.

What I eventually realized was that none of these teams really wanted the specific feature they’d asked for. Each wanted to personalize Perfetto for their own workflow without imposing their choices on everyone else. The underlying need wasn’t any one feature but rather the ability to extend the UI. When a connection like that finally clicks, it’s one of the best feelings in the job: several awkward requests collapse into a single idea, and possibilities open up that none of them hinted at on their own.

That feeling, though, is exactly when I have to be careful, because a common shape is only a hypothesis and elegance is not evidence. When it happened with extending the UI it turned out to be real, but I’ve been fooled before.

In another recent case I was convinced that building a transparent caching system for querying Perfetto traces would solve issues with sharing large traces and repeated queries. It was only as I wrote the RFC and built a prototype that I realized the elegance was a lie: the two problems wanted genuinely different solutions. I reluctantly split the design in two, both halves of which have since shipped.

Pressure-test before building

You’d think this would be the moment I start building, but it usually isn’t. How far I go depends on how sure I am that the idea works and that people actually want it.

If something is useful and low-risk enough, I act straight away: I send the change and let my manager know. When I’m unsure whether an idea will work or how much effort it will take, I build a throwaway prototype instead; it exposes the failure points and gives me something concrete for others to react to. And when an idea is big but I’m convinced by it, I commit to the full effort: weeks or months of work and the hard yards of building support across other engineers and teams.

Through all of it, I’m not only trying to convince other people; I’m also trying to convince myself. Sometimes the honest answer is to stop: if people don’t see the value I do, or we hit a major technical wall, I’d rather drop the idea now than build something no one uses or that becomes a maintenance nightmare. And sometimes it holds up but the timing is wrong, so I park it, ready to spring into action the day it becomes an org priority.

When an idea does hold up, I don’t necessarily need to be the person who builds it. I might implement it, someone else on my team might, or it might change what the org focuses on. Finding and shaping the right problem can have an impact even when I don’t own the implementation.

The Perfetto extensions idea was worth that full effort. We were already building plugins to modularize the UI, but they weren’t enough: teams had to open source all their plugin code, which wasn’t an option for many internal use cases. So before building anything new, I took the problem and my proposal to my manager, teammates and the client teams. I ended up writing two RFCs, having several 1:1s and giving a couple of talks, refining it as the feedback came in.

In the end, I designed and implemented macros as “lightweight extensions”: a way to automate actions in the UI without writing a plugin. Extension servers took the idea further by letting teams share their macros.

Instead of implementing every requested feature ourselves, we gave teams ways to adapt Perfetto to their own needs. Dozens of teams inside Google now use macros and extension servers, and several other companies use extension servers internally too.

Solving useful problems helps me find the next one

The more often I go through this process, the easier it becomes. When I show genuine interest in someone’s problem, ask useful questions or help solve it, they remember. They start coming to me earlier and bring me into conversations with other people facing related issues.

That gives me a wider view of what is happening across the organization, making it easier to spot patterns and build things people actually need. Solving one of those problems brings me into more conversations, and the loop continues.

Those successes build the kind of trust that comes from long-term stewardship. Early on, I had to turn many of these ideas into something real myself to prove that my judgment was sound. Over time, my manager and org gave more weight to my assessment of what mattered. That allowed me to influence the roadmap without needing to own every project.

This differs from the idea that becoming a staff engineer means replacing technical work with meetings and coordination. For me, conversations are inputs into what I build, not the end result.

Conclusion

That is what I wanted my mentee to understand: finding problems worth solving isn’t separate from the rest of the job. It comes from staying engaged with people’s work long enough to see what no single request can show you.

Background

We are the Ad Platform team at Zalando Marketing Services (ZMS). We build the data pipelines that power Sponsored Products, the paid placements that advertisers use to promote their items across Zalando's catalog and product card pages.

When a customer sees or clicks a sponsored item, that interaction needs to be matched against the original auction that placed it. The auction record carries the ad server's delivery decision context; the interaction event carries the result. Joining the two is what turns a raw click into a billable charge and closes the feedback loop to our ad serving system.

This join has to happen in near-real time and within a bounded window. We limit it to 15 minutes: interactions that arrive after that are dropped. Missing an event means an advertiser action goes unbilled, so the match rate is a business-critical metric.

This article focuses on the near-real-time stream that supports ad server logic. Correct billing and campaign reporting also rely on a separate batch data pipeline. Together they form a near-classical lambda architecture, but that is a topic for another post.

The homegrown solution was developed more than 7 years ago at an ad-tech startup later acquired by Zalando. It was simple and worked, but accumulated limitations that improvements alone couldn't fix.

Homegrown solution

It was a distributed Java application, which consumed Kinesis and Nakadi (abstraction on top of Kafka) streams, storing bid events in a basic in-memory cache. On each user interaction event, it searched for the corresponding bid event by key in the cache and tried to match, enrich, and produce a billing event. Since we had quite a heavy load, up to 200MB/s, we used multiple pods for this application.

Because it was very straightforward, it worked fine, but with some caveats that should have been addressed:

1. Pods couldn't be cleanly partitioned across both streams, so each pod had to read all interaction events in addition to its share of bid events.
2. It didn't have any memory state backup or checkpointing, losing stored events on each deploy or rescale operation.

Those issues led to overprovisioning of pods, to avoid re-scaling operations.

Alternatives

Option 1. Keeping the old application: We could improve it by implementing checkpointing and aligned stream partitioning, but better tools already existed.

Option 2. Nakadi SQL: Zalando's internal stream processing framework. A PoC showed only ~50% event match rate, which was not viable. The pipeline dropped a significant portion of events without a clear explanation, and we didn't invest further in diagnosing it.

Option 3. Apache Spark: Has adoption at Zalando, but other teams who had evaluated both reported better latency and autoscaling behavior with Flink, and that was enough to rule Spark out.

Option 4. Flink: Already in production at Zalando across multiple teams. Fits our use case well: keyed state, event-time joins, checkpointing, horizontal partitioning.

The Tech Radar TRIAL status and limited team experience were real risks, but the Zalando Flink guild and existing production deployments across other teams made the risk acceptable.

Architecture: What We Built

The join logic stayed the same: consume two streams, match events within a 15-minute window, produce billing events. We made two improvements over the homegrown solution.

We added a RocksDB state backend with incremental 3-minute checkpoints. On any restart or rescale, the pipeline recovers from the last checkpoint instead of losing up to 15 minutes of buffered events. This is what made safe autoscaling possible.

We also buffer state for both streams instead of just bid events. The homegrown solution dropped any interaction event that arrived before its corresponding bid. Buffering both sides handles this out-of-order case and increased the event match rate by 0.5%.

The Road to Production: What Actually Happened

Infrastructure: AWS managed vs K8s

We chose the AWS managed deployment option for faster prototyping and quickly discovered it was simple but not flexible or transparent. Later we migrated to K8s deployment, using Zalando application templates.

API Choice

There are 4 levels of abstractions in Flink API. We chose Datastream API since we had deep Java knowledge and could solve potential issues, and we needed configuration flexibility.

Join choice

For our scenario, applicable high-level options are Sliding and Interval join, where Sliding is overkill by definition, because we don't need the sliding feature.

Interval join

First approach

Initially we thought interval join would be a good fit for our use case: It supports keyed partitioning and the same time boundaries we needed.

orangeElem.ts + lowerBound <= greenElem.ts <= orangeElem.ts + upperBound

The system worked but used quite a lot of workers and resources, more than our status quo application.

Flame graph and seeks

The critical finding was in the Flame Graph in the Flink UI. Interval join used about 15-30% of CPU just on seek operations. Looking at the interval join implementation, the matching path opens a RocksDB iterator and performs a seek before reading any records, regardless of how many entries the partition contains. Since our bid IDs are unique, each partition holds exactly one entry, but the seek fires on every element processed. A seek requires locating the starting position across sorted files on disk, making one seek per element a significant overhead at our throughput.

Interval join also uses expiration timers stored in a RocksDB-backed sorted set with an in-memory cache. With a 15-minute window at our throughput, this queue grows to ~70 million entries permanently. When the cache is drained, Flink seeks into RocksDB to reload the next batch, generating a second source of constant scans.

Why KeyedCoProcessFunction solves it

To solve this, we wrote our own matching logic using KeyedCoProcessFunction. We created two ValueStates, one for bid events and one for interaction events, both partitioned by bid ID. Since bid IDs are unique, each lookup is a direct point lookup in RocksDB with no iterator and no seek.

State expiry is handled by RocksDB's TTL compaction in background instead of explicit timers, so there is no per-element timer registration and no ~70 million entry queue to maintain. Watermark advancement no longer needs to scan a timer queue at all, eliminating the second source of seeks.

RocksDB full state increase

After migration from the Interval join to low-level Process join we had problems with growing state and cleaning up the cache. Since there were no timers, we needed other options for cleanup. We also used incremental checkpoints with a state backend, and the only available out-of-the-box cleanup option is during state compaction. Otherwise, we would need to manually create timers.

We increased write buffer and compaction file sizes so compaction runs less frequently but processes more data per run, giving the TTL filter more opportunity to collect expired keys.

Infrastructure: Kinesis streaming

Connector

The Kinesis Flink connector library surprisingly had no support for reading aggregated events using Protobuf. We had to write our own deaggregation layer inside Flink to be able to read such events.

Pod evictions

Karpenter (the K8s node autoscaler) evicted our pods during rescaling operations before they were ready to consume from streams. Flink has its own autoscaler and K8s shouldn't interfere with it. Adding the appropriate annotation to block K8s from interfering resolved it. A PodDisruptionBudget is an alternative way to enforce the same protection.

OOM kills

Memory tuning wasn't straightforward either. We needed to find out the required CPU to Memory ratio, because the load type was different due to checkpoints. Flink used much more CPU than our status quo. The OOM kills were not Java OutOfMemoryError exceptions. The Linux kernel was terminating the process. We increased pod memory from 16GB to 20GB, creating a 4GB buffer above the JVM process size.

Migration

We built a full shadow pipeline for the migration, with dedicated input and output streams for convenient analysis. In the downstream service that consumes traffic events, we integrated a Nakadi client to also read the shadow stream and write to a shadow table in parallel with the main pipeline. This let us compare real production data without any customer impact.

Results

The most important improvement was checkpointing. Previously, any restart or rescale lost up to 15 minutes of buffered events. With 3-minute incremental checkpoints, the pipeline recovers from the last checkpoint instead, making aggressive autoscaling safe without state loss and eliminating the need to overprovision pods.

This reduced average pod count from a constant 20 to 5 on average (range 1-8). The reduced pod count drove three improvements:

1. CPU reduced from 30 to ~20 cores.
2. Memory reduced from 320GB to ~100GB.
3. EC2 costs reduced from ~€80 to ~€30 daily.

The two-way matching algorithm also increased event enrichment success rate by ~0.5%.

What's Next

This was just a first step. We plan to move event aggregation directly into Flink, eliminating a downstream service that currently does this after the fact. We also plan to migrate the Sponsored Brands equivalent of this pipeline, which has the same homegrown join architecture.

Cache consistency: strategies to keep data fresh

A cache that has drifted from your database will happily serve wrong prices, expired permissions, or phantom inventory, and it won't feel a shred of guilt about it. Cache consistency is the discipline behind keeping that drift small, so cached values don't wander away from the source database. The speed advantage of caching depends on knowing how far cached values can drift from the source data, then keeping that drift within the bounds your app can handle.

Caches sit in front of nearly every high-traffic system today, which means keeping them in sync with a source database is a question that comes up constantly. This guide covers what cache consistency means, why caches drift, how the classic patterns trade off, and how event-driven sync keeps data fresh when time-to-live (TTL) timers alone can't.

What cache consistency means

A cache is consistent when its values match the source of truth, and every consistency strategy is really about shrinking the interval where the two disagree. That drift framing gives us the practical definition: stale cache entries appear whenever cached values fall out of step with the database behind them. Staleness isn't binary; it's a window. Every strategy in this article shrinks that window, bounds it, or decides which data deserves the tightest one.

How you propagate updates determines how consistent your reads are:

• Synchronous updates: propagate a change to every relevant node before readers see it, so clients are designed to see the same data once the update completes.
• Asynchronous updates: trade that guarantee for speed and land you in eventual consistency, where some nodes may temporarily serve stale data.

Most real systems sit somewhere on this spectrum, and picking your spot deliberately beats discovering it during an incident. Even hyperscalers treat this as a long-term engineering project: Meta improved TAO's cache consistency from six nines to 10 nines over years of dedicated invalidation work.

Why caches drift: TTL timing, write ordering & instance races

Caches drift for three well-known reasons: TTL windows that outlast the data, write-ordering races between the cache and database, and multiple app instances stepping on each other during cache fills. The drift Meta spent years fighting comes from this same short list of mechanisms, and you'll likely recognize at least one from your own production history. The mechanics differ, but the result is the same: the cache keeps serving a value after the source of truth has moved on.

TTL timing windows

In expiry-based caching patterns, each entry gets a TTL, a lifespan after which the entry counts as stale and gets removed or refreshed. But a cache relying on TTL alone serves stale data for the entire remaining window after the database changes. If your TTL is five minutes and the price changed ten seconds in, readers see the old price for four minutes and fifty seconds.

The expiry mechanics themselves add wrinkles worth knowing, and Redis is a good concrete example because its behavior is well documented. Redis expires keys two ways: passively, when a client touches an already-expired key, and actively, through a background cycle that samples 20 keys every 100 milliseconds and reclaims the expired ones it finds, repeating whenever a sample turns up too many expired keys. That active pass keeps the count of expired-but-still-in-memory keys bounded instead of letting it grow. The lesson generalizes: no cache deletes keys the instant their TTL hits zero, so the "expired" window is always a little longer than the TTL suggests.

Write-ordering races

TTL windows are at least predictable; races aren't. A race condition happens when two operations overlap in time and the outcome depends on which finishes first, which can leave the cache holding a value from the wrong moment. A stale set happens when concurrent updates get reordered, leaving the cache holding a value that doesn't reflect the latest write. A cache fill can interleave with a database transaction so the cache ends up with a mix of old and new data, and if the invalidation message then fails, nothing self-corrects until the next write or TTL expiry. The racing window for these interleavings is tiny, but at hyperscaler scale, with enormous query and cache-fill volumes, even rare races show up often enough to matter.

Multi-instance population races

Horizontal scaling multiplies the racers. Picture two app instances: instance A deletes a cache entry and is about to update the database. Before A's write lands, instance B gets a cache miss, reads the old value from the database, and writes it back to the cache. A then commits. Now the database has the new value and the cache has the old one, with no invalidation coming to fix it.

Misses also pile up. A thundering herd happens when many requests miss simultaneously and hammer the database, each one racing to repopulate the same key with whatever snapshot it happened to read.

Cache-aside vs. write-through: the classic trade-offs

These failure modes are why your caching pattern matters: each one draws the consistency line in a different place. That line determines whether writes wait, reads risk staleness, or your app carries more of the coordination work.

Cache-aside (lazy loading)

In the cache-aside pattern, the app manages the cache directly. It loads data only on demand:

1. Check the cache: the app tries to read the item from the cache first.
2. Fall back to the database: on a miss, the app reads the item from the data store.
3. Populate and return: the app writes the item into the cache and returns it to the caller.

On writes, the app updates the database and then invalidates the cached entry so the next read repopulates it fresh. It's a flexible and common pattern, but the flexibility has costs. Cache-aside doesn't guarantee consistency: an external process can change the data store at any time, and the cache won't notice until the item reloads. The first request for uncached data pays the first-request latency penalty, running as slow as a normal database call. And the multi-instance races above are cache-aside's signature failure mode.

Write-through

Write-through flips the write path: writes go to both the cache and database in the same synchronous operation, so readers typically see the new value after a successful write, assuming both writes complete. That buys you read-your-writes behavior, which matters a lot for data like balances and orders.

You pay for it three ways. Every write waits on two systems instead of one. A partial failure between the two writes leaves an inconsistency you have to resolve yourself. And everything written lands in the cache whether or not anyone ever reads it, which can waste memory on cold data. A freshly spun-up node also starts empty and stays that way until writes repopulate it. There's a third sibling worth knowing: write-behind, where writes hit the cache first and flush to the database asynchronously. It can offer faster writes, but it usually has weaker consistency, and a cache crash before flushing can lose data, so it fits write-heavy workloads with lower-risk data like analytics events and counters.

Event-driven sync closes the gap TTL can't

Neither classic pattern helps when data changes outside your app: a batch job, a database administrator (DBA) running a manual update, another service writing straight to the database. Event-driven invalidation reacts to the change itself instead of waiting for a timer.

Keyspace notifications for cache-side changes

Redis ships a building block for this. Keyspace notifications let clients subscribe to publish/subscribe (pub/sub) channels and receive events when keys change, so one app instance can evict its local copy the moment another instance modifies a key. Pub/sub is fire and forget, though: a client that disconnects and reconnects misses every event in between, so consider adding a reconciliation path. Expired events also fire when Redis actually deletes the key, not the instant the TTL theoretically hits zero, so there can be a delay.

Change data capture for database-side changes

For changes originating in your source database, change data capture (CDC) is the heavier-duty answer. Log-based CDC tools such as Debezium read the transaction log the database already maintains for replication, convert committed row changes into events, and let a consumer invalidate or refresh cache entries in near real time. Debezium provides at-least-once delivery: it is designed so committed changes are not missed, but a record may arrive more than once, so consumers have to handle duplicates.

Uber's CacheFront shows both the power and the sweat involved. Its earlier design paired TTLs with CDC and hit read-your-writes problems, where a row that was read, cached, and then updated could keep serving stale values until invalidated or expired. That kind of race shows why cache writes need freshness checks instead of accepting whichever value arrives last. Even then, the eventual consistency of TTL plus CDC became a limiting factor in some cases, so Uber layered on a write-through protocol where each node validates freshness with the database before serving. That system later saw a higher cache hit rate with longer TTLs.

Across these systems, event-driven signals often do most of the freshness work while TTL stays on as a safety net; the ordering, dedup, and replay logic is where the engineering hours go.

Picking a strategy: write volume & staleness tolerance

You probably don't need Uber's three-mechanism setup. Start with how much staleness your data can handle and how write-heavy the workload is, then map to a pattern. As general guidelines, depending on your workload:

• Read-heavy with data that can handle staleness: cache-aside with TTL usually works. A product catalog that's a few seconds stale is invisible to shoppers.
• Read-your-writes required: write-through fits data where correctness is the point, like account balances and payments. Users tend to handle write latency better than read latency, so the double-write cost lands in the right place.
• Write-heavy with lower-risk data: write-behind absorbs write bursts for counters and event streams, in exchange for a data-loss window on cache failure.
• Data changes outside your app, or staleness costs money: add event-driven invalidation on top of whichever pattern you run, and keep a conservative TTL as the backstop for missed events.

These are starting points, not laws, and TTL length alone can shift a strategy from fine to harmful. A five-minute TTL that's harmless for a homepage would be rough on live inventory, so segment TTLs by how fast each type of data actually changes.

Keep your cache fresh with Redis

The teams getting cache consistency right layer their mechanisms: a pattern that matches the workload, a TTL backstop, and an event-driven freshness signal. The hard part is deciding how much of that third layer your team wants to build and operate itself.

Building that third layer by hand means running CDC connectors, managing ordering and duplicates, and writing your own race-condition logic. Redis Data Integration (RDI) builds that layer for you. RDI is a change data capture system that tracks changes in a source database and applies them to Redis, first through a full snapshot, then by streaming changes as they happen. In supported RDI deployments, updates from your source database can reach the cache within a few seconds, depending on source load, connector configuration, and network conditions, which helps prevent stale data without custom pipeline code. You define how relational tables map into Redis hashes, JSON, or streams in configuration files. Delivery is at-least-once, and RDI preserves the change order per source table, so updates arrive in the sequence they happened. Supported sources include major relational and document databases, and RDI runs self-managed or fully managed in Redis Cloud on Amazon Web Services.

Redis is a fast, in-memory, real-time data platform that serves your cache reads and keeps them in agreement with your source of truth.

Neo4j Virtual Graph is now in public preview

Earlier this year, we launched Neo4j Virtual Graph in private preview. The response was clear: enterprises want to create knowledge graphs from the data they already have, and they do not want to move data to do so. Today, Virtual Graph moves to public preview, open to every Aura customer, with support for Snowflake, Databricks, and now Google BigQuery. Virtual Graph is built on a flexible foundation designed to extend to additional sources.

Why it matters

Agents are only as good as the context they reason over. GraphRAG outperforms flat retrieval because a knowledge graph captures the relationships that flat retrieval misses; multi-hop questions, such as “which accounts share a beneficial owner?” need a graph to navigate multiple hops out. That set of relationships is what we call a knowledge layer: the context an agent needs to answer a question like that one. Most of that data sits in warehouses and lakehouses that were never built to serve it in graph shape. Copying it out creates pipelines, duplication, and a second system of record susceptible to drift. Virtual Graph removes that trade-off with a zero-copy architecture: your data stays where it is, governed by your existing controls, and you query it as a knowledge graph.

What you get

Connect Virtual Graph to a Snowflake, Databricks, or BigQuery source, and you have a working graph in minutes. Built-in AI tooling proposes a graph model from your tables: which entities become nodes, which keys become relationships (even when foreign keys are not declared), and which columns become properties. You review, adjust, and start querying in Cypher.

Behind the scenes, your Cypher graph query is translated into optimized SQL and pushed down to the source engine. The translation is deterministic, not LLM-driven: the same query produces the same SQL every time, with predictable performance and cost. The heavy lifting runs on compute you already pay for, inside the governance perimeter you already trust, and results come back graph-shaped over Bolt. They are served from a database that looks and behaves like a regular Neo4j database, so all our existing tooling in the platform and client-side can work with it directly.

Three components do the work. The graph data model, optionally generated by AI from your source schema; you own it, and you edit it. The Cypher-to-SQL translation layer, which pushes queries down to the source. And the graph compute layer, which handles the patterns, traversals, and paths that SQL alone cannot express efficiently.

When to use it

Virtual Graph and a graph stored natively in Neo4j solve different problems, and mature graph estates run both.

Reach for Virtual Graph when the data cannot or should not move: too large, too governed, too operational. It suits workloads that tolerate warehouse-grade latency: GraphRAG over reference data, batch enrichment, analyst exploration, and agent workflows that think in seconds. Some estates use a virtual graph as the first step toward a native one; for many, it is the permanent operating mode. Both are valid end states.

Reach for native Neo4j (AuraDB or self-managed) when the workload needs millisecond traversal: real-time decisioning, online fraud scoring, live identity resolution, or continuously updated graphs with ACID writes.

The simple rule: agents that think in seconds fit Virtual Graph. Agents that act in milliseconds want the graph stored natively. And when a hot subset needs both, you will be able to materialize it into native Neo4j without redoing the modeling work.

What is next

Public preview is a step, not the destination. On the roadmap: the ability to materialize a virtual graph into native Neo4j, keeping your model and access setup while unlocking full Cypher, graph algorithms, and native performance on the subsets you choose; federated queries that span an AuraDB graph and a virtual graph in a single Cypher statement; graph algorithm support over virtual graphs; and more sources, including operational databases; and support for self-managed Neo4j deployments.

Get started

Virtual Graph is available today in the Aura Console for Snowflake, Databricks, and Google BigQuery. Connect a source, generate a model, and run your first Cypher query in minutes. Public preview is free to use. We plan to begin billing on September 1st at parity with AuraDB Pro pricing, with general availability to follow shortly after.

Five steps to a first query: connect your Snowflake, Databricks, or BigQuery credentials in the Aura Console; review the AI-proposed graph model and adjust nodes, relationships, and properties; create the virtual graph; connect over Bolt or query directly in the console; explore the results with your existing Neo4j tooling. Watch the demo below. Tell us what you build.

Welcome to the latest installment of No Dumb Questions, the series where Stack Overflow’s least technical writer asks technical staff the simple questions people are too afraid to ask. Phoebe is joined by Michael Foree, Stack’s Director of Data Science, to learn about what’s causing the latest AI adoption bottleneck, what exactly AI context is, and why context engineering is so important to the future of our AI systems. Plus, Michael shares what we can all do to become better context engineers.

Phoebe Sajor: Hey Michael, thanks so much for joining me for the latest No Dumb Questions. Soooo….everybody is talking about AI all the time, but I’m hearing we've hit this wall with AI where we're not getting as much adoption and evolution with the tools. I've been hearing the words “AI bottleneck" a lot. So what is the AI bottleneck and why do we care?

Michael Foree: I've also picked up on an adoption blockage in the conversations that I've had. A couple months ago I went to a conference and I surveyed attendees about what they do with AI. The attendees of the conference were inherently technical but there were also some ardently non-technical people there. I got this really interesting mix of CTOs, engineers, and analysts, but also some graphic designers and Project Managers, sharing with me about their usage of AI. This was six months ago and was a great opportunity to talk and learn.

One of the things I heard a lot was that AI is competent and capable of doing most of the things that people want it to do. Where it seems to lack is in its connectivity with the actual things we work on everyday.

One particular example that I heard was that AI is capable of reading and responding to an email, but what it lacks is the actual context of the email exchange. It can understand that, hey, here's the email that someone sent me. But it doesn’t grasp the context about who that person is or the conversation that I've been having with them in the email thread, in Slack, and in various meetings. It’s missing everything around the email itself. So, as I use AI to reply to emails, I have copy-paste content from all the other places I work that is relevant to this conversation, just so my favorite AI can respond to this one email.

But once it has that context, then I can say something really simple like, "Hey AI, how do I respond to this?" And it’ll spit out a response. But I'm still going to spend one or two back and forths editing that response. And only then am I going to copy it from my AI tool into my email program. And then finally, after all of that, I'm going to hit the send button on my email.

PS: Seems like a lot of work for one email.

MF: It is! And what’s more, AI is perfectly capable of doing each of the individual things I listed. What a stand-along AI tool lacks is the right context. And when you really think about it, there’s a lot of context that goes into a single email reply, and for an AI to perfectly and autonomously reply to an email, it needs all of these connections. Usually, it doesn't even have a connection to, say, the email tool you’re using, which is the very basis of being able to reply to an email.

And there's this very small spot where a human being needs to have an opinion and say, "Close, but not quite.” And then the AI tool needs to be able to fix the issue and have the human give a thumbs-up and say, “Okay, now I'm good. Go ahead and send.” That is also missing in the AI workflows that currently exist.

PS: That’s the human-in-the-loop part of AI workflows that everyone is talking about!

MF: Exactly. What these conversations with everyone from CTOs to graphic designers told me, Phoebe, is that there's an issue of context engineering right now in AI. What AI is missing is the right context. Right now, out of the box, AI can’t say, “Oh, this is the particular context of why the human decided this email needs to be responded to and it needs to be responded to right now.” Understanding human judgement and context is massively important for getting something as simple as an email right. It can’t say, “Here's some other adjacent conversations that are relevant to this email, I’m going to make sure I include that context in my reply.” Even your favorite AI, who you interact with everyday, simply doesn’t have access to all of that information. It’s siloed across all of the countless tools we use in our work.

To get something like an autonomous email workflow to work takes real effort by a human. There’s certain setup and implementation that the human has to do to say, "AI tool, I want you to have access to all of these emails." Cool, now it has the email context. But it also needs to have access to your briefs and RFPs and whatever else you’re working on. So the human has to go and connect their Google Drive and say, “I want you to have access to these documents.” Okay, cool, but what about all of the related information happening internally? The human has say, “I want you to have access to Slack channels.” Okay, now it has all the relevant context. But even after all that, the setup isn’t done. Now, the human has to give their AI permission so it can even send an email. They have to go into this workflow and, “I also want you to have the ability to hit the send email button when I tell you to.”

Personally, as a user, I don't know if I want to go to that much trouble to set all this stuff up because I might use this, you know, once or twice a day, at most—maybe even only once or twice a month. Is it really worth going and setting all this up to write the occasional email?

PS: Yeah, seems like it would be less work just to write the email yourself.

MF: I mean, if you were to calculate out the effort to value tradeoff, eventually the time to set everything up will payoff. But a lot of people are asking ourselves whether it will pay off enough to justify the effort today? And in an Enterprise setting, the effort of setting up new software is even higher. I think that’s where the bottleneck in AI adoption is happening. Do I feel that I benefit enough today to be willing to go and set all this stuff up? Most of the time the answer is no. We think to ourselves, “I actually have a lot of problems, a lot of other stuff I have to work on. I don’t have the time or patience to manually set-up an email responder. I'm not going to solve this problem.” That's the type of thing that I kept finding.

PS: I didn't realize that it was so much context engineering that has to go into something as simple as writing an email. Plus, with all the data enterprises have, it seems the token cost of working with that kind of data is way more than what it would cost just to write the email yourself. Do you think the expense plus the difficulty of working with enterprise amounts of data are part of the AI bottleneck?

MF: Yes. If you think about the context engineering part of what I described, clearly you’ve got to look at lots of different data. If you think about every piece of data that is involved in an email, and every action we have to take to formulate a response to that email, you’re looking at a lot of different data that we, as humans, have to ingest. In AI, that all becomes context engineering.

Oh, Joe sent me an email—that’s a piece of data. Now, based on the context I already have about how important Joe is, I know I need to respond to it right away. That’s another piece of data. When I go to look at the email that he just sent me, I realize there's actually an entire email thread here I’ve forgotten about. There’s some data I need to ingest. Now, I'm going to look back and forth through the entire thread. Now, for humans, that feels pretty straightforward. But if you really look at it, there's a judgment call that needs to be made. Do I look through the rest of my emails for emails from Joe and determine if each of those are relevant to the response that I give? Are there specific keywords that Joe is talking about that help me decide what needs to be included? Oh, he wants an update on project XYZ. As a human I might think, “Okay, where is the latest update on project XYZ? Is that found in Slack or Jira or is there a meeting that I had about project XYZ?”

As a human, I might be able to quickly identify, okay, this is the best place to know about project XYZ. But, for an AI to go and find each of these different things…they typically get overwhelmed or distracted with irrelevant context. In those situations, there's a distraction component that becomes pretty dominant, and that is related to cost. And then, on the other side of the distraction issue, there's still missing information that the AI doesn’t readily know about. You can think about a distraction like this—if I tell my AI, “Hey, I want to jump over this log and by the way, there are blueberries over there. Now tell me, how do I jump over the log?” It's going to think that blueberries are relevant and it's going to give me some answer about blueberries. Likewise, it's not going to know to ask if there's a puddle or how big the log is or something like that unless it's trained to look for that type of thing. It has to be specifically trained to care about the size of a log. But if it doesn't know to ask that, it's going to guess. It's trained to guess.

So, in the case of the email to Joe, it'll get distracted by all the irrelevant things about project XYZ I have in my docs and all the emails mentioning Joe and all the Slack messages with my team about the project. It’ll ingest all those distractions and give me a response that doesn’t fit the specific context of Joe’s email. Those distractions cost time and tokens.

The latest LLMs being rolled out are doing better at kicking out distracting content. They’ve gotten better at knowing when to say, “Yes, and…” They’re asking more follow-up questions. But they had to be trained to do that. You know, three years ago, they were abysmal about knowing when to guess and when to ask for additional context and when to not get distracted. So, they're getting a lot better, but there's a side component here. They're getting better on information that's publicly available.

So, jumping over a log. Sure, you train it to ask, “How big is the log?” But that is very different from private proprietary information about a specific company and their specific procedures. Most of these AI labs are not able to get access to our proprietary information to train on our specific procedures. And you wouldn’t want them to, because it’s proprietary!

But that is a challenge for companies working with AI. For it to be really useful, AI needs to understand their specific processes. This is something we're working on with Stack Internal right now. It’s helping to alleviate some of these issues with private proprietary information. We're not training any LLM or any AI on private proprietary information. We’re just creating that knowledge connector so that AI has the context it needs. And we’re creating that context engineering by bringing in humans to validate the proprietary information that AI is using. Now, the AI can say, "Hey, somebody asked a question about this. And based on what I can see, I'm pretty sure that the answer is this. But you're actually an expert in this particular process. Can you confirm that I answered the question correctly?” And now a human with expertise can come in and say, "Actually, in this situation, in this scenario, this is what you're actually supposed to be doing."

This taps into some of the holy grail context engineering that the big AI labs simply can't buy because no one wants to sell their proprietary data to them. It's confidential, it's part of my secret sauce as an enterprise so, no you can’t touch it.

PS: You mentioned you spoke with a wide variety of folks from analysts to graphic designers. Would you say, across industries and across people, is context engineering the problem for everyone? You know, whether I'm the director of engineering or a contractor doing graphic design, does the AI adoption bottleneck boil down to context and distraction?

MF: The answer is yes, but in unique ways for different industries. In the interviews that I did with non-technical people, what really came to bear was the lack of connected tools available to them. So, for example, one person took a picture of their living room and they wanted to redesign and change the paint or the drapes or rearrange things. So they took a picture and they submitted it to their favorite AI and it was able to correctly identify what the right color combinations would be for the drapes and paint. It could identify how they could rearrange their furniture for a better walkway. And it could make recommendations on whether you should paint this or you should move this piece of furniture. But there was a breakdown in the connectivity of the tools where they could not rapidly iterate.

So they had to get up from their computer, take a picture on their cell phone, send it to themselves by emailing it to their computer and then ask all of these questions on their main computer. And you know, in the interview that I had with them, I didn't want to be like, well, wasn't there an app for that? Because there is an app for that. But it made me realize that the technology works, but it's not properly connected. This kind of disconnect raises a specific problem when it comes to autonomous agents. Like, I want to repaint my walls, give me recommendations on what colors work. I’d probably want it to show me a paint store near me that has the right color paint. And if it was being really helpful, it would let me buy that paint without leaving my AI tool.

That shouldn't be rocket science, right? Google Maps has already solved the problem of, “Where am I and where are the paint stores near me?” And some paint store out there is going to be grateful to sell paint to a bot. So it’s a win-win for everyone. So why isn't this a thing we can do? Why haven’t we solved for this?

In my assessment, this lack of connectivity is not a particularly difficult problem to solve. What it comes down to is that people need to do the work, but we aren’t incentivized to actually solve and connect each of these different things. It goes back to what I was saying about setting up autonomous email replies. If you’re only doing it for one or two emails a month, how much are you actually benefitting today? We also have to decide if all the work involved in buying a bucket of paint with AI—connecting it to our Google Maps, giving it our credit card information, sharing context on our favorite local paint store—is going to pay off enough to be worth the work.

To illustrate this, you can, for example, use AI to create a grocery list. But instead of writing a grocery list, you can say things like, “Here are the people in my family. Here are their allergies. We need breakfast, lunch, and dinner. And I don't want to spend too much time making the meals because I’m busy.” You can get an AI to create that for you, but as soon as you say, “Go ahead and order all the food,” suddenly you’re at a sticking point. There are a lot of grocery stores out there that have online ordering if you use their app. But they're not going to expose their API in that way. To them, exposing their API just so your AI can buy you groceries is a security risk. And your single AI shopper doesn’t incentivize anybody else to go and buy from their store so what’s the point? I think Instacart is the only place that gives any kind of incentive. If I make an app that does everything I just described, then get a small kickback as the app creator when people use it. But very few grocery stores incentivize someone like me to make an app like that. So nobody bothers to make one because why would they? I can just work my normal engineering job and…

PS: …go to the store yourself.

MF: Yeah, exactly. Even if an app like that would be really useful to both shoppers and the grocery stores, no one wants to take the time to do it. And for anyone that's reading this, you could go and probably vibe code an app that does exactly what I described and then connect it to the API of your local grocery store. It's not rocket science. You could probably do it entirely for free. Post it. I'm sure people will use it and love it and maybe that'll be something that'll get started. Source this out and build it because the future is grand, right? We don't have to do grocery shopping and building lists individually. There's a better way. We can do a better job, guys. The future is now. Let's do it.

PS: Yeah, the future is now! I recently wrote a piece about being a builder and artisan in the age of AI, and to your point, it seems like we've entered a phase where anyone can build anything faster than ever. And that really does open us up to endless possibilities, right? Have you found that, from a technical perspective, we're becoming more creative? There's some discussion with AI that it’s making us less creative. But I think, anecdotally, the opposite has been true for me. In your conversations with people, have you found that AI has made people more or less creative?

MF: Yeah, Phoebe, I agree with you that it’s giving people a new creative outlet. I'll give you a personal example—I really like photography. I started with film, black and white photography because I could really sink my teeth into it and have a lot of fun. And then digital came, and then pictures on your phone. Black and white film photography is really, really, really difficult to get into. I've got a camera and I never use it. And the pictures I take on my phone are completely different from the pictures that I took with my film camera. And so in one regard, film photography has become less popular because of digital. But in another way, a whole new art form has been created and exploded. Now we can make cheap videos that you post on YouTube or just take a little selfie whenever you want. The concept of a selfie was invented after I started photography.

I think with coding or creating things with AI, some parts of traditional artistic expression are going to fall to the wayside. It’s actually kind of sad. But I think other forms of expression are absolutely being created. And because the bar is lower, it allows all these other people to use code as a means to their ends. It’s no longer about writing code just because I want to write code. It's about using code because that's going to help me accomplish some goal, or create something that’s never existed before. And yeah, Phoebe, I don't know your coding background, but you probably create an app that does your grocery shopping in an afternoon.

Personally, I took a class years ago on using AWS cloud service. I'd never hosted a website before but in an afternoon I went from barely being able to spell AWS–

PS: Ha!

MF: –to launching my own hosted website. Now I can have my own personal website on AWS doing whatever it is I want, not because I know how to manage a server rack and network but because AWS made it so much simpler with the cloud. AI is doing the same thing in my opinion.

PS: It seems like we have a lot of powerful tools at our fingertips, but there’s still this problem of context. For our readers, what would you say is step one for context engineering? How can we use context engineering to make AI more useful?

MF: I'm going to take a note out of my elementary school children's learning curriculum. They're teaching kids to observe and wonder. You look at the world around you, you pause, and then you ask, “What's going on here? What is cool or different or unique or noteworthy about whatever it is I'm looking at?” It forces the children to take a breath, pause, and then come to conclusions about things. That's relevant here when thinking about context engineering. Again, it’s not particularly rocket science. But you do have to think about what’s happening and what context you need to send that email to Joe. You do need to stop and think out loud, “Okay, if I were to write this email, what are the sources of information that I would consider and not consider?” This is something that’s automatic to humans and not to AI. So when you’re context engineering, you have to observe and then you have to wonder. You have to force yourself to think through everything you’re doing and say, “I might want to know about this information or that information or this other information.” Then you need to write that down and then go to your next email and do the same thing.

And as you do this, you're going to get a list of things your email responder app should have access to—different sources and content. And you’ll figure out where the problems are—my email responder also needs to reject extraneous information that looks like this, this, this, and this.

It's really quite elementary. But the part that makes it hard is that we as humans do it automatically because we do it so often. AI is not going to do this automatically unless you teach it. So you have to start by pausing and thinking through things—why would I want to kick out some information? What's going to be distracting? Then go ahead and build it. Vibe code it. And then you have to test your context engineering. I'm gonna tell my AI, in a mock situation, to respond to this. Okay, well, it’s response is not what I was expecting. I wonder where it got the idea that it should look at this and not this other thing. This becomes its own kind of creative problem-solving. You have to scratch your head a little bit and try and work through it. How do you kick this part of distracting information out? Or, maybe it's that you forgot to give my AI this specific piece of context it needs. And then, when it works, you can really have some creativity. How can you jazz it up so it’s really useful? How can you expand this and make it cooler? Maybe you want to document somewhere that you sent an email about such and such—that’s helping you build an even better context architecture.

But context engineering starts with observing and wondering. That's what I would encourage our readers to do. You have to stop and think about what you're doing.

PS: And especially now in this age of AI, I think a lot of us are not stopping to think. It seems like context engineering takes a bit of a reversal of everything we’ve learned about AI. We've been going so fast. Don’t think, just do. And now the pendulum has swung the opposite direction and, oh actually, we need to stop and think a little bit.

MF: Exactly.

PS: You’ve had the chance to talk to a lot of folks about the AI bottleneck and how people are working with AI. What do you imagine is going to happen in the future of AI? What do you think we need to change about how we use AI to actually benefit from it?

MF: I think, based on the conversations I’ve had, one of the sticking points for the future of AI is the polarization of AI. People have these preconceived notions about how it's great and it's going to solve all their problems. It's not. But on the other side, people think it's the worst thing ever and it's going to destroy humanity. It's also not. I think AI is better at some things than humans are. Let's lean into that. I also think AI is worse at some things than humans are. Let's lean into that,too. It's just another piece of technology that, as a society, we're going to incorporate and work with. I don't think it's going to go away. And I think that it's wise and prudent, like any tool that we have, to learn where it works and where it doesn't. Learn to use it when it's the right tool and don't use it when it's not the right tool.

PS: Putting tools in the hands of smart people has always been a good thing for society. Fingers crossed we'll put the right tools in the right hands.

Vector search infrastructure is hitting a cost wall at 100M to billion-scale indexes, where RAM-based HNSW becomes expensive and can bottleneck on memory. HNSW delivers the lowest latency for small-to-medium collections, but disk-based ANN options like SPANN and DiskANN cut storage costs by shifting most data to SSD/object storage and optimizing for sequential or minimal disk I/O. For large RAG, semantic search, and agentic memory workloads, the key trade-off is accepting higher, more variable latency in exchange for dramatically lower infrastructure spend.

How we built a DuckDB transpiler

Overview

As a data practitioner, it’s common that you’ll encounter the use of multiple databases. Maybe you’ll work with Postgres for one task, and ClickHouse for another. The subtle differences between each dialect let errors and inconsistencies slip through the cracks. One slight difference or error can cause your entire analysis to be wrong.

We ship a dialect-subset of DuckDB that transpiles to other dialects, which we call CocoSQL. This article will focus on how we built it.

Why build a transpiler?

Our core driving principle is to Make working with data effortless. As part of our product, we allow our customers to combine visual transformations – such as renaming a column, applying an Excel-style filter, or editing values inline – with custom SQL to fit their every need.

We also want customers to be able to work with their data, wherever it lives. Most of the time that means in a database or data warehouse that we don’t control.

Being able to offer both of these requires building a unified execution layer. We also didn’t want to create our own custom query solution that customers would have to learn. SQL is ubiquitous and extremely flexible.

The only reasonable solution was to use a single ANSI SQL dialect, and create a transpiler.

Why choose DuckDB?

On top of the most obvious reason – performance – DuckDB’s dialect is arguably the most ergonomic and forgiving in the world. It’s runnable locally, extremely well-written internally, and easy to extend. We designed our whole product with DuckDB as the core execution engine for these very reasons.

Ergonomics

DuckDB’s dialect rocks! Common “gotchas” that occur in other dialects, like trailing-comma errors, do not exist. Niceties like function .dot() chaining are built in, and quick shortcuts like FROM table make most operations extremely quick.

Most of these benefits translate nicely into other dialects as well.

Type system & flexibility

DuckDB’s type system is rich, and forgiving. Compare 1 to 1.0, concatenate a number onto a string, cast a messy string to a date – DuckDB does what you’d expect, and without a bunch of hacks or workarounds. The software is designed to work for you, not against you.

Additionally, there are a ton of helpful built-in types. Because DuckDB’s types are a superset of most target dialects, we can always represent the user’s intent locally, and then decide how to lower it into the target dialect’s, usually narrower, type system.

Extensibility

DuckDB exposes its Parser, Binder, and function catalog directly through its C++ API.

With the internals, we’re able to plug right into the Parser, Binder, create our own functions, override existing functionality, etc., all using the native DuckDB types.

What does the transpiler actually do?

In simple terms, the transpiler converts provided CocoSQL (DuckDB) into an equivalent query in the target dialect. Statement keywords, functions, casts, comparisons, offsets, null handling, etc. are converted to produce a result that is semantically identical to the DuckDB statement.

Additional considerations

In certain cases, the transpilation is not enough. Certain databases or warehouses have settings that produce different results for the same query. The most prominent example is ClickHouse, which has about 1 million different “Session Settings”. In cases like these, we create the transpiled query, and then set the correct settings at the session level before executing the query.

How does the transpiler work?

We structure our transpiler at the core of our product. It drives visual interface changes, as well as custom SQL, and custom columns. Anything that you do in Coco Alemana will inevitably be passed through our transpiler.

We extend DuckDB’s internals using C++ to make use of two critical components – the parsed abstract syntax tree (AST), and the Binder. Specifically, we parse the query tree once, maintaining the user’s query structure, then we run it through the Binder to attach types to each expression and column. This comes in handy when specific edge cases are not solvable without knowing the types.

Traversing the AST

We built a base class that gives us structure to walk each node in the AST. It also gives us default implementations that exist for DuckDB. This means that for most extensions, we simply create a new DialectTranspiler, such as RedshiftTranspiler, and only override virtual methods when we absolutely need to.

Internally, the query is structured as a tree of ParsedExpressions, with each node having children, or constants, depending on its type. This allows us to re-walk the tree, and expose methods like std::string ConvertExpression(const duckdb::ParsedExpression& expr) on a per-dialect basis.

Applying type-based corrections

Not all conversions can occur via looking at the parsed node alone. For some dialects, additional checks are needed at transpile time in order to prevent runtime issues that wouldn’t throw an error, but would produce incorrect results. This is where the importance of traversing the Binder comes into play. It allows us to have not only expression-based logic, but also expression-type-based logic.

A perfect example is BigQuery’s finicky modulo support (%). BigQuery doesn’t have a % operator at all – the closest thing is MOD(), which only accepts exact numeric types, refuses to mix them, and throws on a zero divisor. DuckDB’s % has none of these restrictions, so the correct translation depends entirely on the operand types.

Floats

If either side is a float, MOD() can’t be used at all. We fall back to an injected custom function, which matches DuckDB’s IEEE math bit-for-bit:

-- price is a DOUBLE
SELECT price % 2.4 AS result FROM my_table

-- becomes
SELECT CUSTOM_FMOD(CAST(price AS FLOAT64), CAST(2.4 AS FLOAT64)) AS result FROM my_table

Integers

If both sides are plain integers, MOD() works natively. The NULLIF matches DuckDB’s behavior of returning NULL on a zero divisor, instead of erroring:

-- price is an INTEGER
SELECT price % 3 AS result FROM my_table

-- becomes
SELECT MOD(price, NULLIF(3, 0)) AS result FROM my_table

Mixed integers & decimals

If an integer is mixed with a decimal, MOD() refuses to reconcile the two types, so we promote both sides to BIGNUMERIC:

-- price is an INTEGER
SELECT price % 2.4 AS result FROM my_table

-- becomes
SELECT MOD(CAST(price AS BIGNUMERIC), NULLIF(CAST(2.4 AS BIGNUMERIC), 0)) AS result FROM my_table

IEEE-754 conformance

The IEEE Standard for Floating-Point Arithmetic, IEEE 754, defines how floating-point numbers (FLOAT, DOUBLE) are represented, and more importantly for our case, how they’re operated on.

For whatever reason, many database providers don’t have consistent arithmetic logic. Part of this may be due to legacy implementations, performance, or other platform specific decisions like storage formats. DuckDB mostly conforms to this standard, and our transpiler matches its functionality.

Handling null values

Each system has its own way of handling NULLs, in places like LISTS, arguments of functions, offsets, etc. Some systems like ClickHouse can’t even represent NULL when you have a LIST type.

How we chose what to build

Value equality

Obviously the most important single point. This means that the result of a query yields the same cell values, columns, and order (if applicable) across systems.

Order equality

This means that the rows come in the order that is expected based on the input query. This takes into account differences like NULLS FIRST or NULLS LAST defaults, etc.

One-to-many conversion

We realized quickly that we really only need to convert DuckDB into target dialects, instead of having a many-to-many, or a bi-directional solution. This drastically simplified the problem, and made development quicker.

Type encapsulation

Results should come back with DuckDB’s types, no matter which backend produced them. A BOOLEAN column should stay a BOOLEAN – even when the warehouse’s wire protocol quietly erases it. Because the Binder already computed the expected type of every output column at transpile time, we can restore that fidelity at the executor level after the results arrive.

SELECT statements only

We decided that statements outside of SELECT didn’t make sense to replicate, because for the majority of analytical tasks, reading is the whole job.

Testing

To guarantee we get the results we expect, we make extensive use of E2E testing as well as individual unit testing. As it stands, we have over 3,500 test cases that cover all parts of the query – from structure, to functions, types, null handling, division consistency, and more.

Working backwards

This weekend I visited a friend who studies at an art school. She showed me around the studios, walls covered in figure drawings, still lifes, and portrait studies in various stages of completion. It brought back memories from my own school years. I studied at Hyper Island 25 years ago, back when the program was called New Media Design, believe it or not. Different school, different medium, but the same feeling of a place where people are there to learn how to see.

Because on paper, what she does and what I do have little in common. Design solves a problem. There’s a user, a constraint, a business goal, someone waiting on the other end who needs to accomplish something. Art carries no such obligation. Donald Judd said it best in seven words:

“Design has to work. Art does not.”

That difference shapes everything. When I design something, my taste is in service of someone else’s outcome. When my friend paints, the outcome is the painting. Neither is harder or nobler than the other, but they are different jobs. Still, the longer I walked around, the more I noticed how much of her training would make anyone a better designer.

The habits on the wall

In one of the rooms, someone had taped up a printout titled “Good Habits.” Four items: step back to look at your work from a distance, use a mirror or flip your piece upside down, squint to see the big shapes, take a break and clear your mind.

That’s painting advice. It’s also a better summary of senior design behavior than most things I’ve read in design books. Understanding the big picture is often what separates senior designers from junior ones. Juniors polish the component in front of them. Seniors step back and ask whether the screen makes sense in the flow, whether the flow makes sense in the product, whether the product makes sense at all.

And squinting has a direct design equivalent. In my book I write about the one-handed test, which came out of my work at Summer Health. A parent booking a doctor’s appointment usually has one hand free because the other one is holding their child, and their focus is split at best. Squinting at a painting and using an app while bouncing a feverish kid do the same thing: they strip away the details and reveal whether the big shapes hold up. If the hierarchy only works for a calm person with two hands and full attention, it doesn’t work.

Next to the habits hung another printout: “Focus Glasses.” Alignments, angles, measurements, implied lines. Ways of looking, each one training your eye on a specific relationship within the work. This is the essence of craft, and honestly, it’s the skill I fear is going missing as more people design things through prompts. The tools will catch up on some of it. I already use a couple of skills in Claude Code to check that font sizes hold a sensible ratio, that spacing follows a scale. But a tool that checks ratios can’t tell you what to look for when something feels off and the checklist comes back clean. That still takes a trained eye.

David Hoang wrote about this recently in a piece called Training design senses. His argument is that taste and judgment don’t arrive with a job title, they get built through deliberate practice, the way a musician or boxer trains. His first year of art school allowed no color: charcoal, still lifes, figure drawing, and copying the old masters. His advice for interface designers is to do the equivalent. Put a great screen on your canvas and rebuild it from scratch, noting the margins, the spacing, the type sizes, and asking why each decision works. He also passes along a line from a colleague that stuck with me: someone can have twenty years of experience at the same skill level.

Deconstructing the finished piece

The thing that struck me most came at the end of the visit. My friend told me about an exercise where they take a finished painting and deconstruct it. They start with the final piece and step backwards through it, layer by layer, to understand how the result was actually achieved. The image below shows what that looks like in practice: the same composition as a sketch, as an underpainting, and as the finished piece, side by side on the wall.

Samuel Hulick did something like this for products years ago. His site UserOnboard broke down onboarding flows screen by screen, annotating what worked and what didn’t, and it’s still online. At the time it felt like a clever content format. Walking through that studio, I realized it was the same exercise the art students do: take a finished thing and walk backwards until you understand how it was built.

I wonder if we need more of this now, not less. We’ve reached a point where an idea becomes a prompt and a prompt becomes an app in the store in days. The finished result appears almost instantly, and nobody steps backwards through anything. Meanwhile these students spend weeks deconstructing a single painting to understand its structure. One of those groups is building the judgment to know why something works. I’m not sure it’s the faster one.

The one piece of career advice I’m giving everyone right now is to get a side project. AI is changing how software gets made, and the best way to upskill yourself (and your team, if you have one) is to design and build something end-to-end.

In the last nine months I’ve learned more, and at a faster pace, than in any period of my career. That’s because I’ve been building things. I made a web app that helps parents deal with school emails. I built Pegs Out, an app that helps you work out if you should hang your laundry out to try today. And I have a second iOS app which will be out in the next few weeks.

Every one of these has taught me more about how software gets made (and how AI will change that) than anything else I’ve done. You can watch videos, listen to podcasts and read articles, but nothing compares to what you learn by making.

The bar is far lower than you think

Most UX people don’t have a portfolio of side projects because they can’t build software on their own. Until recently, the barrier was just too high. You had to know how to code, keep up with all if the new frameworks and find the time to actually do it. So the only designers with side projects tended to be the ones who already had a technical background.

That barrier is broadly gone. You still benefit from having technical fluency, but the idea that you can’t make software unless you know how to code is simply not true anymore.

The time barrier has shrunk too. Building something small used to take weeks of evenings. Now it might take one or two. I built Pegs Out in two weeks, as a busy parent.

The cost is also relatively low: all you really need is a Claude or ChatGPT subscription, about £20 a month. Nearly every web service (GitHub, Vercel, Supabase, etc.) has a generous free tier, and for a side project you don’t need the paid plans. If you want to ship an iOS app, you’ll need to pay Apple’s annual developer fee, but that’s about it. For people in an industry that pays above average, that’s not an insurmountable investment given how much you get back in learning.

Go beyond Lovable and co.

Most UXers dip their toes into making software by trying something like Lovable, Replit or Figma Make. Type in a prompt and out comes a website – magic! You can get quite far with these, but you’ll quickly reach a local maximum of what you can learn.

Tools like Lovable abstract away almost everything that’s happening under the hood. You don’t have to worry about things like version control, but you also don’t learn about them. You get the result without understanding how things are made.

What I’m suggesting goes a step further: use tools like Claude Code, Codex or Cursor to build the real thing and do more of it yourself. When you’re working a little closer to ‘the metal’, then you learn more. Lovable is nothing like how products get built commercially, whereas using Claude Code to build an iOS app is. Tools that do everything for you aren’t going to teach you much.

How to get started

If you’ve never built a digital product, how would you know where to begin? A proper guide is a newsletter in its own right, but the general approach is...

Start with a small idea. Not a side hussle or a business, just a little utility or helper, ideally one that uses an existing free API. Solve a small problem in your own life. This is what Pegs Out is.

Then resist the magical thinking that tools like Lovable encourage. Don’t open Claude and ask it to build the whole thing straight away. That skips a lot of steps which as a UXer, you know matter.

Do your discovery and a bit of design first. When I started on Pegs Out, I got Claude to research the physics of how clothes dry and best practices for building an iOS app, because I had no idea on either. Some people think the ‘double diamond’ is outdated, but I think it’s still worth doing some of that first diamond so you know what you’re building before you start writing code.

When it’s time to build, don’t ask it to just make the whole thing. Ask it how you should make it: I’m not technical, I want to build this, what are my options? What would a developer do here? What does every project need that I don’t know about? It’ll mention something like GitHub, so you ask what GitHub is, and why you need it. That back and forth is where you learn the most. Build it with AI, rather than getting it to do everything for you.

Don’t let “I’m not technical” hold you back

I think a lot of people get intimidated by this stuff: I’m not a developer, this is too technical. It is slower to get started because you have so many concepts to learn, but every time you get stuck, you can just ask AI what to do next.

I’ve got a computer science degree and nine months of building things with AI, and I still paste screenshots into Claude and ask “what is this, what do I do now?” all the time.

Building software with AI isn’t hard. The difficulty comes from the discomfort of working on something where you don’t know what you’re doing. Just remember that you can take it at your own pace and ask AI as many ‘stupid’ questions as you like – it’s a teacher with infinite patience.

Have the idea, then have a go

Not everyone has the time, energy or interest for a side project, and that’s fine. Maybe now isn’t the right time. But stay open to it and next time an idea for some little app pops into your head, don’t assume it’s out of reach. For years it was, for most people in UX. It isn’t any more.

And unlike a work project, whatever you build is yours. It can be something genuinely useful in your own life, and it’s fun and rewarding to make. You’re making something you want to exist.

Whether you’re an IC or a team leader, this is the fastest way to build the skills that we’re all going to need in the years to come. All you need to do is to take that first step and see how much you learn.

iOS 27 introduces a major AI-powered Siri upgrade that addresses three key weaknesses: world knowledge, conversational interactions, and consistency across Apple devices. Siri can now answer broad questions, maintain conversations with follow-ups and chat history, and provide the same capabilities across iPhone, Mac, Apple Watch, iPad, and, eventually, Apple TV and HomePod. Deep integration with Apple apps also enables personalized assistance based on messages, notes, and email, potentially giving Siri a significant advantage for everyday Apple users.

The playback stack

Video infrastructure was built to store, encode, and deliver files to people watching on a player.

When teams add AI, they bolt on transcription, VLMs, vector search, metadata stores, ffmpeg jobs, and custom orchestration.

• Storage

  S3 · GCS · buckets · archives

• Encoding + delivery

  ffmpeg · transcoding · HLS · CDN · players

• Human playback

  DRM · thumbnails · analytics · watch pages

• AI bolted on

  ASR · OCR · VLMs · embeddings · vector DBs

• Custom glue

  metadata stores · queues · webhooks · brittle pipelines

Built for watching · Extended for AI · Hard to scale

Learning Before Leverage

Teams need room to expose confusion before AI turns it into pressure.

Like many software startups this year, my company went all-in on Claude Code in February. It was a strategic executive decision that manifested as a coordinated mandate that rolled through all our functional departments one by one. We held hackathons to encourage experimentation as the best way to learn was by doing.

My design team wasn’t a stranger to AI tools. We’d used Claude Chat and Figma Make pretty regularly. But working with Claude Code, MCPs, the BuildOps codebase, GitHub, IDEs, terminal, and getting our application to run locally was all new to us. How to fit this new tool into our existing workflow was an unknown that required each of us to try different things. We held co-working sessions just to be able to ask questions live as we each encountered various blockers. We’d share workflows we discovered individually that were born from the experiments.

Yet all the while, my designers felt the pressure to look competent before they had time to become competent. They asked each other in DMs, they’d confide in me during one-on-ones. We were still grasping in the dark. Eventually, things started to coalesce. Just like starting a workout routine, it felt impossible at the start. But through grit and determination, we got to a good rhythm after two or three months. We’re not alone on this journey.

Slack Design Ops practitioner Sheila Kazan saw this in a question designers sent privately: “How do I use AI?” Her team responded by building a place to learn together. Their Builder Days gave designers permission to try the tools with other designers, including the ones who were unconvinced or unsure where to start. Some participants came away with a more honest understanding of their own relationship with AI, including the parts that interested or unsettled them.

Kazan’s story and mine get at something leaders can actually design: permission to be uncertain in front of other people. Once confusion becomes visible, a team can work on it. When it stays hidden, people copy whatever appears to work and hope nobody asks them to explain it.

Organizational researcher Vaughn Tan argues that companies often treat uncertainty as if it were measurable risk. They turn a new idea into a large, visible commitment, then ask teams to defend forecasts they cannot know yet. Small reversible experiments produce evidence before the organization has spent enough money or political capital to make changing direction embarrassing.

The same principle applies inside the project. Anthropic technical staff member Thariq Shihipar uses prototypes to find unknowns while changes are cheap. A plan can capture what the team already knows to specify. A prototype exposes requirements that only become apparent once someone can see and use the thing. Exploration becomes part of writing the brief instead of ending when implementation starts.

These experiments also need candor. Creative director Gemma Phillips makes work by acknowledging uncomfortable truths instead of polishing them away. A team learning AI needs the same permission. Someone has to be able to say that the generated interface is generic, the agent misunderstood the task, the workflow is slower, or the promised savings never appeared.

And one designer learning a tool will only carry a team so far. Phil Morton points out that a designer can learn Claude Code and still hand a different artifact across the same organizational boundary. Design and engineering have to change the workflow together. They need a shared repository, components both sides understand, and review practices that account for generated work. Otherwise the person changes while the production system stays put.

The cost of getting this wrong is already visible. In Noam Segal and Lenny Rachitsky’s survey, 63 percent of designers selected “overwhelmed by the pace of change.” Another 61 percent selected “expected to do more for the same compensation.” AI leverage becomes workplace pressure when every saved hour simply raises the output expected next time.

I want AI to give my team leverage. That requires room to question the tools and test them on real problems. People need time to compare what they learned without performing certainty for one another, and experiments need to be small enough to survive being wrong.

The next time a designer sends “How do I use AI?” in a private message, the answer should be an invitation to bring the question to the team.

What I’m Consuming

Can mindfulness help you overcome your cognitive biases? Stephanie Dorais distinguishes between mindfulness as active noticing and mindfulness as emotional regulation. Different biases may require different interventions: curiosity can reveal information we overlooked, while nonreactivity can help us tolerate the discomfort behind loss aversion. The useful practice is learning to notice an impulse before mistaking it for a reasoned decision.

Decision Fatigue: Why You Feel Exhausted Without Having “Done” Anything Physically. María Sáez examines the cognitive cost of accumulating small choices and is candid about where the underlying research remains disputed. One finding has immediate use: making a concrete plan can quiet the mental loop around an unfinished task before the work itself is done.

IBM CEO Arvind Krishna Has Nowhere to Hide From AI. Tim Higgins catches IBM CEO Arvind Krishna in an awkward squeeze. AI is advancing quickly, IBM’s hybrid-cloud strategy has moved more slowly, and the company’s quantum-computing ambitions remain three to five years away by Krishna’s estimate.

The American E.V. Has Been Crushed. Will It Take the U.S. Auto Industry With It? Matthew Shaer traces how Ford, G.M., Stellantis, and other automakers retreated from electric vehicles just as global adoption accelerated. Returning to profitable trucks and S.U.V.s may relieve the immediate financial pressure, but it also risks leaving the American industry dependent on technologies and supply chains developed elsewhere.

I tried Maxon’s free After Effects alternative, and I don’t want to go back. Paul Hatton has used After Effects for nearly two decades, but Maxon’s free Autograph won him over with GPU-powered motion design, compositing, and 3D tools in one application. The interface feels more like a 3D package than a familiar Adobe design tool, so switching won’t be frictionless. Built-in cameras, SVG extrusion, and native 2D/3D workflows still make it a substantial alternative without another subscription.

The Internet Is Still Fun

From time to time, I will link to cool websites or apps. Here’s a batch for this week to make your Sunday a little brighter.

a11y.quest. Test yourself with 128 questions covering WCAG 2.2, semantic HTML, ARIA, keyboard access, and contrast. Dave Davies built it while studying for the Web Accessibility Specialist exam, while cautioning that the spec-correct answer doesn’t always serve real people best.

NameThatUI. A visual dictionary for those moments when you can describe a UI element but have no idea what it’s called. Entries identify the anatomy and implementation details, while guides translate more than 60 components across plain English, AppKit, and SwiftUI.

clipart.studio. Pick an old magazine from the Internet Archive, snip out anything that catches your eye, and assemble the pieces into a digital collage. You can also import your own PDFs, export the result, or hang it in the site’s public gallery.

Lunacy. This native Mac app resurrects Lunatic Fringe, the interactive space-shooter hidden inside the 1990s More After Dark screensaver collection, complete with an optional CRT shader and a gloriously unnecessary global leaderboard. You’ll need macOS 13 or later and your own copy of the original game module.

Prentis, a new AI research lab focused on computer use models, co-founded by serial entrepreneur Ritankar Das and tech heavyweights Reid Hoffman and Mark Pincus, is in talks to raise $100 million at a $1 billion valuation, according to two people familiar with the discussions.

Launched in April, Prentis is training models to learn how office workers navigate routine workflows across documents and systems, with the goal of building AI agents that can control computers to automate those tasks.

Prentis will ostensibly develop agents tailored to these customers’ needs, such as handling insurance claims and automating customs duty refund exceptions without needing a human to hunt down paperwork.

The startup has already signed contracts worth up to $50 million with several customers, including healthcare management service organization, a manufacturer, and goods and clothing manufacturers, the two people familiar with the discussions tell TechCrunch. This echoes investor materials obtained by TechCrunch that predict an estimated $75 million annualized run rate by the third quarter of this year. (Prentis’ pitch deck notes those figures reflect estimated annualized value based on a contracted fee equal to 20% of savings realized, not recognized revenue, and are “performance-dependent and subject to final execution.”)

By its own account, Prentis says its Hive-32B model outperforms rivals, including OpenAI’s GPT-5.4 and Anthropic’s Claude Opus 4.6, on two computer-use benchmarks: WindowsAgentArena, which measures end-to-end task completion on real Windows applications, and ScreenSpot-v2, which tests a model’s ability to locate the right on-screen control.

In its pitch deck, the company argues its edge comes from running a much smaller, cheaper model. In fact, it claims roughly 10 times lower cost per task than frontier APIs, saying it’s more economical to deploy across everyday workflows. TechCrunch hasn’t independently verified the company’s benchmark results.

The startup is betting that automating everyday office tasks will soon outpace coding as AI’s biggest use case, but it’s a crowded market. Anthropic, Open AI, and Mira Murati’s Thinking Machines Lab are also working on developing AI agents for computer use, one of the sources said. Anthropic has also been acquiring talent in the category directly — it bought the Seattle computer-use startup Vercept earlier this year, folding in its founders and shutting down its product.

Prentis didn’t respond to TechCrunch’s request for comment.

Ritankar Das, CEO of Prentis, is also the founder of Titan, a holding company that builds and operates AI companies. Das, now 31, was UC Berkeley’s youngest University Medalist in more than a century, graduating at 18 with a double major in bioengineering and chemical biology before earning a master’s in biomedical engineering at Oxford.

He founded Titan in 2014 after dropping out of an AI PhD program at Cambridge, where he’d been a Gates Cambridge Scholar. Das has described Titan as an intentional throwback to an old-fashioned holding-company model like Berkshire Hathaway, one that’s funded by its own exits rather than outside limited partners.

Other businesses launched and operated by Titan include AI-powered virtual care provider Tala Health, which raised a $100 million seed round last year, and Forta Health, an autism care startup that raised $55 million led by Insight Partners in 2024. Titan-founded disease prediction company Dascena was acquired by CirrusDx in 2022.

Prentis is a side project of sorts for its two other co-founders. Hoffman, the LinkedIn co-founder and Greylock partner, said last month that he was stepping down from Microsoft’s board after nearly a decade to go “founder mode” on Manas AI, an AI drug-discovery startup he’s also backing; he was an early OpenAI investor and co-founded Inflection AI with Mustafa Suleyman before Microsoft absorbed most of that team in 2024.

Pincus, the Zynga founder, now runs the investment firm Reinvent Capital with Hoffman as a senior adviser, and published a memoir, “Life at the Speed of Play,” last month.

Prentis has already hired more than 25 employees, including researchers who previously worked at OpenAI, Google DeepMind, Meta, Tencent, and Alibaba, according to its website.

Baseten's API for GLM-5.2 has peak speeds of 280 tokens per second and average speeds of around 100 tokens per second. It has more than double the performance of the launch-day API. The company has also built a Fast version of the API, which is focused on reducing latency for coding and agents. It plans to roll out another improvement to its speculative decoding algorithm soon that will further optimize the performance of GLM-5.2.

celeris-1 is a general-purpose language model that delivers near-GPT-5 level intelligence with 15x faster response times. It uses a new inference architecture that uses diffusion techniques, unlocking dramatically better speed while maintaining frontier-level intelligence. The model delivers p50 response latency of 157ms and a throughput of 1,280 tokens per second. A link to a post on how the model was built along with full benchmarks is available.

Nvidia calls for US government policies to support open-weight AI models, arguing this fosters innovation and enhances AI leadership. The company emphasizes how open weights allow researchers to build on existing models, speeding up advancements. Nvidia's push aligns with broader industry efforts to encourage transparency and collaboration in AI development.

Nanbeige4.2-3B is a compact, dense model intended to make capable agentic behavior practical on consumer and workstation hardware, and Laguna S 2.1 is an 118-billion-parameter Mixture-of-Experts model that uses sparse access to a much larger pool of learned parameters.

Full article content is not available for inline reading.

Read the original article →

Waymo plans to offer robotaxis in its own app starting in January, and its contract with Uber ends in May.

Automatic programming, when done well, means to assume the role of Linus, with the AI agents assuming the role of the different maintainers of the different subsystems.

AI is Oil, Not God

I trust the market over ghost stories

Hi friends 👋,

Happy Saturday! The past couple of days have been a technology business strategy nerd’s Super Bowl plus a Pro-Progress Person’s Super Bowl. Whenever you combine commoditizing complements with fighting the precautionary principle, I’m not going to be able to stop myself from writing something.

This is lightly edited and is just my quick thoughts.

Tell our enemies that they may take our lives, but they'll never take our freedom!

Let’s get to it.

Yesterday, Satya Nadella and Jensen Huang, among other leaders in tech, signed and shared a letter in support of open-weight models. Jensen even set up a twitter account to share it.

I loved it, and tweeted that this was Capitalism at its best.

From a business perspective, this is classic tech strategy stuff.

“Smart companies try to commoditize their products’ complements.”

– Joel Spolsky, Strategy Letter V

All of the signatories either make open-weight models models or are complementary to them. Models run on Nvidia chips. They power Microsoft products. a16z and Y Combinator back companies that support, use, and compete with models. Palantir’s Alex Karp has been banging this drum because he wants to disintermediate models in the enterprise. Meta is more open source than the other similar American labs. Hugging Face is where open models live. Dell sells the machines they run on. CrowdStrike benefits if defenders have access to the same intelligence as attackers. Box, ServiceNow, Replit and Perplexity all get better when capable models become cheaper, more plentiful and easier to customize. Google, a new signatory, got off of its slow, bureaucratic ass and signed because Google will be fine either way, and maybe open source models will allow it to serve search ads more cheaply at some point. Mariana Minerals needs to give its comms person a raise, or send a big thank you to a16z.

The more competition there is at the model layer, the less Anthropic and OpenAI are able to run away with the frontier and build a duopoly, the more each of the signatories benefits.

Signing this letter is a selfish act that also happens to align with the interests of consumers and millions of businesses.

And that’s great! That’s capitalism, baby!

Their self-interests are aligned with our interests as consumers and businesses using these models and living in a world in which these models (and the leaders of the labs that create them) exist.

Not everyone agrees with me or the signatories. There are people who believe that it’s better to lock these models down because they might be dangerous.

I think it comes down to whether you think AI is more Oil or God, more “an economically useful commodity that can be scaled and refined to act as a multiplier on everything we do” or “supremely intelligent and powerful being that’s going to wake up and make humanity subservient (if we're lucky).”

I am strongly on the Oil side, as I wrote in The Goldilocks Zone in June 2024. My belief is unchanged since.

In fact, I think it’s super interesting that Satya and Jensen, both of whom are at the very bleeding edge of this technology, wrote in support of open-weight models. Surely, if they saw something that made them believe that these models were going to become conscious and take over the world, they wouldn’t be as openly pro-diffusion, even if they were short-term incentivized to be. I think they know that AI is Oil, too.

But aren’t they going to be dangerous?

My friend Matt Kaufman at Collaborative Fund, who is a lot smarter than me, emailed me after seeing my tweet with just that concern:

Just saw your tweet on the Jensen note and I’ve seen a few others supporting open-weight model proliferation. I’m on the fence here and if you have 2 min to type out a quick reply, I’d be very curious how you’re thinking about it.

I’d definitely prefer a more open ecosystem were it not for risks like models being fine-tuned to remove bio / cyber safeguards, and then resulting in harm. This seems trivially easy and perhaps isn’t a big issue now, but likely will be once 1) the open ones are much more powerful and 2) dangerous actors outside our tech bubble catch on to the capabilities here.

I think there are many domains where LLMs more favor offense rather than defense so this feels like a scary reality and I therefore sympathize with arguments that they should be tightly regulated like weapons.

I quickly banged out a response and I wanted to share it with you, even though it was dashed off in a few minutes and a little messy, because I’ve been meaning to write something and this is close. I’ll add some footnotes on things I’ve thought more about since hitting send.

During Tesla’s earnings call on Wednesday this week, there was a very interesting discussion on the topic of whether SpaceX and Tesla will eventually be merged. When the analyst asked the question, I’m not even sure he was expecting an answer that almost confirms that such a merger is perhaps a question of when, not if. In response to the question about potential synergy between Tesla and SpaceX, this is what Elon Musk and Brandon Ehrhart (Tesla’s General Counsel) said:

Elon Musk

as you can tell from the many collaborations on so many fronts with SpaceX and there’s a lot -- there’s more and more overlap, especially with Terafab, that’s really going to be a gigantic project.

So -- but obviously, we can’t talk about combining companies and that kind of thing on an earnings call. It’s got to be done with the appropriate process. And with that, I’ll turn it over to Brandon, our General Counsel.

Brandon Ehrhart

that’s exactly right. We continue to benefit from our relationship with SpaceX, and we’ve -- they’ve been a great partner, and we have numerous beneficial transactions with them. And earlier this year, we deepened our relationship through an investment and a framework agreement. This will allow us to continue to work with them on projects that Elon mentioned like Terafab and Digital Optimus.

Elon Musk

Yes. And there’s many other things, too. Obviously, you’ve got Grok in the car. So -- and Grok helping drive Digital Optimus. You also got Starlink being integrated into the Cybercab and Starlink will be integrated into all of our vehicles, at least for markets that Starlink is active. Because for a robotaxi situation, you need to have coverage everywhere. And there are many places even in Silicon Valley where the cellular coverage is terrible or sometimes nonexistent, which is surprising for Silicon Valley. But I know when I want to drive to work with the first 10, 15 minutes, I can’t actually do any calls because the cellular connectivity is so bad.

So we can’t have robotaxis getting stuck in these like Bermuda triangles of lack of cellular connectivity. So Starlink with its ability to do connectivity anywhere is actually quite important. So we don’t have robotaxis missing in action. And then obviously, if people are sitting in the car that they’re going to want to do high productivity stuff or entertainment. And with Starlink, you can watch 4K live sports in the car and with very low cost per gigabyte of data that’s really not feasible via the cellular system. And there’s many other situations.

That's perhaps about as loud as a CEO can wink without formally announcing a merger. If I were either SpaceX or Tesla shareholder, I think I would actually welcome the merger.

The reality is Neither of these companies is best understood or valued on their current core operating businesses. Both companies are essentially a portfolios of deep out-of-the-money call options attached to a cash-generating core. Given the cash generating core today doesn’t really come anywhere close to explain their current valuation, investors are likely paying hefty premium for the deep-out-of-the-money call options.

To realize the full potential of such call options, both Tesla and SpaceX may need to deploy gargantuan amount of capex in the coming years. Tesla’s unsupervised FSD or robotaxis, Optimus, SpaceX’s orbital data centers, and frontier model ambition, or their joint Terrafab project…all of these call options seem pretty capital intensive. And it’s really hard to know which of them will take off at what timeframe. Given such uncertainty, it may indeed make sense to have all the call options under the same roof and then add fuel to the ones that actually start to pan out. I have no idea about the probability of success of any one of such call options. In some ways, Elon Musk’s companies increasingly seem to be “Berkshire inverted”. Buffett assembled a portfolio of short-vol cash streams and used float to buy more of them; following the very likely merger of Tesla and SpaceX, MuskCo mostly appears be a portfolio of deep-out-of-the-money calls funded partly by an industrial float but perhaps increasingly more and more be the generosity of the strangers by their willingness to pay such a hefty premium for a portfolio of deep-out-of-the-money call options. I wouldn’t touch it anytime soon, but I would also not bet against the possibility that if a couple of the call options come to fruition, that may “justify” the valuation of MuskCo in the coming years.

Companies like Anthropic and OpenAI claim some AI models are too dangerous to be developed in the open and must be tightly controlled for safety. The rest of the industry says that open source AI models must remain open for people to further develop technologies and build new businesses. China's rapid progress in open source AI models is escalating the fight. US officials are still debating whether to approach regulating Chinese open source models as a national security issue or issue a kind of blanket ban.

Open-weight AI models can expand access, lower costs, strengthen competition, reduce vendor lock-in, and let organizations adapt and run models on their own infrastructure. More than 70 technology companies and organizations argue that policymakers should support compute access, shared AI infrastructure, model evaluation, and targeted safeguards rather than imposing broad restrictions that could concentrate advanced AI among a few closed providers.

AI has shifted the main cost of many small engineering changes from writing code to reviewing and owning it, making generated patches a fast way to price uncertainty, validate scope with concrete evidence, and focus human judgment on maintainability, risk, and long-term ownership rather than implementation effort.

Full article content is not available for inline reading.

Read the original article →

AI is shifting from standalone models into core infrastructure, with AI-native SQL, dynamic model routing, sandboxed agents, and tighter cost control across compute, memory, and storage. The discussion also stresses design-first development, domain-specific agents, and emerging machine-to-machine payments as key foundations for reliable autonomous systems.

QueryTuner analyses and rewrites SQL queries, providing prioritised performance fixes, security risks, and shareable reports without connecting to your database.

Helsinki, Finland — Aiven has acquired Flow AI, a company building infrastructure for production-grade analytical AI agents. The integration of Flow AI technology will accelerate Aiven's product roadmap and make it easier for customers to securely and scalably run production AI applications and agents next to their data.

Aiven acquired Flow AI because the interface to data is changing. Agents are becoming the new medium for businesses to interact with their data, and this shift demands infrastructure to build reliable agents that run alongside production data. By bringing Flow AI’s agent infrastructure together with Aiven’s managed open-source data platform, Aiven is removing the complexity that typically stalls AI projects, allowing customers to deploy agents with the confidence and scalability they already rely on for their data.

“Agents are only as good as the infrastructure serving them - the freshness of the data, the reliability of the context, and the quality of evaluations they are exposed to. The Flow AI team brings deep, hard-won expertise in building reliable agent infrastructure. We are building towards a unified platform that brings together agent-native tooling and production data, so customers can confidently run AI workloads in production. We are delighted to welcome Flow AI to Aiven,” said Oskari Saarenmaa, CEO and Co-Founder, Aiven.

The team behind Flow AI has been developing production AI systems since 2020. Their first product, Flowrite, was one of the earliest and most widely adopted AI productivity tools with hundreds of thousands of users. The Flow AI team then turned their attention to harder infrastructure problems like how to make AI agents act safely and reliably over structured data at scale. Today, Flow AI builds infrastructure for production-grade analytical agents, providing the runtime, data layer, and evaluation tools that enable SaaS teams to embed reliable, customer-facing AI agents directly into their products.

“We've spent years making AI work on real data and were involved in the evolution from the earliest LLMs to today's agentic systems. Aiven has spent a decade building data infrastructure the right way, open, portable and developer-centric. Bringing our harness, evaluation and context work onto that foundation lets us ship agents to far more teams than we could reach alone. I'm convinced Aiven can become a generational AI infrastructure company, and we couldn’t be more excited to be part of it,” said Aaro Isosaari, CEO and Co-Founder, Flow AI.

About Aiven
Aiven is the open data platform for production AI. It runs managed Kafka, PostgreSQL, ClickHouse, Valkey, OpenSearch, and DataHub under one control plane across AWS, GCP, Azure, and other clouds so platform and data teams can ship AI applications on fresh, governed data without vendor lock-in or infrastructure overhead. Every service runs on genuine open source with no proprietary forks so data and workloads stay portable. Aiven is headquartered in Helsinki, Finland, with teams across Europe, North America and Asia-Pacific.

About Flow AI
Flow AI builds the harness for production-grade analytical AI agents: the runtime, data layer, and evaluation tooling that lets modern SaaS teams embed reliable, customer-facing data agents inside their own products. Flow AI was backed by Project A, Lifeline Ventures, and Seedcamp.

Amazon is overhauling Prime Video's home screen through an internal project called Lighthouse, personally driven by Jeff Bezos, replacing the static tile grid with AI-generated, personalized recommendations. The redesign follows a tense meeting last fall where Bezos pushed executives to lean harder into AI. It remains unclear whether AI-driven rankings will genuinely reduce paid placement bias or simply shift it into the algorithm, leaving real benefits to viewers uncertain until testing expands beyond the current small user group.

AI agents are eliminating the traditional designer-developer handoff, letting teams build live together in real time instead of manually translating Figma files into code.

AI agents can work through loops to complete complex tasks. Domains that work around problems with verifiable answers are susceptible to these kinds of loops. This is a new form of brute force. It is like having a million scientists working in a million labs, all taking swings at the same thing. While AI may not be smarter than humans, it is faster. Being fast enough might make up for the difference in intelligence.

This post provides a high-level overview of how AI models learn new skills and behaviors.

Seller is an early experimental app from Meta that aims to be to sellers what tools to edit videos were to content creators. It is free and will also be available in web browsers. The app has AI features that make selling items less of a hassle. Users can sign into Seller with their Facebook accounts, and listings are automatically synced.

Supabase and Grafana Labs have launched a one-click integration that gives Supabase users access to Grafana Cloud observability directly from their dashboard.

Palantir's polished AI and low-code marketing can collide with slow, consultant-heavy implementations.

Hyundai Motor Group has opened UX Studio Shanghai, a dedicated user experience research hub in Jing'an district, completing its four-city network alongside Seoul, Frankfurt, and Irvine facilities. The three-story studio splits into a public Open Lab, with Explorer, Test, and Archive zones, and an invite-only Advanced Lab housing a simulation platform for concept-to-validation co-development.

WhatsApp is testing redesigned chat bubbles on iOS that feature significantly rounder, softer corners, bringing them closer to iMessage and better aligning with its recent Liquid Glass redesign. The update also removes the green outlines around photos, videos, GIFs, and link previews for a cleaner, more integrated appearance. The changes are currently limited to select TestFlight users, with no confirmed date for a wider rollout.

Chart your design system across six axes of maturity: foundations, documentation, governance, adoption, measurement, and AI readiness.

A UX reviewer argues against using interruption pages, a GOV.UK Design System pattern meant to pause users with warnings mid-journey. Interruption pages offer multiple actions, use unconventional button labels, and look like dismissible banners rather than part of the form flow. A standard question with radio buttons is recommended instead, since it forces conscious decisions without needing a visually distinct screen.

The Horniman Museum has partnered with London agency Anatomy on ‘It's in our Nature', a vibrant campaign celebrating its 125th year and new nature-themed outdoor space. Instead of using an existing typeface, Anatomy created a bespoke alphabet from shapes, textures, architecture, plants, and specimens found across the museum and gardens, balancing playful character with readability while leaving room for future community contributions. Individual letterforms were given distinctive, handcrafted animations, complemented by a warm color palette and community-led photography to create an identity that reflects the Horniman's eclectic mix of nature, culture, and curiosity.

Coca-Cola's newly overhauled brand identity, designed by JKR, gives greater prominence to the "Better With" serif typeface, created by Brody Associates in 2023 for a holiday campaign.