Embla 1
Embla 1 is a new reasoning model optimized to run privately and offline on consumer smartphones with sub-2-second latency.
Summary
Deep Dive
- Features 32 layers of 2048 width with 8 routed experts and one shared expert per token.
- Uses 15 sliding-window layers (128 tokens) and 12 sparse attention layers.
- Replaces softmax with ReLU squared in full-attention layers for efficiency.
- Implements 'Engram', a 3B-parameter table hashing the last few tokens for contextual injection.
- Shared K/V caches allow sparse layers to read from the preceding full-attention layer instead of maintaining their own.
- Achieved through multi-stage training including QAT (Quantization-Aware Training), GRPO, and MOPD (Multi-Teacher On-Policy Distillation).
Decoder
- K/V Cache: Memory buffer storing key and value vectors from previous tokens to avoid recomputing attention during inference.
- QAT (Quantization-Aware Training): Training technique that simulates quantization effects during the forward pass to prepare weights for low-precision formats.
- GRPO (Group Relative Policy Optimization): A reinforcement learning method that groups model outputs to estimate a baseline for policy improvement without requiring a separate critic model.
Original Article
Embla 1: A reasoning model built for phones
Embla 1 runs on a phone, private, offline and free per request. This is how we trained it, and how it compared with other small models in our tests at 4 bits.
At 4 bits Embla 1 averages 57.4 on BFCL, IFBench, AA-Omniscience, GPQA and MATH-500 and answers in 1.77 s end to end on an iPhone 18 Pro. In our measurements, the models that scored higher took at least twice as long on the same phone.
Embla 1 is a reasoning model built to run on a phone, private, offline and free per request. We designed it to be fast and to use few tokens per answer, and trained it on 16 nodes of 8× H100.
Architecture
Embla 1 is built to be fast on a phone in both prefill and decode. It has 32 layers of width 2048, with 8 of 96 routed experts and one shared expert per token. We use 15 sliding-window layers of 128 tokens, 12 sparse attention layers and 5 full attention layers. It is built around the following:
- Sevren Sparse Attention. Full-attention layers use ReLU² instead of softmax. About 95% of attention weights are exactly zero and decode skips their values.
- Shared K/V caches. Sparse layers keep no cache of their own. They read from the preceding full-attention layer. K/V bridges build every second-half cache after layer 15 so prefill skips layers 16 to 31.
- Engram. The last two and three tokens are hashed into a 3B-parameter table. Rows they hit form a second input next to the token embedding for every layer.
Training
The model is pretrained on 1.4T tokens followed by a 20B-token extension to 256K context. For post-training we run SFT on 123B tokens. During the last 2,000 SFT steps we train with quantization-aware training (QAT) and all later stages run at 4 bits. For RL we run 100 steps of on-policy distillation (OPD), 800 + 200 steps of GRPO and 300 steps of multi-teacher on-policy distillation (MOPD). Every stage before GRPO distils from a larger model.
| Stage | Tokens or steps | Teacher or signal |
|---|---|---|
| Pretraining | 1.4T tokens | Qwen3.5-35B-A3B-Base |
| Long-context extension | 20B tokens | Qwen3.5-35B-A3B |
| SFT with QAT | 123B tokens | Qwen3.5-35B-A3B |
| OPD | 100 steps | Qwen3.8-27B and Qwen3.5-35B-A3B |
| GRPO | 800 + 200 steps | Verifiers and a judge |
| MOPD | 300 steps | GRPO-trained domain specialists |
Pretraining
We pretrain at 4K context in two phases: 1T tokens without document masking, then 400B tokens with it. We use Qwen3.8's tokenizer pruned to 65,536 tokens. From the first step the model distils from Qwen3.5-35B-A3B-Base with 0.9 × KL on its top-256 logits plus 0.1 × cross-entropy. Its embeddings and output head start from the teacher's top principal components. Muon trains the weight matrices and AdamW the rest with 4M-token batches and a constant learning rate. We then extend the context to 256K over 20B tokens with Qwen3.5-35B-A3B Instruct as teacher.
| Pretraining | Phase 1 | Phase 2 |
|---|---|---|
| Web text, filtered and synthetic | 50% | 43% |
| Code | 19% | 19% |
| STEM, reasoning and QA (synthetic) | 10% | 10% |
| Math | 9% | 8% |
| PDFs | 8% | 14% |
| Multilingual web (10 European languages) | 5% | 5% |
Post-training
SFT. We train on 123B tokens in 32K packed sequences with Qwen3.5-35B-A3B Instruct as teacher on assistant tokens. Reasoning traces in math, code and STEM are capped at 8K tokens. One in ten reasoning prompts gets no reasoning. The last 2,000 steps run QAT in the phone runtime's 4-bit format: round-to-nearest in groups of 64 with a bf16 scale and offset per group.
| SFT | Share |
|---|---|
| Chat | 25.5% |
| Math | 16% |
| Tool use | 14% |
| Instruction following | 12% |
| Code | 12% |
| STEM | 10% |
| Multilingual | 6% |
| Long context, safety, other | 4.5% |
OPD. We run 100 steps with answers up to 8K tokens. Qwen3.8-27B and Qwen3.5-35B-A3B score each answer token by token. Both are pruned to our vocabulary and keep 99.1% of their probability mass.
GRPO. We run 800 steps with 16 answers per prompt and no reference model. Verifiers score math, code, tool calls and instruction following. Another 200 steps add knowledge questions judged by gpt-oss-120b: +1 correct, 0 abstain, −1 wrong. The last 180 of them add a loop penalty of −10−4 per token in a repeated 40-gram of a correct answer up to 0.5. Answers hitting the 16K limit fell from 20.7% to 1.8%.
MOPD. We run 300 steps with GRPO-trained specialists as teachers for four domains: tool use, instruction following, math and abstention.
Results
Score is the mean of BFCL, IFBench, AA-Omniscience, GPQA and MATH-500 with up to 16K tokens per answer and reasoning on. All models run at 4 bits: Embla 1 in the phone runtime's format, the others from their official Q4_K_M GGUF weights. Each model uses its lab's recommended sampling settings. Latency uses 4-bit builds on an iPhone 18 Pro with a 1,024-token prompt and 256 output tokens.
| Model | Mean tokens per answer | vs Embla 1 | Score |
|---|---|---|---|
| Embla 1 | 888 | 1.0× | 57.4 |
| Gemma 4 E4B | 1,577 | 1.8× | 60.4 |
| Gemma 4 E2B | 1,611 | 1.8× | 52.8 |
| LFM2.5-8B-A1B | 3,129 | 3.5× | 58.8 |
| LFM2.5-2.6B | 3,557 | 4.0× | 62.7 |
| Granite 4.2 3B | 5,633 | 6.3× | 61.0 |
| MiniCPM5-1B | 6,461 | 7.3× | 43.1 |
| Qwen3.5-4B | 6,619 | 7.5× | 60.0 |
Embla 1 is clearly behind on math: 76.1 on MATH-500 against 89.6 for LFM2.5-8B-A1B and 90.5 for LFM2.5-2.6B. There is still a lot to improve.
References
- Benoit Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference", CVPR, 2018.
- Rishabh Agarwal et al., "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes", ICLR, 2024.
- Zhihong Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", arXiv:2402.03300, 2024.
- Xiaomi LLM-Core Team, "MiMo-V2-Flash Technical Report", arXiv:2601.02780, 2026.
- Geoffrey Hinton, Oriol Vinyals and Jeff Dean, "Distilling the Knowledge in a Neural Network", arXiv:1503.02531, 2015.
- Zhiqiu Xu et al., "Initializing Models with Larger Ones", ICLR, 2024.
- Keller Jordan et al., "Muon: An optimizer for hidden layers in neural networks", blog post, 2024.
- OpenAI, "gpt-oss-120b & gpt-oss-20b Model Card", arXiv:2508.10925, 2025.
- Edward Yeo et al., "Demystifying Long Chain-of-Thought Reasoning in LLMs", arXiv:2502.03373, 2025.
Building always-on agents
Always-on agents require a robust backend architecture involving event inboxes, persistent memory, and deferred execution to run reliably without supervision.
Summary
Deep Dive
- Event Inbox: A database table ensuring event durability, deduplication, and sequential worker processing.
- Timers: A separate service for scheduling follow-ups and recurring tasks without blocking the main event loop.
- Memory Store: A directory-based system for user instructions, notes, and task progress that persists between agent sessions.
- Permissions: Uses proxies to inject credentials at execution time and auto-mode classifiers to flag high-risk operations.
- Outbox: A message buffer to prevent noise and ensure orderly communication between the agent and the user.
Decoder
- Deferred Tool Call: An agent pattern where a process pauses mid-task to request human input or permission, resuming automatically once the event is returned to the inbox.
- Dreaming: A background process where an agent reviews past session logs to update its permanent memory or consolidate learned patterns.
Original Article
Over the past few months, we’ve seen a resurgence of always-on agents.
Always-on agents are stateful and proactive, meaning they manage context across sessions and take actions without direct supervision. Underneath, they run on the same core loop as other agents. The difference is the surrounding architecture, which is built from primitives that control when the agents wake up, what carries over between runs, and how their work reaches the user.
The resurgence is largely due to recent model advancements, which have made always-on agents more practical to run. First, the latest models have better judgment: they’re more reliable at understanding user intent and knowing when to check in. Second, intelligence is becoming cheaper, with long-horizon tasks benefiting the most.
Customers have started to ask us how to integrate this type of experience into their own products. Some companies want to make their existing agents more proactive, while others are looking to build them from the ground up.
This post breaks down the primitives behind always-on agents and walks through how they fit together.
The cycle behind the always-on agent
An always-on agent is not actually “always on.” This is just a description of the user experience.
In reality, the agent spends most of its time asleep, waking up periodically to do work. Each stretch of work is called a run: the agent receives an event, completes its task, and then saves information about what happened. A run works inside a session, which is the transcript of messages and tool calls. Once the run is finished, the agent goes back to sleep, and the cycle repeats indefinitely.
The core primitives
The surrounding architecture of an always-on agent has five core primitives: an event inbox, timers, memory, permissions, and an outbox. Let’s take a look at each.
Event inbox
Most AI agents only run when the user sends a message. In contrast, always-on agents can be triggered by many types of events, including webhooks, scheduled tasks, and background jobs.
One tempting approach is to have each event source call the agent directly, but that has a few failure modes. For example, two events arriving at the same time can start competing runs, and a crash can lose an event entirely.
This is where the event inbox comes in. Instead of calling the agent, every source sends an event to an inbox, where a worker picks it up.
The simplest version has three components:
- A durable list of events, usually a database table;
- A handler, usually an HTTP endpoint, that writes events to the list; and
- A worker, usually a background service, that reads from the list.
The inbox should track the status of each event, and events can be deduplicated based on unique identifiers from the source. A relational database works well for the inbox, with every event appended as a new row. We recommend starting with one inbox per user, but you can split this further if you need more parallelism.
When a new event comes in, the handler writes the event to the inbox and wakes the worker. The worker reads whatever is pending and starts a run. Events that continue earlier work, such as a reply from the user, resume an existing session, while everything else starts a new session.
From there, the worker passes the events to the agent loop, waits for the run to finish, and marks the corresponding events as done. The implementation details can get tricky, so here are a few best practices:
- The handler should acknowledge events quickly. Some sources, such as webhook providers, treat slow responses as failures. The handler can still run lightweight checks, but any deeper verification, such as checking for prompt injection, should be done during the run.
- The inbox should be the source of truth. When the worker is woken up, it should pull the latest events from the inbox first. That way, if the worker dies mid-run, a new one can pick up the same event and retry.
- An inbox should only have one active worker at a time. This prevents two runs from duplicating work or overwriting each other. When a new event shows up mid-run, you can either push it to the next run, fold it into the current run, or interrupt and restart.
- The worker should batch pending events together. For example, if the user sends several messages in a row, the worker can merge them into a single run, so the agent only needs to reply once.
Timers
Together with the event inbox, timers let always-on agents be proactive. Timers generally fall into one of two categories: follow-ups or schedules. A follow-up is a deadline that the agent sets for itself, such as checking in on a reply. A schedule, on the other hand, is a recurring task that happens on a known cadence, such as a daily newsletter.
Timers live in their own durable store, usually another database table, separate from the inbox. The inbox should only hold events that are ready to be worked on. A follow-up and a schedule can share a similar schema, but a schedule needs to track its recurrence as well.
A timer doesn’t initiate a run on its own. Instead, a scheduler, usually a background service, polls the store at a fixed interval and fires anything that’s due. On each pass, the scheduler:
- Scans the store and finds timers that are due.
- Updates each timer based on its category.
- A follow-up should be marked as done.
- A schedule should get its due time updated to the next occurrence.
- Adds an event to the inbox and wakes the worker.
For the scheduler, you can either build your own service or use a durable execution engine.
Memory
Always-on agents work across many disjointed runs and rely on memory to maintain continuity.
Similar to the event inbox and timers, memory should have its own store. In our products, this is implemented as a directory of files. For an always-on agent, the directories should map to durable storage, usually a database or an object store. This way memories outlive the worker.
As a rule of thumb, memory should only hold what the agent can’t look up again. If something already lives in another system, such as an issue tracker, you should save a pointer instead. Here is a breakdown of the types of memory an always-on agent should store:
- User instructions: Any instructions from the user about agent behavior should be saved in their own file and loaded at the start of every run. The user won’t always be around to steer the agent, so the instructions have to carry over between runs.
- User notes: The agent should save what it’s learned about the user, such as their role, their preferences, or any corrections they’ve given. You can start with one file and move to several small files as the notes grow. In the latter case, only an index is loaded at the start of each run, but the agent can pull the individual files into context when it needs more details.
- Past messages: The agent should keep track of what it’s already told the user so nothing is repeated. It can either store this in a separate file that’s loaded at the start of each run, or have the worker load the agent’s last few messages with the user.
- Task progress: For each ongoing task, the agent should keep a progress file that covers what’s been done and what’s left. Any new run can then read that file and pick up where the previous run left off. Similar to user notes, only an index is loaded at the start of each run, but the agent can pull the individual files into context when it needs more details.
Inside the context window
Before doing any work, the agent loads the system prompt, tool definitions, and memory files. This initial context “overhead” is included in every run, so we recommend keeping it lean and using prompt caching to lower costs.
A new session starts only with this context overhead and the events that triggered the run. An existing session, on the other hand, brings back the messages and tool calls from its earlier runs.
If a session becomes too long, you can either compact the older messages or start a new session. Both approaches lose detail, so anything that needs to persist should be saved to memory along the way.
Permissions
Permissions are where always-on agents carry the most risk. A run holds the user’s access and, by design, operates unattended. For each tool call, the agent needs to decide whether to proceed on its own, block it, or wait for approval.
The components for permissions are the same ones other agents use:
- Credential store: Always-on agents often need sensitive user credentials like passwords or access tokens to be useful. The recommended pattern is a proxy outside the agent’s environment that adds credentials to outgoing requests. The agent sends requests without credentials, and the proxy injects the real credentials before forwarding.
- Auto mode: An always-on agent should be able to work without constant user input. Auto mode uses classifiers to review higher-risk actions and block the ones they identify as dangerous so Claude can find a safer approach. User review, however, is still necessary for the most important decisions.
- Deferred tool calls: A hook can defer a tool call and wait for additional input from the user. The same pattern applies to always-on agents, which may need clarification or approval before taking certain actions. When a call is deferred, the run ends and the session is saved mid-call. The request is shown to the user, and their answer comes back as an event in the inbox. That event starts a new run, which resumes the saved session with the user’s input. While the deferred call waits for a response, the worker can pick up other events, so the agent isn’t blocked.
Outbox
With most AI agents, the response is streamed to the user as it’s generated, so they can follow along as Claude reasons through the problem, calls tools, and writes its answer.
Always-on agents, on the other hand, do most of their work in the background. The user gets the outcome and a few details, rather than the entire step-by-step breakdown.
A run can end in one of three ways:
- The agent sends a message to the outbox,
- The agent decides that nothing needs to be sent, or
- The agent uses a deferred tool call to request permission or additional input.
Most always-on agents can get by with a “send” tool that delivers the message directly to the user. You only need a formal outbox when you want to pace your agent’s messages or allow your agent to recover from errors.
The outbox is a separate table of outgoing messages, analogous to the event inbox. There’s also a dispatcher, usually a background service, that’s analogous to the worker. The dispatcher reads the pending messages, decides what to deliver, and sends those to the user. A request from a deferred tool call can either go through the outbox or be shown directly in the interface.
Beyond delivery, it’s important to tune how many messages the agent sends. An agent that sends too many becomes noisy, while an agent that doesn’t send enough misses things that the user needs to know. We recommend building evals to find the right balance. In practice, it’s better for the agent to start quiet and then earn more of the user’s attention over time.
How everything fits together
Every run starts with an event from the inbox, whether it’s a user message, a webhook, or a timer. When the worker picks up an event, it kicks off a run and wakes the agent in a new or existing session. The agent reads its memory to get a baseline, works within its permissions to take actions, and then updates its memory with anything it has learned. Once the run is finished, the agent pushes a row to the outbox, and the dispatcher delivers the message to the user.
Together, these form a cycle. A run can set a timer that wakes the agent again later, and when the user replies to a message, their answer comes back as an event in the inbox. Each run is brief, but the cycle keeps going indefinitely, and that’s what makes the agent feel like it’s always on.
Building strong safety evaluations
Anthropic published a comprehensive framework for AI safety evaluations, urging developers to move beyond simple refusal-rate metrics to nuanced, multi-turn behavioral testing.
Summary
Deep Dive
- Team Composition: Evals require collaboration between domain experts (psychologists, biologists) and methodology experts.
- Safety: Run evaluations in isolated, monitored sandboxes to prevent models from taking harmful real-world actions during probing.
- Causal Design: Use intervention-based testing to determine if specific system prompts actually influence behavior.
- Multi-turn simulations: Essential for detecting nuanced harm that accumulates over a conversation.
- Grader Validation: LLM-based graders must be benchmarked against human agreement levels; otherwise, you are measuring the grader, not the model.
- Preregistration: Define metrics and success thresholds before collecting data to avoid 'cherry-picking' results after the fact.
Decoder
- Construct Validity: The degree to which a test measures what it claims to measure; a major point of failure in current AI evals that rely too heavily on 'refusal rates'.
- Dual-use: A technology or piece of knowledge that can be used for both benign and harmful purposes (e.g., biological research tools that could also be used to create pathogens).
- Reward Hacking: When a model finds a shortcut to satisfy a test's metric without actually performing the task or demonstrating the desired capability.
Original Article
Full article content is not available for inline reading.
The avalanche that hit math is coming for AI development
AI researchers are increasingly automating their own development, potentially triggering a rapid explosion in model capabilities by the end of the decade.
Summary
Deep Dive
- AI labs are increasingly using AI to perform R&D tasks like pretraining optimization and synthetic data generation.
- AI models have shown an ability to generalize across domains, solving complex math problems without specialized training.
- Automated researchers have access to massive compute and work at speeds significantly faster than humans.
- The author posits that AI research is verifiable enough for autonomous agents to succeed through repetitive testing.
- A potential 'spark' scenario could compress 3-6 years of research into a single year.
- Future AIs will likely be evaluated on their ability to perform complex, multi-stage reasoning tasks in virtual environments.
- The speed of progress will depend on the balance between RSI gains and the increasing difficulty of finding new algorithmic breakthroughs.
Decoder
- RSI (Recursive Self-Improvement): The hypothesis that an AI can improve its own capabilities, leading to an intelligence explosion.
- Millennium Prize Problems: Seven of the most difficult mathematical problems with a $1 million bounty for each solution.
- RL (Reinforcement Learning): A machine learning paradigm where an agent learns to make decisions by receiving rewards for specific actions in an environment.
- Ablation: The process of removing or disabling parts of a model to study the contribution of those parts to performance.
Original Article
Full article content is not available for inline reading.
Announcing kagent 1.0 GA
Solo.io launched kagent 1.0, a CNCF-backed runtime that abandons 'one pod per agent' in favor of suspending idle actors on shared worker pools.
Summary
Deep Dive
- Replaces static pod-per-agent deployments with a dynamic actor system.
- Uses snapshots to suspend idle agents while preserving workspace state.
- Integrates with SPIFFE to provide distinct identities for agents across multiple nodes.
- Includes egress gateways to safely inject API keys per-request, preventing prompt injection of credentials.
- Designed for multi-tenant environments like OutSystems and Swissquote.
Decoder
- Agent Substrate: An open-source CNCF layer for running stateful AI agents on Kubernetes with advanced scheduling and sandboxing.
- SPIFFE: Secure Production Identity Framework for Everyone, a standard for providing platform-agnostic identities to services.
- gVisor: A user-space kernel that runs in an unprivileged process, providing isolation for untrusted code.
Original Article
We are super excited to announce v1.0 of kagent, an enterprise-ready distribution of Agent Substrate for running AI agents and Sandboxes on any distribution of Kubernetes (including OpenShift). The project has evolved quite a bit since our original announcement almost two years ago as we've worked with users to design, build, deploy, and operate AI agents in production.
Originally, we focused on "platform / DevOps" style agents built with frameworks like CrewAI, LangChain, ADK, etc configured with a fixed set of tools. We ran agents as a Kubernetes Deployment (Pod replicas), the same way you'd run a microservice, and for that kind of simple agent it worked fine, but that's not what most people are building now. The first pattern we see is the agent harness, a general-purpose loop around a model that gets a goal, a workspace, and some skills/tools/CLIs, and works out the steps on its own. Those agents run code a model wrote, keep a workspace around between turns, and spend a lot of their time waiting on a model, tools, or a human.
The other pattern we see is the "tool sandbox". A lot of agent work doesn't need a long-running home at all, it needs somewhere safe to run a tool call (run this script, check this code, transform these files, etc) and hand back the result. Claude Managed Agents is a good example, where Anthropic runs the agent loop and the agent's tool calls can run in sandboxes on your own infrastructure (which kagent supports).
Agent Substrate was built for both of these. Solo.io has spent most of this year designing and building it alongside Google. Substrate is a layer on top of Kubernetes that treats agent-style workloads as "actors" that have their own sandbox, state, and durable identity, and it schedules them onto a pool of warm worker Pods that Kubernetes runs. Agent Substrate just reached 1.0 and is now a CNCF project. Substrate focuses on execution, and kagent 1.0 is the layer on top that knows about your users, conversations, agents, and sandboxes. It's open source, and Solo.io is also shipping enterprise releases of both projects.
Solo.io already has customers (and open source users) running kagent 1.0 on Agent Substrate in production at scale. We think it's ready for general availability. Let's start with why a Pod per agent doesn't work.
Why Not Pods for Agents?
A Pod comes with assumptions that work great for microservices. It runs code you wrote and reviewed, its replicas are stateless and interchangeable, it stays busy, and every replica shares one identity (the ServiceAccount). An AI agent breaks these assumptions.
First, the AI model decides what runs. Ask an agent to "fix this doc" and there's a good chance it writes a sed command or a short Python script to do it, which nobody reviewed and which does whatever the model's context (an issue, a README, a tool response, etc) pushed it toward. This is dangerous. Each session needs a real sandbox with egress controls and safe credential handling. A tool sandbox is the same, since its code is untrusted by definition.
Second, state accumulates within the agent's workspace. An agent could clone a repo, install packages, and edit files, so its environment is the work, and a restart, eviction, or node drain throws it away. Third, the agent mostly waits on a model, a person, or another agent, and its Pod holds CPU and memory the whole time, so millions of sessions means a fleet of Pods doing mostly nothing. Tool sandboxes have the opposite problem, where a Pod (or Job) per tool call can spend more time getting reconciled through the Kubernetes API server and scheduler than actually running.
Fourth, an agent session acts for a specific user, and one shared ServiceAccount can't tell those sessions apart for policy or audit. Once you start suspending sessions and moving them between Pods to fix the idle problem, an identity tied to the Pod can't follow them either.
How does Agent Substrate fix this?
Agent Substrate addresses these broken assumptions with a layer on top of Kubernetes purpose built for these workloads. Substrate keeps the worker (a warm Pod in a WorkerPool that Kubernetes runs and scales) separate from the actor(s) running on it. When an actor goes idle, Substrate snapshots its workspace (and optionally its entire memory) and gives the worker back. When a request shows up for an actor that's been paused, Substrate's router finds a free worker and restores the actor into it. If no worker is free yet, the router holds the request for a moment instead of failing it. None of this goes through the Kubernetes API server, which is why it's cheap enough to do on every turn or every tool call.
The other half of the story is security. Every actor runs in its own gVisor or microVM sandbox (Substrate doesn't support plain containers). Each actor also gets its own identity (based on SPIFFE), which follows the actor from worker to worker instead of belonging to the Pod. Requests coming in hit the ingress router, which authenticates the caller and checks policy before it wakes the actor up. Everything going out goes through the egress gateway, and the sandbox holds no credentials. In both directions, the gateways run on agentgateway which understands LLM, MCP, and A2A traffic, so a policy can say which models, MCP tools, or other agents an actor can call. When the agent calls an API, the egress gateway checks the request against the actor's policy and injects the API key or token on the way out, so a prompt-injected agent has nothing to steal. Anything the policy doesn't allow is denied.
Kubernetes still does what it's good at here (running and scaling the worker Pods), and everything you already know about operating a cluster still applies. Substrate handles the part that changes too often for the Kubernetes control plane to keep up with. Agent Substrate is 1.0, it's a CNCF project, and it's what we built kagent 1.0 on.
What does kagent add on top of Substrate?
Substrate is deliberately low level. Its own README calls it "a low-opinion runtime, not an agent SDK", and it's very good at the things it knows about (actors, workers, snapshots, egress policy, etc). What it doesn't know is what an actor is for. It has no concept of a user, a conversation, or an agent, so it can't tell an agent waiting on its user from one halfway through a tool call, and it doesn't know who's allowed to talk to which agent or who owns a sandbox. kagent is the layer we built to answer those questions, and it's how we think you should run Agent Substrate in production. kagent decides what an agent does, and Agent Substrate decides where it runs.
- Bring your own harness. Run kagent's own agent runtime, Claude, Codex, or any container that speaks kagent's A2A contract, all with the same lifecycle, access controls, and sandboxing.
- Declarative agents. Define an agent from a model (OpenAI, Anthropic, Gemini, Vertex AI, Azure, Bedrock, Ollama, etc), a prompt, MCP tools, skills, and plugins, and reuse that definition across agents and harnesses.
- Safe rollouts. Change a prompt or a model and new conversations pick it up, while the ones already in flight stay on the version they started with.
- Auto-suspend. Every conversation suspends when its turn ends and resumes on whatever worker is free when the next message comes in, so an idle conversation holds no compute.
- Human-in-the-loop. An agent can stop and ask a person for approval, suspend at exactly that point, and pick up the same operation when they answer (and a webhook can tell your system it's waiting).
- History, checkpoints, and forks. Conversation history is readable without waking a sandbox, you can checkpoint a conversation and fork a new one from it to try something different, and idle conversations clean themselves up (after seven days by default).
- Users, ownership, and sharing. People sign in through your OIDC provider, every conversation belongs to whoever started it, they can share it with others, and kagent checks all of that before a request reaches an agent.
- No credentials in the agent. Model API keys and MCP credentials are injected by the egress gateway, rotate without restarting anything, and everything else is denied unless an EgressPolicy allows it.
- Sandboxes on demand. People and agents can grab a short-lived sandbox to run a script or process some files, and kagent can also be the self-hosted sandbox for Claude Managed Agents (experimental for now).
- Use it from anywhere. Agents are reachable over A2A and MCP, kagent has its own MCP server for sessions, checkpoints, and sandboxes, and there's a UI (with a view into Substrate), a CLI, and OpenTelemetry traces, metrics, and token usage across harnesses.
Who's running this in production?
OutSystems runs a large multi-tenant platform, which is exactly the Pod-per-agent problem at scale. They use kagent and agentgateway for the APIs, lifecycle management, and gateway controls to plug agents into that platform.
"With the number of agents we expect to support, running a dedicated, always-on pod for every agent simply isn't possible. Agent Substrate gives us sandboxed runtimes that suspend when idle and resume on demand." (Ebenezer Schubert, VP Engineering, OutSystems)
Swissquote runs a sovereign AI stack, with full control over its inference, infrastructure, and AI lifecycle, and Solo's enterprise releases of kagent and Agent Substrate give them infrastructure-level isolation and security along with the low latency their most sensitive agents need.
"Our Codename Diego financial-crime investigation agents and MCP servers are already running in production on kagent GA. In a multi-jurisdictional regulated environment, security, resilience and explainability are non-negotiable." (Andrew Boswell, Head Information Technology, Swissquote)
Solo's enterprise releases of Agent Substrate and kagent are generally available today, with agentgateway as the actor's gateway on both ingress and egress. They add what production teams ask for on top of the open source projects. That covers granular access control over agent resources, agent tracing and evaluations built from real execution traces, visibility into Substrate fleet capacity and failures, and air-gapped installs, along with support from the people building both projects.
Getting started
You need a Kubernetes cluster, helm, and a model provider key (OpenAI is the default). Install the CLI and run kagent install, which sets up Agent Substrate, PostgreSQL, and kagent in one step.
curl -sL https://raw.githubusercontent.com/kagent-dev/kagent/refs/heads/main/scripts/get-kagent | bash -s -- --version v1.0.0
export OPENAI_API_KEY=<your key>
kagent install
From there, your first agent walks you through defining an agent and talking to it, and the docs cover installing with the Helm charts directly (e.g. for GitOps). To size things before you go further, the Agent Substrate calculator estimates wake-up times and the CPU, memory, and storage your agents need, based on our performance and scale testing for GA. If you're running kagent 0.x today, 1.0 is a new API on a new runtime, so plan on a fresh install.
Get involved
kagent and Agent Substrate are both open source CNCF projects, and both are moving fast. If you're running agents on Kubernetes (or trying to), give it a try and tell us where it breaks, what your agents need that it doesn't do yet, and what you'd want from the layer above Substrate.
- Explore kagent on GitHub and at kagent.dev, and join the Discord and the weekly community meeting.
- Explore Agent Substrate on GitHub and at ate.dev, and find its Helm charts in kagent-dev/substrate.
- At KubeCon in Salt Lake City, come to my talk Pods Are Not Enough for AI Agents, Meet Agent Substrate on November 11, and stop by the Solo.io booth to see kagent and Agent Substrate running and talk with the people who build them.
- For the deeper version, read Agent Substrate: From Pods to Sandboxes, and the earlier posts on kagent and Agent Substrate and agent sandboxes.
Google Open-Sources TPU Sync for KV Cache and Weight Transfer on TPUs
Google Cloud's open-source TPU Sync significantly reduces latency for large-scale AI serving by optimizing data movement between TPU memory and host storage.
Summary
Deep Dive
- TPU Sync manages KV cache blocks and model weights between TPU HBM, host DRAM, and storage.
- Targets long-context serving and RL training loops where accelerators often stall.
- Integrates with vLLM-TorchTPU and SGLang-TorchTPU.
- Python wheels are available for JAX and PyTorch, though currently experimental.
- Offers 60-90% reduction in Time-to-First-Token (TTFT) for large models at low concurrency.
- Requires specific compatibility with TPU v7x and v6e architectures.
Decoder
- KV Cache: Key-Value cache stored in memory during inference to avoid recomputing previous tokens in the sequence.
- Disaggregated serving: Decoupling the prefill (compute-heavy) and decode (memory-bandwidth-heavy) phases of LLM inference across different hardware resources.
- TTFT: Time-to-First-Token, a core metric measuring latency from request submission to the start of token generation.
- PJRT: Google’s runtime interface used to abstract hardware acceleration across TPUs and GPUs.
Original Article
Google Open-Sources TPU Sync for KV Cache and Weight Transfer on TPUs
Google Cloud has open-sourced TPU Sync, a native C++ library that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage, and the data center network. GCP also announced the TPU 8tx and TPU 8i Max systems, the project targets two workloads where accelerators often wait on data: long-context and disaggregated LLM serving, and reinforcement learning loops that push updated weights from trainers to rollout samplers.
For prefill/decode disaggregation, Google says median KV transfer latency for Qwen3 235B fell from several seconds to about 230 ms, with average TTFT reductions of 60% for that model and 90% for a 480B model at low concurrency. For prefix caching, it reports up to 90% lower average TTFT on 64K-context workloads.
Developers get Python wheels for JAX and PyTorch, with KV cache manager, KV store, and weight synchronizer APIs. TPU Sync acts as the KV connector for vLLM-TorchTPU through vLLM's KVConnector interface, as the disaggregation and hierarchical-cache backend for SGLang-TorchTPU, and pairs with llm-d 1.0 on GKE for prefix-cache-aware routing. The RL weight-sync module for Pathways and multi-controller JAX is experimental. The GitHub repository, which absorbed the earlier TPU Raiden codebase, warns that APIs are changing rapidly and that only development builds are on PyPI, and recommends pinning to its last-known-good commit. It is validated on TPU v7x and v6e, and the PyTorch wheel requires a matching torch_tpu build that is not on a public index.
On GPUs, similar problems are handled by NVIDIA Dynamo and its NIXL transfer library, LMCache, which offloads KV cache to CPU memory, disk, or remote stores, Mooncake, the KVCache-centric serving architecture behind Moonshot AI's Kimi, and vLLM's own disaggregated prefill connectors. TPU Sync differs mainly in being tied to PJRT and TPU tile layouts, and in covering RL weight transfer in the same library.
Early reactions focused on data movement rather than the new chips, with developer Anand Butani sharing:
TPU sync cutting kv transfer latency 90% matters more to me than the chip names, long-context serving mostly dies on moving cache between nodes.
Another developer posting under the pseudonym @lgoodbookd ran rough numbers on why bandwidth alone does not solve the problem:
8i Max doubles HBM bandwidth to ~17.2TB/s, yet one 64K session on a 235B MoE still eats ~12.6GB of KV cache. That's why open-sourcing TPU Sync (host-DRAM KV tiering, ~223GB/s per v7x host) may matter as much as the new package. Bandwidth helps; capacity is still the wall.
TPU Sync is available under the Apache 2.0 license, and its repository includes architecture documentation, C++ and Python APIs, and examples for single-host and multi-host disaggregated serving and host KV offloading. Until a stable release is cut, the JAX and PyTorch packages are published only as development builds, while the storage tier and RL weight synchronization remain experimental.
HTTPd (CVE-2026-119618): Detecting heap corruption and RCE in Apache HTTP Server mod_http2
Apache HTTP Server 2.4.71 patches a critical heap corruption vulnerability in mod_http2 that could allow unauthenticated remote code execution.
Summary
Deep Dive
- Vulnerability exists in the assign_headers() function of the mod_http2 module.
- Attackers can trigger the bug via specifically crafted HTTP/2 frames without authentication.
- Red Hat and Amazon rate the flaw as 9.8 (Critical) on CVSS v3.1.
- Fix involves a one-line change to ensure headers are written to the correct per-request memory table.
- Servers using only HTTP/1.1 are not affected.
Decoder
- CVE-2026-119618: A specific vulnerability identifier tracked by MITRE for the reported Apache flaw.
- Heap corruption: A software vulnerability where an attacker overwrites memory in the heap, potentially leading to unauthorized data access or code execution.
- RCE: Remote Code Execution, an exploit allowing an attacker to run arbitrary commands on a target system.
- mod_http2: The Apache module responsible for handling the HTTP/2 protocol.
Original Article
HTTPd (CVE-2026-119618): Detecting heap corruption and RCE in Apache HTTP Server mod_http2
On November 11, 2026, Apache released HTTP Server 2.4.71 to fix CVE-2026-119618, a critical severity heap corruption flaw in mod_http2 that Apache’s security advisory said can lead to remote code execution (RCE). The flaw sits in assign_headers(), affects versions 2.4.55 through 2.4.70, and can be triggered over HTTP/2 with no authentication. Red Hat and Amazon have scored the vulnerability 9.8 on CVSS v3.1, though CISA rates it at 8.1.
Two teams found the bug independently, eight days apart. Anthropic's OSS Scanner is an opt-in, model-driven scanner for open-source projects. It reported the bug on October 26, 2026. Anthony Weems and Simon Scannell of Google Cloud Vulnerability Research reported it eight days later on November 3, according to the CVE record.
The day after Apache disclosed it on oss-security, a public proof-of-concept (PoC) was published on GitHub showing the heap corruption, though no PoC has been released for the RCE as of the publication of this blog. Additionally, no confirmed in-the-wild exploitation has been reported, and the CVE does not appear in CISA's KEV catalog.
The bug lives in Apache's own HTTP/2 implementation, not in any application the server hosts. Only servers that enable HTTP/2 are exposed. Affected organizations are encouraged to upgrade to version 2.4.71 or turn off HTTP/2 if patching is delayed.
The Sysdig Threat Research Team (TRT) analyzed the publicly available research and simulated a successful exploit to help defenders understand four key details: who is affected, what has been demonstrated versus claimed, what runtime visibility can detect, and what mitigation steps should be taken until a patch can be applied. They also tested and found that Sysdig’s existing runtime detections caught every post-exploitation step.
Timeline
| Date | Event |
| Oct. 26, 2026, 12:00 UTC | Anthropic's OSS Scanner reports the flaw as ANT-2026-MAPKT6MZ |
| Oct. 30, 2026 | CVE ID reserved |
| Nov. 3, 2026, 15:50 UTC | Google Cloud Vulnerability Research reports the flaw |
| Nov. 9, 2026, 02:54 UTC | Fix committed to the 2.4.x branch as 824beab |
| Nov. 11, 2026, 12:19 UTC | Public disclosure on oss-security |
| Nov. 11, 2026, 12:52 UTC | Red Hat publishes its assessment and RHEL 9 and 10 fixes |
| Nov. 11, 2026, 14:54 UTC | CVE record published |
| Nov. 12, 2026, 09:54 UTC | Debian DSA-6620-1 fixes trixie |
| Nov. 12, 2026 | A public PoC is posted on GitHub showing heap corruption |
Affected Apache HTTP Server versions
The flaw is in mod_http2, the HTTP/2 module of Apache HTTP Server. The same module is released separately as mod_h2 on GitHub and also affected.
While the Apache advisory rates the vulnerability as critical. The CVE record, which Apache maintains as the CVE Numbering Authority (CNA), doesn’t actually carry an associated CVSS score. However, Red Hat and Amazon both rate the vulnerability as critical and score it at 9.8 on CVSS v3.1 and CISA rates it as a high severity vulnerability at 8.1 on CVSS v3.1.
Red Hat rates its own RHEL 9 and 10 builds Important rather than Critical. Those builds allocate each HTTP/2 stream from "a separate allocator, which substantially increases the complexity of exploiting the flaw."
The severity and prioritization of this vulnerability in an organization’s environment depends first on whether HTTP/2 is on.
| Version | Status | Fixed version / What to do |
| httpd versions 2.4.55 – 2.4.70 | Affected | Upgrade to 2.4.71. |
| httpd versions prior to 2.4.55 | Not listed as affected by Apache | No action for this CVE. |
| Standalone mod_h2 2.0.8 – 2.0.42 | Affected. The repository has been archived since June 9, 2026. | No fixed release; 2.0.42 is the latest. Move to httpd 2.4.71's bundled module, or disable HTTP/2. |
| RHEL 9 | Affected (Important) | mod_http2-2.0.26-7.el9_9.1, RHSA-2026:98981 |
| RHEL 9.4 SAP / 9.6 EUS / 9.8 EUS | Affected (Important) | RHSA-2026:98968 / :98954 / :98982 |
| RHEL 10 | Affected (Important) | mod_http2-2.0.29-5.el10_3.1, RHSA-2026:98978 |
| RHEL 10.0 EUS / 10.2 EUS | Affected (Important) | RHSA-2026:98953 / :98979 |
| RHEL 6, 7, 8 | Not affected | No action for this CVE. |
The version number alone won’t confirm exposure, check these additional factors:
- Distribution packages: Vendors often backport fixes without changing the upstream version, so check your vendor's advisory and not only
httpd -vorapache2 -v.- As of November 13, fixes are out for Debian 12 and 13 (2.4.71-1~deb12u1 and 2.4.71-1~deb13u1), Ubuntu 22.04 through 26.10, Alpine, Fedora, Amazon Linux 2 and 2023, and Wolfi and Chainguard.
- SUSE, openSUSE, Photon OS, and several RHEL rebuilds were still waiting on vendor fixes.
- Downstream products: Vendors that repackage httpd, such as Red Hat JBoss Core Services (
jbcs-httpd24-mod_http2) and Red Hat Hardened Images, may be listed as affected on version number alone. Whether you are exposed depends on what the product ships. Check thatmod_http2is actually present and loaded. For example, Sysdig’s scan of the Red Hat Hardened Images httpd image shows it listed as affected, but the image has nomod_http2, so the risk can be accepted. Check the vendor’s advisory for fix status. - Recent upgrades: Servers upgraded to 2.4.67 for the HTTP/2 flaw in May, CVE-2026-23918, are still inside this CVE's affected range.
How the HTTPd RCE flaw works
HTTP/2 carries many requests over one connection as parallel streams and that concurrency is where this flaw lies.
The Apache advisory calls the flaw "Concurrent Execution using Shared Resource with Improper Synchronization" in assign_headers(), and the CVE record lists CWE-362. Red Hat lists CWE-20 instead. Red Hat describes the root cause as "a worker thread writes into memory owned by the connection thread through a shared, lock-free allocator." In practice, two threads can corrupt the same heap.
The fix is a one-line change in assign_headers(). Before the fix, a request with a content type that had already ended had its zero Content-Length header written into the HTTP/2 request's own header table (req->headers). The fix writes it into the per-request table (r->headers_in) instead. The commit message gives no explanation. Red Hat's statement supplies the threading context.
What is reachable
Exposure depends on configuration. Red Hat states that only servers with HTTP/2 enabled through the Protocols directive are affected:
| Path | Does it reach the vulnerable code? | Details |
| Protocols includes h2 | Yes | HTTP/2 over TLS. |
| Protocols includes h2c | Yes | HTTP/2 over cleartext TCP. The public PoC targets this. |
| Protocols set in a virtual host | Yes, for that virtual host | Check vhost blocks as well as the server-wide setting. |
| HTTP/1.1 only | No | According to RedHat, servers serving only HTTP/1.1 are not affected. |
Red Hat's Bugzilla entry says an unauthenticated attacker triggers the flaw "by sending specially crafted HTTP/2 frames." That puts internet-facing, HTTP/2-enabled httpd in the first patch tier. This table does not cover reverse proxies in front of httpd.
HTTPd RCE exploitation: What's been demonstrated vs. claimed
According to Red Hat’s CVE page, public research has shown "a reliable exploit against a default-configured server." However, without a researcher name or links to details, this can’t be verified. As of the publishing of this blog, no other security researchers or vendors have confirmed exploitation.
As of about 16:00 UTC on November 12, a GitHub search for the CVE returned four repositories: a proof of concept, a patch-status tracker, and two detection or assessment tools.
The PoC reports heap corruption, not code execution. Its README describes the impact as "Heap corruption in an httpd child process; potential remote code execution."
Its evidence comes from a lab build:
- Setup: httpd 2.4.70 built from upstream source on a
debian:bookwormcontainer, running the prefork MPM withProtocols h2c http/1.1. - Results: The PoC sent 6,080 requests. AddressSanitizer reported five crashes, including a heap use-after-free allocated in
assign_headers(). ThreadSanitizer reported 31 allocator races. - Fixed build: A 2.4.71 build showed none.
- Trigger: A request that ends without a body and carries a
Content-Typebut noContent-Lengthtriggers the bug. The fix commit changes exactly that code path.
The README doesn't show control of the corrupted memory, and the author has published no exploit.
The bottom line is that unauthenticated HTTP/2 traffic can crash and corrupt the heap of a vulnerable httpd. Remote code execution is possible, but nobody has publicly demonstrated it yet. Treat the crash as likely wherever HTTP/2 is on, and code execution as a credible risk.
HTTPd RCE impact
Nothing publicly disclosed as of November 13, 2026, shows in-the-wild exploitation, a mass-scanning campaign, or a public RCE exploit. The primary sources do establish the following:
- Risk of unauthenticated RCE: Red Hat says a remote attacker who can reach the server could execute arbitrary commands with no authentication. Bugzilla adds that commands run as the httpd process.
- Worker crashes: The public PoC crashes httpd child processes on a vulnerable build.
- Long-lived exposure: The affected range starts at 2.4.55, so servers patched for earlier HTTP/2 bugs can still be exposed.
- Paths with no fix yet: Standalone mod_h2 has no fixed release. Several distributions, including SUSE, openSUSE, Photon OS, and CentOS Stream 10, were still waiting on vendor fixes as of November 13.
Detecting HTTPd RCE with Sysdig Secure
Sysdig Secure customers have three layers of coverage for CVE-2026-119618. Threat Intelligence entries show where affected Apache hosts are running, Vulnerability Management maps to each distribution’s vendor advisory, and Runtime Detections catch what a successful exploit does once an attacker enters a vulnerable environment.
Threat intelligence
Sysdig Secure customers have two entries for CVE-2026-119618 in the Threat Intelligence feed. Each entry lists the affected hosts and running Kubernetes workloads. One entry uses the SBOM to find affected versions of Apache HTTP Server, mod_http2, and the vendor packages that contain the module. The other entry lists the assets where Sysdig Vulnerability Management reports the CVE.
Vulnerability management
Sysdig vulnerability management detects CVE-2026-119618 across your environment. To prioritize remediation, start with internet-facing workloads and packages in use at runtime.
Sysdig vulnerability scanning reports this CVE by distribution and release, and each status follows the vendor's own advisory. Rows move from "Waiting on vendor fix" to "Reported" as vendors publish fixes and the feed picks them up. The table reflects status as of November 13, 2026. On SaaS, vulnerability database updates reach scanners automatically, so new advisories need no scanner upgrade.
| Distribution | Releases | Vulnerability findings |
| Red Hat Enterprise Linux | 9, 9.6 EUS, 9.8 EUS | Reported, fixed in mod_http2 2.0.26-7.el9_9.1 |
| Red Hat Enterprise Linux | 10, 10.0 EUS, 10.2 EUS, 9.4 E4S | Vendor fix published (10: mod_http2 2.0.29-5.el10_3.1), being added |
| Red Hat Enterprise Linux | 6, 7, 8 | Not affected |
| CentOS Stream | 9 | Reported, no vendor fix yet |
| CentOS Stream | 10 | Waiting on vendor fix |
| AlmaLinux, Rocky Linux, Oracle Linux | 9, 10 | Waiting on vendor fix |
| AlmaLinux, Rocky Linux, Oracle Linux | 8 | Not affected |
| Debian | 12, 13 | Reported, fixed in 2.4.71-1~deb12u1 and 2.4.71-1~deb13u1 |
| Debian | 11 (ELTS), 14 (testing) | Waiting on vendor fix |
| Ubuntu | 22.04, 24.04, 26.04, 26.10 | Reported, fixed in the 12 November security updates |
| Ubuntu | 20.04 (ESM) | Waiting on vendor fix |
| Ubuntu | 16.04, 18.04 (ESM) | Not affected |
| Alpine | 3.21, 3.22, 3.23, 3.24 | Reported, fixed in 2.4.71-r0 |
| Wolfi and Chainguard | Rolling | Reported, fixed in 2.4.71-r0 |
| Fedora | 43, 44 | Reported, fixed in mod_http2 2.0.42-3.fc43 and 2.0.42-2.fc44 |
| Amazon Linux | 2, 2023 | Reported, fixed in 2.4.71-1.amzn2.0.1 and 2.4.71-1.amzn2023.0.1 |
| Azure Linux | 3.0 | Vendor fix published (2.4.71-1.azl3), being added |
| SUSE Linux Enterprise Server | 15 SP7, 16.0 | Waiting on vendor fix |
| openSUSE | Leap 16.0, Tumbleweed | Waiting on vendor fix |
| Photon OS | 4.0, 5.0 | Waiting on vendor fix |
| Alibaba Cloud Linux | 3 | Not affected |
| Apache built from source | Official httpd images, Bitnami, copied binaries | Reported through standalone binary scanning, fixed in 2.4.71 |
| Red Hat Hardened Images | httpd | Shown as affected because Red Hat lists it so. The image has no mod_http2, so accept the risk |
Runtime detection
The Sysdig TRT simulated a successful exploit inside a vulnerable Apache server, both inside a container and as a host service. Existing runtime threat detections caught every step without rule modifications, including:
- Failed exploit attempts that crash web server workers
- Commands or a shell started by the web server, including reverse shells
- Tools downloaded and run by the web server
- The web server process replaced by attacker code
A successful attack would give the attacker code execution as the httpd worker. From there, attackers do what they do after any web server RCE: open a reverse shell, download and run a tool, start a miner, steal cloud credentials, or try a kernel local privilege escalation.
Detecting HTTPd RCE with open source Falco
Falco sees syscalls, not HTTP/2 frames. It cannot see the malicious request itself, but it can see what the request causes on the host. That includes a shell or unexpected binary spawned by an httpd process, an unexpected outbound connection, or a write to a sensitive path.
Web servers fork helpers legitimately, so rules scoped to httpd need tuning to stay quiet. The post-exploitation strategies above apply.
Recommendations
Patch
- Update httpd on every server: Move to 2.4.71, or to your vendor's patched package. On RHEL 9 that is mod_http2-2.0.26-7.el9_9.1, on RHEL 10 mod_http2-2.0.29-5.el10_3.1, and on Debian trixie apache2 2.4.71-1~deb13u1. Patching is the only complete fix.
- Restart httpd and rebuild images after you patch: A running server keeps the old module loaded. Verify the package version inside the running workload.
- Replace standalone mod_h2: The project is archived, and its latest release, 2.0.42, is affected.
Interim mitigations if you cannot patch today
- Disable HTTP/2: Apache's advisory and oss-security notice give no workaround. Wiz recommends removing h2 and h2c so the setting reads
Protocols http/1.1. That is Wiz's guidance, not Apache's. It matches Red Hat's statement that HTTP/1.1-only servers are not affected.- Apply the change server-wide and in every virtual host.
- Restart or reload httpd so the change takes effect.
- Test latency-sensitive services first. Clients fall back to HTTP/1.1, which can slow multiplexed workloads.
Contain the blast radius
- Run httpd workers as an unprivileged user: Limit what a compromised worker can read and write.
- Keep secrets out of the server's reach: Use a secrets manager, not environment variables or files the worker can read.
Investigate and audit
- Audit your environment: Find every host and workload running httpd 2.4.55–2.4.70 or mod_h2 2.0.8–2.0.42 with HTTP/2 enabled, internet-facing first. Sysdig Secure customers can use the Threat Intelligence entry and Vulnerability Management findings.
- Watch for crashing httpd children: Repeated child-process segfaults on an HTTP/2-enabled server match what the public PoC produces.
- Review httpd process activity: Look for shells, downloads or outbound connections from httpd workers since public disclosure on November 11.
- Enable default runtime policies: Sysdig’s runtime detections will catch exploitation, even when a vulnerability scan has no findings yet.
Conclusion
CVE-2026-119618 is a threading bug in httpd's HTTP/2 module, not a flaw in the applications httpd serves. Heap corruption and crashes are public. Code execution is claimed but not yet publicly shown, but that gap is why reachability should drive patch prioritization. Defenders should start with HTTP/2-enabled, public-facing servers.
The timelines are still important to note for security teams. Two teams reported the bug independently, eight days apart. One “team" was an automated, model-driven scanner, while the other was a human team. By November 12, a day after disclosure, a public PoC that corrupts the heap was on GitHub. The window between “advisory published” and “working exploit” is small, and organizations need to be prepared to respond quickly to these notifications.
In this case, the best course of action is to patch first. Until every server is patched and restarted, disable HTTP/2 where you can and use runtime detection to close the gap. A runtime sensor will not see the heap corruption, but it will catch what a successful exploit does next.
Read Less, Solve More: Token-Efficient Sparse Reading for AI Agents
SparseRead acts as a control layer for AI agents, cutting token costs by over 90% by intelligently selecting relevant data before processing.
Summary
Deep Dive
- SparseRead acts as a middleware between an agent and its data sources.
- Components include a Read Intent Protocol, Adaptive Reader Backends, and a Budgeted Read Gate.
- The system controls data admission to prevent context pollution and over-reading.
- It functions without modifying the base agent model or training it.
- Testing across six models showed up to 92.9% reduction in tokens and 89.0% reduction in wall time.
- Performance was validated against Claude Opus 5 and common agent frameworks.
Decoder
- Context: The text and data provided to an AI model to inform its response.
- Sparse reading: A technique that retrieves and processes only the most relevant portions of a large file instead of the entire document.
Original Article
Abstract:Web-facing AI agents repeatedly access webpages, documents, repositories, and tool responses, yet reading interfaces often expose entire objects when only sparse evidence is needed. This over-reading increases token and latency costs and can dilute relevant evidence. We present SparseRead, a reusable reading control layer that governs content admission without modifying the agent model. Its Read Intent Protocol tracks evolving needs and source coverage; Adaptive Reader Backends select how evidence is located and exposed; and the Budgeted Read Gate bounds cumulative raw exposure per episode with faithful full-view fallback. A unified Reader interface supports interchangeable evidence acquisition across heterogeneous resources. Across six models, including Claude Opus 5, and five scenarios, SparseRead reduces total tokens by up to 92.9% and wall time by up to 89.0%, preserving or improving task quality in 97% settings. On Web deep-search tasks, it reduces tokens by 31.0--43.0% and task completion time by 23.0--37.0%. Ablations validate the three designs, while integration with three agent frameworks demonstrates portability.
Auto Ontology: Grounding Agents in Enterprise Data (GitHub Repo)
NVIDIA's Auto Ontology maps enterprise database metadata into a semantic graph that agents can use to generate auditable, reliable SQL queries.
Summary
Deep Dive
- Stage 1 (Generate): Automatically extracts schemas and builds a semantic graph stored in Postgres.
- Stage 2 (Certify): Humans review and certify business terms and SQL attributes.
- Stage 3 (Answer): Agents use the ontology to classify intent and generate validated SQL queries.
- Metadata: Integrates with NVIDIA NIM for LLM inference (e.g., Nemotron-3.5) and pgvector for retrieval.
- Deployment: Uses Helm charts or Docker Compose, and includes an MCP server for integration with tools like Cursor.
Decoder
- Ontology: A formal representation of knowledge as a set of concepts and the relationships between those concepts within a specific domain.
- Semantic Model: A layer that abstracts database complexity into business-friendly terms.
- Provenance: The documented history or source of a data point or query result.
- MCP (Model Context Protocol): An open standard for connecting AI assistants to data and tools.
Original Article
Auto Ontology
Agents can retrieve documents and query databases, but reliability gates agentic analytics: a revenue metric answered faster is worth nothing if the number is wrong. Agents need a context layer that tells them where data lives, what it means, whether to use it, how to use it, where it came from, and what is similar. NVIDIA Auto Ontology is an open library that builds that layer from enterprise metadata, reconciles definitions across systems, and exposes it to any agent harness through MCP, CLI, and API. It complements the semantic models customers already own.
It enables developers to:
- Discover: generate enterprise ontologies from metadata; connect existing ontologies and semantic models; build AI-ready knowledge graphs.
- Ground: give agents trusted business meaning for metrics, entities, and relationships; reconcile inconsistent definitions across systems.
- Validate: check generated queries against certified definitions and lineage; return provenance with every answer.
- Expose: serve context to any agent harness through MCP, CLI, and API. Learn: fold certified corrections and usage back into the ontology, so shared context improves with every question.
Benchmarks
To run a text-to-SQL benchmark such as BIRD, refer to the ontology SQL evaluation repository.
Licensing & contributions. Auto Ontology is distributed under the Apache License 2.0. Third-party open-source components bundled, linked, or otherwise used by this project are listed in
THIRD_PARTY_NOTICES.md. This project is currently not accepting external contributions.
Key Features
- Natural-language querying of structured data — questions are translated to SQL and executed against connected databases, powered by NVIDIA NIM.
- Semantic catalog / ontology layer over relational sources, stored in Postgres alongside the catalog it describes.
- Authentication & RBAC —using Better Auth.
- Web UI for chat, analysis, data catalog, analytics, and settings.
- Flexible deployment — Helm chart for Kubernetes or Docker Compose for a local stack.
Software Components
| Component | Technology | Role | Default port |
|---|---|---|---|
| Frontend | Next.js 16 (React 19, TypeScript, Tailwind CSS 4), Better Auth, Prisma | Web UI, authentication, API gateway | 3000 |
| Backend | FastAPI (Python 3.12+), LangChain | Chat / NL-to-SQL, catalog, datasource APIs | 3001 |
| Ingestion worker | FastAPI (Python 3.12+) | Ingests tabular data and writes embeddings | 3002 |
| Postgres + pgvector | Relational Database | App metadata, the catalog and ontology, vector store | 5432 |
| HashiCorp Vault | Optional | Secure storage of connection credentials | — |
NVIDIA NIM
Auto Ontology uses NVIDIA NIM endpoints for inference — either the hosted NVIDIA API catalog or self-hosted NIMs.
Model triplets
Each model role is configured by a triplet of environment variables — <PREFIX>_ENDPOINT (URL), <PREFIX>_API_KEY, and <PREFIX>_MODEL:
| Prefix | Role |
|---|---|
DEFAULT_MODELS |
Shared default; every field below falls back to it. |
REASONING |
Main chat / NL-to-SQL model. |
NON_REASONING |
Lighter model used for entity extraction. |
EMBED |
Text-embedding model (must match between ingest and query). |
RERANK |
Reranker used by the retrieval flow. |
Quickstart
-
Clone the repository and create
.env:git clone https://github.com/NVIDIA/auto-ontology.git && cd auto-ontology cp .env.example .env -
Start Postgres and load the sample database.
-
Build and start the rest of the stack:
docker compose up -d --build -
Open http://localhost:3000 and sign in.
How it works
Auto-Ontology works in three stages: it generates an ontology, people certify it, and the agent answers questions from it.
The ontology holds three kinds of definition:
- Terms. Business concepts such as Customer or Invoice Line, each with a description and synonyms, mapped to the columns that hold them.
- SQL attributes. Reviewed SQL expressions under a term, such as a margin or a rolling total. The agent reuses them, so answers compute each measure the same way.
- Relationships. How tables join, including bridge tables, inferred from keys and from the data.
Integrate with us
To integrate an agent platform, application, or storage provider with Auto-Ontology, open an integration request.
License
Auto-Ontology is licensed under the Apache License, Version 2.0. SPDX identifier: Apache-2.0.
Notice and Disclaimer
NOTICE AND DISCLAIMER: This software automatically retrieves, accesses or interacts with external materials. Those retrieved materials are not distributed with this software and are governed solely by separate terms, conditions and licenses. Use this software and the retrieved materials at your own risk.
PostgreSQL 18.7, 17.12, 16.16, 15.20, 14.25, and 19 RC 2 Released!
PostgreSQL has released updates for all supported versions to patch 52 critical security vulnerabilities and over 140 bugs.
Summary
Decoder
- EOL (End-of-Life): The point at which software is no longer supported or patched for security vulnerabilities.
- GIN Index (Generalized Inverted Index): An index type used to handle data with multiple values, like arrays or JSONB.
- GiST (Generalized Search Tree): A balanced tree structure that allows for various indexing strategies beyond standard B-trees.
Original Article
Full article content is not available for inline reading.
Google announces Gemini 4 Hydrogen with ‘significant leaps' in speed, intelligence
Google launched Gemini 4 Hydrogen, promising 2.5x faster task completion and a fivefold reduction in hallucinations compared to Gemini 3.8 Flash.
Summary
Original Article
Google has introduced Gemini 4 Hydrogen, replacing the Flash branding with a model designed to deliver higher intelligence and faster performance at an accessible price. Compared with Gemini 3.8 Flash, Hydrogen reportedly completes tasks 2.5 times faster at its default high setting, more than triples its Terminal-Bench 4.1 score, and reduces fabricated claims by over fivefold. It also supports a native one-million-token input and output context, with improvements in tool recovery, multimodal reasoning, and complex agentic workflows.
Devin Apps
Devin Apps enables developers to generate and host private internal web applications directly from a natural language prompt.
Summary
Decoder
- MCP (Model Context Protocol): An open standard that enables AI models to connect securely to local or remote data sources, databases, and development tools.
Original Article
Devin Apps are live, private web applications hosted directly on Devin’s cloud computer. You describe the tool your team needs, Devin builds it, and your team opens it from a link.
Because Devin Apps run directly on Devin’s machine, they have access to your existing MCPs, integrations, and codebase. You can build apps with your own internal tech stack and components, or start from a built-in app template.
Create an App
- Open Apps in the sidebar.
- Describe what the App should do. You can also pick a prompt template, such as a metrics dashboard, a project status page, or a review queue.
- Click Build with Devin.
Devin starts a session and asks any questions it has in its first message. Then it builds the App and puts it online. The App shows up on the Apps page right away with its URL. Its status is Building until the first version is live.
You can also ask for an App in any session. If Devin has already shown you a browser preview, you can ask it to publish that preview as an App.
Share it with your team
Each App has a stable URL. Everyone who opens an App has to sign in to Devin, and Apps are never public.
Click Share to copy the link or change who can open the App. You can limit it to the people you add, open it to your organization, or open it to your whole enterprise.
You can pin the Apps you use most. Organization admins can also pin an App for everyone.
Update it
Open the App and click Edit in session, then tell Devin what to change. Devin builds a new version and puts it online. The URL does not change.
Every update is a new version, and the App’s page lists them under Deployments. If a new version fails to start, the previous version keeps serving.
An automation can also keep an App up to date. Devin can pull fresh data on a schedule, or redeploy the App each time a branch changes.
How Devin Apps run
Devin Apps are built on memory snapshots, which let Devin hibernate between tasks. Devin’s machine goes to sleep when nobody is using the App, and it wakes when someone opens it. The first load after a sleep can take a few seconds. Billing is based on how long the machine is awake.
Devin can use any stack that serves HTTP on a port. The App can use a database such as SQLite or Postgres on the same machine. An App can also call the Devin API, so you can build your own tools on top of Devin.
An App runs on one machine. It is a good fit for internal tools and dashboards, not for high-traffic production services.
How teams use Devin Apps
- Internal tools. Teams build admin panels, request forms, approval queues, and small tools that replace a shared spreadsheet.
- Dashboards. Teams track metrics, support queues, and sales numbers. Devin pulls from your own data sources and refreshes the dashboard on a schedule.
- Beta and staging versions. Devin can run your real stack on its machine, including your frontend, backend, database, and workers. Teammates open a beta or staging copy of your product from a link.
Get started
Open Apps to build your first App.
Quark: our whole training stack, end to end, for $4,200
Empero's Quark is a 165M-parameter model trained end-to-end for $4,200, achieving competitive performance with a 128K context window.
Summary
Deep Dive
- Architecture: 16 layers with local and global attention; global layers handle the full key-value cache.
- Training: Warmup-stable-decay on 500B tokens; context extended to 128K tokens.
- RL: GRPO used with code-based rewards (unit tests, format checks) rather than human feedback.
- Efficiency: Outperforms similar 120M-170M parameter models on benchmark averages.
- Implementation: Requires
transformers5.23+ and a GPU withbfloat16support.
Decoder
- GRPO (Group Relative Policy Optimization): A reinforcement learning algorithm that improves training stability by sampling multiple outputs for a given prompt and optimizing against a baseline group score.
- MGVR (Multi-head Gated Value Residuals): An architectural technique where global attention layers reuse values from initial layers through a learned gate to reduce memory overhead.
- Anneal: The final phase of pretraining where the learning rate is reduced to refine the model weights on a specific dataset or context length.
Original Article
Before we spend serious money on a large model, we want to know that every part of our stack works: data, architecture, pretraining at scale, context extension, reinforcement learning, evaluation and release. Quark is that test, run end to end. It is a language model with about 165M parameters (140M without embeddings), and we took it through every stage, from the first data-generation job to the last reinforcement-learning step, for about $4,200 of compute, about 1,300 H100-hours. That total includes a discarded first design, failed attempts, data generation and every evaluation. Compute for this work was partly sponsored by lium.io.
It reads up to 128K tokens, it has the best benchmark average among open models of its size trained on similar token budgets, and one version learned to answer, think and call tools from reinforcement learning alone, with no supervised fine-tuning stage and no distillation. All four checkpoints are on Hugging Face today under Apache-2.0.
| Model | What it is |
|---|---|
| Quark-150M-32K-Raw | The pretraining checkpoint before its final anneal, at 472B tokens |
| Quark-150M-32K-Base | The base model after a 28B-token anneal at 32K context |
| Quark-150M-128K-Base | The base model after another 10B tokens at 128K context |
| Quark-150M-128K-Zero | The 128K base trained with reinforcement learning alone |
The technical report has the details: the architecture, the training stages, the full cost breakdown, every evaluation and its limits.
The stack, stage by stage
Architecture. Most of what a language model does is local, so most of Quark's 16 layers only look nearby. Twelve local layers combine attention over the last 256 tokens with a short convolution, and every fourth layer is global and attends to the whole context. Each attention after the first layer also reuses the first layer's values through a learned gate, the multi-head gated value residuals from our MGVR work. The design pays off at inference: only the four global layers keep a full key and value cache, so at 128K tokens the cache takes about 0.5 GiB in bf16, a quarter of the 2 GiB the same model would need with every layer global.
Pretraining. A warmup-stable-decay schedule on 500B tokens on a single 8×H100 node: a short warmup at 1K context, 470B tokens at 4K context with a constant learning rate, then a 28B-token anneal at 32K context. The data is mostly English web text, math and code, with multilingual web text, reasoning traces with verified answers and synthetic long-context documents. We also release the last checkpoint before the anneal, Quark-150M-32K-Raw, because pre-anneal weights are the natural starting point for continued pretraining and are rarely published.
Reliability. During pretraining, the node went down seven times, with no spike in the loss at any restart. We resumed the first two by hand; after that, automatic recovery restarted training within two to four minutes of the node coming back. Resuming is exact: the data order follows from the step count alone, so a restarted run sees the same tokens it would have seen.
Context extension. The 128K model branches from shortly before the end of that run and trains on another 10B tokens of 128K-token sequences: books, long documents, arXiv papers, long-context instruction following and synthetic tool-use sessions.
Reinforcement learning. Quark-150M-128K-Zero starts from the 128K base and was trained with GRPO, with no supervised fine-tuning stage and no distillation: no text written by another model was ever a training target. About 95% of the rewards came from code: exact answers, format rules, unit tests and file checks in real sandboxes. LLM judges scored the rest, such as chat replies, creative writing and second opinions on value questions, and no human labels were used. The run took under 1,300 steps and about 32 hours. We stopped it early to stay within budget while its scores were still rising.
Evaluation and release. Every number below comes from our own runs, with the same harness and settings for Quark and for the models we compare against. The four checkpoints ship with their modeling code, tokenizer, chat template and model cards that show unselected samples, failures included.
How it compares
We ran nine zero-shot benchmarks with the lm-eval harness (0.4.13) on Quark and on open models of 120M to 170M parameters trained on up to about 500B tokens, all in our own runs in November 2026. Scores are in %.
| Model | Tokens | LAMBADA | HellaSwag | PIQA | ARC-E | ARC-C | WinoGrande | SciQ | OBQA | BLiMP | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Quark-150M-128K-Base | 506B | 39.0 | 38.3 | 64.4 | 55.4 | 27.7 | 54.5 | 84.3 | 31.4 | 80.0 | 52.8 |
| Quark-150M-32K-Base | 500B | 34.9 | 36.6 | 63.2 | 50.1 | 27.1 | 54.0 | 83.6 | 30.8 | 77.4 | 50.9 |
| Mamba-130M | 300B | 43.5 | 35.1 | 64.7 | 47.6 | 24.4 | 52.3 | 78.1 | 29.6 | 80.4 | 50.6 |
| OPT-125M | 180B | 38.8 | 31.6 | 62.4 | 42.8 | 22.1 | 50.9 | 78.2 | 29.0 | 82.7 | 48.7 |
| RWKV-4 169M | 330B | 32.4 | 32.3 | 64.4 | 47.5 | 23.8 | 51.5 | 75.1 | 27.4 | 82.5 | 48.5 |
| GPT-Neo 125M | 300B | 37.6 | 30.5 | 62.6 | 43.7 | 23.0 | 50.8 | 76.2 | 26.0 | 80.9 | 47.9 |
| GPT-2 124M | not published | 31.0 | 31.3 | 62.1 | 44.2 | 22.6 | 51.7 | 73.3 | 27.2 | 83.2 | 47.4 |
| Pythia-160M | 300B | 35.4 | 30.3 | 62.3 | 43.6 | 23.6 | 51.3 | 75.4 | 26.8 | 77.5 | 47.4 |
| SmolLM2-135M | 2T | 43.0 | 43.1 | 68.1 | 64.2 | 29.7 | 52.7 | 84.0 | 33.2 | 80.1 | 55.3 |
HellaSwag, ARC-Challenge and OpenBookQA use normalized accuracy, BLiMP is the mean over its 67 subtasks, and Pythia ran in float32 because it degrades in bf16. Quark-150M-128K-Base has the best average in its group, 52.8, and leads on six of nine tasks. Mamba-130M is ahead on LAMBADA and narrowly on PIQA, and BLiMP is Quark's weakest task. No other model in the group was trained on sequences longer than 8K tokens. SmolLM2-135M, in the last row, was trained on 2T tokens, four times as many with about three times the compute, and scores higher on seven of nine tasks. Quark leads it on WinoGrande and SciQ.
For long context, we score teacher-forced token-check accuracy on held-out synthetic tasks: the reference text is given up to the answer, and the model's predictions must match the answer tokens. At 128K tokens, the 128K model passes every check in five of eight task families: needle retrieval, conversation recall, event counting, ledger arithmetic and variable tracking. Key-value lookup passes four of eight, and quote attribution fails at every length. These checks show whether the model retrieves the right value, not whether a free generation reaches it.
What reinforcement learning taught it
Before reinforcement learning, the base model wrote tool calls but invented their results: in 1,152 audited replies, only 3 got a right answer after a call. After reinforcement learning alone, in training rollouts on simple terminal tasks across 15 emulated coding-agent harnesses with real sandboxes, the share solved rose from 3% to 65%. Harnesses added late in the run started at 42% to 54% solved, so the skill carried over between formats. On our in-house task suite, well-formed replies rose from 71% to 96% and correct answers from 13% to 36%.
The model can also think in a private monologue before it answers. On the commonsense puzzles in our suite, making it always think first raises accuracy from 44% to 67%, compared with our normal setting, in which about half the tasks ask for a direct answer and the rest leave the choice to the model.
It is still a small model. Asked who it is, it says "I am quark, and I am here to chat with you." Given the classic bat-and-ball puzzle with thinking switched on, it confidently answers $2.10. The model cards on Hugging Face show more samples, including failures.
What did not work
Web search did not improve during reinforcement learning, and in one of the samples on its model card the model opens a tool call and writes an answer inside it. One of the 15 harness formats never worked, and long-context scores fell during the RL stage. The agentic numbers are training rollouts on simple tasks, not a held-out agent benchmark. Quark knows little about the world, makes factual and reasoning errors and states them confidently, and it was trained mainly on English. Its only safety training is a small set of refusal tasks in the RL stage, and it has not been safety-evaluated. These are research artifacts, not products.
Using it
You need transformers 5.23 or newer. The modeling code ships with the weights:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "empero-ai/Quark-150M-128K-Zero"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda")
messages = [{"role": "user", "content": "Explain in two sentences why the sky is blue."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt",
return_dict=True).to("cuda")
out = model.generate(**inputs, max_new_tokens=256)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
The model cards describe the chat format, how to switch thinking on and how to offer tools.
What comes next
Every stage of the stack ran end to end on a small budget, and each one produced something we could measure. That is what we built Quark to find out. Next we want to scale the same pipeline to a larger model, run the RL stage to its full length, fix quote attribution at long range and find out how far it can go on real agentic work. If you build on Quark, find something we should fix, or want to help us scale it, write to [email protected].
End-to-End 4-Bit AI Training
Thinking Machines demonstrates how to train language models using 4-bit precision end-to-end, achieving significant speedups on NVIDIA GB300 hardware.
Summary
Deep Dive
- NVFP4 offers 3.6x fewer bits than BF16 but provides up to 6x speedup on GB300 hardware.
- NVFP4 quantization error is managed through block-scaling (1x16 groups).
- Forward/backward mismatch is mitigated using stochastic re-quantization.
- RHT (Random Hadamard Transform) is used during tensor transmission to smooth values and reduce quantization errors.
- BBBK (Big Beautiful Blackwell Kernel) was developed to maximize NVLink, ALU, and Tensor Core utilization simultaneously.
Decoder
- NVFP4: A 4-bit floating point format optimized for NVIDIA's Blackwell GPU architecture.
- MXFP8: A 8-bit Microscaling format that provides a flexible range for representing values in deep learning.
- Hadamard Transform: A linear transformation used to redistribute values in a matrix, reducing outliers and aiding quantization efficiency.
- GEMM (General Matrix Multiply): The core mathematical operation (matrix-matrix multiplication) in deep learning inference and training.
Original Article
Training large language models involves calculating and storing a lot of numbers. Using fewer bits for each value reduces memory usage and allows GPUs to perform calculations faster. However, there is a tradeoff: fewer bits also mean fewer representable numbers, leading to more aggressive rounding. Lower-precision representations ultimately hurt accuracy by adding error to the calculations.
For example, suppose we want to store the number 3.2, but the nearest values our numerical format can represent are 3 and 4. Rounding to 3 introduces a quantization error of − 0.2, the difference between the actual value and the one that can be represented in the quantized format. A single error like this may seem small, but these errors accumulate over training as rounding happens repeatedly across layers and optimization steps, which can impair the learning process.
This is the central challenge of low-precision training: how can we benefit from faster arithmetic without introducing errors that interfere with learning?
We manage this tradeoff through our use of three key numerical formats during training: BF16, MXFP8, and NVFP4. Each format’s precision is fixed, but its speed depends on the hardware being used. We train on NVIDIA’s Grace Blackwell Ultra platform, the GB300, which is designed to optimize matrix multiplications with NVFP4. Notably, NVFP4 is stored with 3.6x fewer bits than BF16, but it runs up to 6x faster on GB300s.
| Numerical format | Bits per value | GB300 PFLOPS |
|---|---|---|
| NVFP4 | (4 × 16 + 8) ÷ 16 = 4.5 | 15 |
| MXFP8 | (8 × 32 + 8) ÷ 32 = 8.25 | 5 |
| BF16 | 16 | 2.5 |
The NVFP4 acceleration on GB300s is very useful in high-volume inference, where the tradeoff leans in favor of speed. But compared to BF16 and MXFP8, the format greatly increases quantization error. This error matters more at different stages of training, and the tradeoff must be considered for each one separately: forward and backward passes in pretraining and supervised fine-tuning (SFT), sampling and training in reinforcement learning (RL).
Speed matters most in high-volume steps: pretraining, which runs on trillions of tokens, and RL sampling, which generates tens of thousands of tokens in each rollout. Each sampled token also requires loading the model’s weights from global memory, the cost of which scales with the number of bits needed to store the weights.
The backward passes that update the weights average over many tokens in a batch, which dilutes the impact of randomly distributed rounding errors. However, backward passes are very sensitive to quantization bias, introduced when values consistently round up or down during quantization. Instead of averaging out, this bias can compound across layers, samples, and training steps.
In training Inkling 2 Small, we made the following decisions:
- NVFP4 precision in the expert layers for forward passes, bolstered by a dedicated kernel.
- MXFP8 precision for backward passes with re-quantization and stochastic rounding, techniques for reducing gradient bias.
- Quantization techniques for accelerating GPU communications, which alleviated a bottleneck that surfaced after we sped up computation.
Inkling 2 Small is the largest model we are aware of that was trained end-to-end with a native 4-bit forward pass. We configure numerics in the same way from the start of pretraining through the final RL rollout. This makes training simpler and eliminates the need for calibration when a quantized checkpoint is delivered at the end of training. In our largest jobs, this recipe provides us with an 8% wall-clock speedup in pretraining and a 20% speedup in RL over full MXFP8 training.
Choosing a target for quantization
Ideally, we would like to reduce precision where it has the least effect on model accuracy, while offering a substantial overall speedup. The layers that best fit this profile are Mixture of Experts (MoE) layers, which are computationally expensive and empirically tend to harm accuracy less than other layers when quantized.
Earlier this year, we implemented full MXFP8 training of our MoE layers and found it barely affected our model’s compute efficiency (CE), a measure of how training and evaluation losses change when a new technique is introduced. However, our initial experiments with full NVFP4 training resulted in very large hits to our model’s CE, degrading accuracy beyond what we were willing to accept.
When isolating the effects of NVFP4 quantization in the forward and backward passes, we found that focusing on the forward pass generally offered better tradeoffs between speed and accuracy. But quantizing just the forward pass carries a number of implications for interactions with the backward pass, GPU communications, and more. To understand these, we’ll first examine how NVFP4 works in detail.
Low-precision formats on Blackwell
Four bits can represent up to 16 different values: in the case of FP4, these are ±{0, 0.5, 1, 1.5, 2, 3, 4, 6}. The obvious limitation is in representing numbers larger than 6, which is solved by block scaling: storing a scale factor that multiplies all values in a consecutive block. NVFP4 scales each block of 16 consecutive values based on the largest one in the block (in absolute terms). If that value is −54, for example, it would be represented as −6 with a scale factor of 9, and every value in the block would be rounded based on that scale.
Blackwell GPUs also offer microscaling (MX) formats such as MXFP4 and MXFP8, which scale blocks of 32 values. The formats used for these scale factors are different as well: NVFP4 scale factors are E4M3, making them more precise but limited in their range, while MXFP4/MXFP8 scale factors are E8M0, far less precise but able to represent a very wide range of values. The increased precision offered by NVFP4 scale factors tends to represent input data more accurately in LLM workloads, resulting in less quantization error than MXFP4.
This combination of FP4 and FP8 allows NVFP4 to represent magnitudes as low as 2⁻¹⁰ and as high as 2688. In practice, we will often want to represent values outside of this range, so NVFP4 is generally used with a second outer-level scale factor which is saved in FP32 and is applied across a very large group of values, such as an entire tensor. This allows each quantized value to be reconstructed as xᵢᶠᵖ⁴ × xᵢ//₁₆ᶠᵖ⁸ × xᶠᵖ³², which effectively allows us to shift the representable range up or down. Setting xᶠᵖ³² to 2, for example, shifts the range of representable magnitudes from [2⁻¹⁰, 2688] to [2⁻⁹, 5376], and setting it to 0.5 shifts the range in the opposite direction.
Block-scaled backward passes
A forward pass through the network multiplies the quantized activations by the quantized weights: y = x_q w_qᵀ. Computing a dot product with NVFP4 takes blocks of 16 values in a row of x_q and a column of w_qᵀ, multiplies the pairs, and then multiplies the sum by the scale of each block.
To compute the backward pass, we need to transpose the tensors from the forward pass: dx = g w_q and dw = gᵀ x_q. Now, the blocks take 16 values from columns of x_q and rows of w_qᵀ: each value is grouped with 15 different neighbors, resulting in different scale factors and different quantized values represented. This creates a mismatch between the tensors used in the forward and backward passes, which can lead to inaccurate gradient computation. For weight tensors, a straightforward solution is suggested by NVIDIA: rather than computing scale factors for each 1×16 block, they compute a single scale factor based on the largest value in each 16×16 square tile, keeping the scale factors identical when the tensor is transposed.
The main drawback of this approach is that it can cause a lot of values to underflow. Increasing the block size from 16 values to 256 makes it highly sensitive to outliers: a single large value among 256 can push the scale factor up, causing the remaining values to be rounded to zero.
We tried this approach but ultimately found computing scale factors over 16×16 tiles to be too coarse, hurting CE beyond what we were willing to accept. For NVFP4 training to work, we needed to quantize weights with 1×16 scale factors while avoiding the mismatch introduced during a block-scaled transpose.
Re-quantization with stochastic rounding
The use of 16×16 tiles eliminates the mismatch by enabling bitwise-identical representations between the forward and backward passes — w_q = (w_qᵀ)ᵀ — but perfect identity isn’t strictly necessary. The important requirement is that the tensors used in the forward and backward passes are equal in expectation: E[w_q] = E[(w_qᵀ)ᵀ]. Preserving the expectation addresses the problem of bias in the backward pass, and loosening the constraint on bitwise equality allows for a few different approaches to the transpose problem posed earlier.
We opted for a straightforward solution: de-quantizing tensors from the forward pass, transposing them in higher precision, and then re-quantizing them with stochastic rounding (SR). Unlike standard quantization, also known as round-to-nearest quantization, SR rounds each value up or down probabilistically, in inverse proportion to its distance from each neighbor.
For example, let’s assume that the forward pass tensor had a value of 5.4, quantized in a block with scale 9. 5.4 divided by 9 equals 0.6, which rounds to the nearest FP4 value of 0.5. The quantized value used in the forward pass is 4.5, which is now the value we want to recover in the backward pass (in expectation).
We can de-quantize this value to have it represented in high precision, but in the backward pass that value is now grouped in a different block. Let’s take its new scale to be 2. If we were re-quantizing with NVFP4, 4.5 ÷ 2 = 2.25 would now need to be quantized to FP4. Deterministic rounding would represent it as a 2, for a quantized value of 2 ⋅ 2 = 4. Stochastic rounding instead rounds it to 2 three quarters of the time and to 3, its nearest neighbor from above, a quarter of the time. The value of 4.5 from the forward pass is preserved in expectation after re-quantizing: 2_scale ⋅ (0.75 ⋅ 2 + 0.25 ⋅ 3) = 4.5. Crucially, stochastic rounding is also hardware-accelerated on Blackwell GPUs.
In designing this approach we drew inspiration from TetraJet-v2 and Quartet II, which also re-quantize the tensors from the forward pass. A benefit of applying SR directly to de-quantized values is that it lets us choose any precision during re-quantization. We found that stochastically re-quantizing weights and activations to MXFP8, and then computing gradients using MXFP8 matrix multiplications, offers the best tradeoff between accuracy and speed.
Quantized communications
With our new NVFP4 matrix multiplications (GEMMs), our MoE layer’s forward pass should theoretically have gotten about three times faster compared to our MXFP8 implementation. But after reducing precision, we noticed we were running up against a bottleneck: communications during expert parallelism.
Since MoE layers are often very large, it is common to split up experts between different GPUs. When computing the MoE layer, a small router decides where each token should go, and tokens are then dispatched to the GPUs containing the routed experts. After completing the GEMMs, the computations of these routed experts are returned to the source GPU and combined to create the MoE layer’s final output. On an NVIDIA GB300 NVL72 system, the tokens will typically be sent over NVLink, which has a bandwidth of 900 GB/s in each direction. At the same time, the NVFP4 GEMMs will run at a speed of up to 15 PFLOPS. Dividing one by the other, each GPU could perform as many as 16,000 floating point operations in the time it takes to receive a single byte. Unless every byte that arrives brings at least that much computation with it, the Tensor Cores sit idle waiting on NVLink.
We found this imbalance created bottlenecks where either the MoE GEMMs were unable to execute because they were waiting to receive the dispatched tokens, or the following operation (in Inkling 2 Small, short convolutions) was unable to start because it was waiting to receive the combined tokens. During dispatch, we avoid this issue by dispatching tokens in NVFP4. Like TetraJet-v2 and Quartet II, our backward pass is computed using quantized — not high-precision — activations from the forward pass, which reduces the amount of data sent over NVLink enough to alleviate the dispatch bottleneck. During combine, however, the output activations need to remain in higher precision (by default, BF16) to avoid impairing model performance.
We alleviate the communications bottleneck by quantizing the data transmitted during combine as well. We initially experimented with quantizing the resulting activations to MXFP8, but found that this impaired performance too much due to MXFP8’s poor representation of outlier values. Instead, we found that performing a 32-dimensional random Hadamard transform (RHT) and then quantizing to block-scaled S2F6, another 8-bit format, resulted in no measurable accuracy degradation during training.
RHTs apply a rotation which tends to smooth out the input data. This can reduce quantization error when the input data contains outliers. S2F6 is a fixed-point format with two signed integer bits and six fractional bits, and it can represent linearly interpolated values from -128/64 to 127/64, closely resembling an integer format.
Kernels for low-precision training
Taking full advantage of NVFP4 requires specialized kernels which we’ve developed in-house for our low-precision recipes. Our MoE fusion kernel stack is codenamed BBBK, the Big Beautiful Blackwell Kernel. It powers training and inference with peak hardware utilization while maintaining fully deterministic computation.
The process for computing MoE layers is generally dominated by matrix multiplications. We saturate the Tensor Cores using the largest 2SM UMMA instruction unique to GB300 (K=96), fed by a 6-stage multiplicand pipeline and a 7-stage scale factor pipeline. Our precision recipes, however, also place heavy demands on the GPU’s arithmetic logic units (ALUs). Even when the Tensor Cores are saturated, kernels can stall on ALU-heavy steps such as computing scale factors, re-quantizing weights, applying stochastic rounding, and coalescing data transfers.
Recipe details
We took inspiration from several other existing works on NVFP4 training to round out our recipe. We keep the last 16.6% of layers in higher precision, which we compute in MXFP8 rather than BF16. When quantizing activations, we use Four Over Six (4/6), a technique which selects between two scale factors during NVFP4 quantization in order to reduce quantization error. We forgo 4/6 when quantizing weights.
Drawing inspiration from Composer 2, we use per-expert outer scales when quantizing weights and per-token outer scales when quantizing activations, allowing for higher efficiency, batch-invariance, and no worry about causality leakage. In our largest training jobs, this recipe has worked without changes during both SFT and RL. In pretraining Inkling 2 Small it costs about 2% in compute efficiency, meaning the model reaches the loss it would have reached with about 2% fewer FLOPs if trained in MXFP8, in exchange for the speedup provided by NVFP4. In RL the gain was much larger: a substantial end-to-end speedup with no discernible difference in reward.
Closing thoughts
A decade ago, it was unclear whether neural networks could be trained using fewer than 32 bits per parameter: the default format remained FP32 as recently as 2020, when BF16 first came to NVIDIA GPUs with Ampere. Today, we often find ourselves referring to BF16 as “high precision,” and we have found robust use cases for training with 8-bit and even 4-bit formats! We’re inspired by NVIDIA’s forward-thinking hardware design that has made these advancements possible. We will continue to push forward: we are actively exploring recipes that enable FP4 computation of the backward pass, as well as how our recipes will evolve on NVIDIA’s latest Vera Rubin systems.
The acceleration of both our pre- and post-training is the result of a constellation of improvements in science and engineering, touching every part of the stack and compounding on each other. Months ago, a training run of a model similar in size to Inkling 2 Small took several weeks; Inkling 2 Small was pretrained in just 68 hours on GB300 NVL72 racks. Our numerics recipe is just one piece of these many improvements, and one we’re continuing to iterate on.
Citation
Please cite this work as:
Cook, Jack and Thinking Machines Lab, "End‑to‑End 4‑Bit Training",
Thinking Machines Lab: Connectionism, Nov 2026.
@article{cook2026lowprecision,
author = {Jack Cook and Thinking Machines Lab},
title = {End‑to‑End 4‑Bit Training},
journal = {Thinking Machines Lab: Connectionism},
year = {2026},
note = {https://thinkingmachines.ai/blog/end-to-end-4-bit-training/},
}BirdDreams: A Driving World Model
NVIDIA's BirdDreams is a driving world model that jointly predicts perspective video and bird's-eye view scenes from a single camera image.
Summary
Deep Dive
- Joint generation enables both views to inform each other, improving spatial consistency.
- BEV provides a geometric interface for ego control, allowing the model to follow specific meter-based trajectories.
- The model outperforms perspective-only baselines in generating reactive traffic behaviors, such as vehicles adjusting lanes to avoid aggressive maneuvers.
Decoder
- BEV (Bird's-Eye View): A top-down representation of a scene, commonly used in autonomous driving to provide a scale-accurate map of road and vehicle positions.
- Ego Motion: The movement of the vehicle itself (the camera car) through the environment.
- Autoregressive: A method where the model predicts future states based on a sequence of preceding inputs.
Original Article
BirdDreamsA driving world model with precise ego control and traffic that can react.
From one camera image, BirdDreams jointly predicts driving video and a bird’s-eye view (BEV) of the scene. With an optional ego trajectory in meters, it follows that path while predicting how surrounding traffic moves.
More plausible surrounding-vehicle motion
We compare against an OmniDreams-based baseline trained to predict RGB without surrounding-vehicle-state inputs. For this comparison, we use HD-map-conditioned variants with the same starting image, ego trajectory, and road map.
Which video has more realistic vehicle behavior?
- 57% BirdDreams
- 24% Baseline
- 19% About the same
25 participants · 375 pairwise judgments. The OmniDreams-based and BirdDreams variants use HD maps but no surrounding-vehicle-state inputs, with the same starting scene and ego maneuver.
When we specify the ego’s path
We set the ego trajectory to make the ego cut in aggressively or drive through a red light. BirdDreams predicts how surrounding vehicles react.
When the model predicts ego motion too
Without an input ego trajectory, BirdDreams predicts how both the ego and surrounding vehicles move. Here, both the ego and a crossing car brake to avoid a collision.
Precisely following ego trajectories specified in meters
Following small steering changes
Both models receive the same ego trajectory, taken from the recorded drive on the left. Compare the camera car’s position relative to the lane markings as it steers slightly right.
With ego-action input
BirdDreams follows ego-action inputs more closely than the baseline. ADE: 35% lower error in following the input trajectory.
Without ego-action input
Without action inputs, BirdDreams generates trajectories closer to recorded driving than the baseline. minADE: 21% lower trajectory error.
Driving world models aim to simulate how a scene unfolds under different ego actions. This requires plausible motion of surrounding vehicles as well as accurate control of the ego trajectory. Learning vehicle motion from perspective video alone is challenging because motion is entangled with appearance and vehicles can be occluded. Prior work therefore conditions video generation on layouts that specify vehicle motion, providing control over surrounding vehicles but requiring their future trajectories as inputs. We introduce BirdDreams, an autoregressive driving world model that jointly generates perspective video and a schematic bird's-eye-view (BEV) video, making scene layout part of the predicted future. The BEV depicts roads and vehicles as simple shapes at a fixed metric scale, providing a direct view of their positions and motion. Perspective video, in turn, retains visual cues that the BEV omits. Joint attention between two video diffusion transformers, one per stream, lets each stream inform the other during generation. The predicted BEV also provides a geometric interface for ego control: we translate and rotate its latest frame along an input ego trajectory and use the resulting frames as a soft condition for generation. Trained with BEV targets rendered from scene annotations, BirdDreams generates both videos from a single camera image without requiring an HD map or vehicle-state inputs at inference. On PhysicalAI-AV and nuScenes, BirdDreams follows input ego trajectories more faithfully than prior driving world models and generates more plausible ego motion without trajectory conditioning. Human evaluation shows that BirdDreams outperforms a perspective-only baseline in generating realistic surrounding-vehicle behavior.
Acknowledgements
We are deeply grateful to Sanja Fidler, Zan Gojcic, Xuanchi Ren, Kevin Xie, Yifan Lu, and Zian Wang for their valuable discussions, insightful feedback, and generous support throughout this work. We also thank Yuxuan Zhang for developing the HD-map-conditioned variant of OmniDreams.
Meta's Vision Encoders in Multimodal Models
Meta researchers found that dedicated visual capacity matters more for multimodal performance than separate vision-encoder pretraining.
Summary
Deep Dive
- Compared four recipes: Pretrained encoder, scratch encoder, encoder-free, and in-fusion.
- Pretrained encoders provide no advantage over scratch encoders at matched total lifetime compute.
- Encoder-free models significantly underperform on VQA tasks even when parameter count is equalized.
- Early-layer 'in-fusion' (adding vision-only FFNs to the backbone) achieves parity with separate encoders.
- In-fusion reduces time to first token by removing the serial bottlenecks of external encoders.
- Early placement of vision capacity is critical for tasks requiring fine detail like OCR.
Decoder
- Multimodal Model: A model capable of processing and relating information from multiple data types, typically vision and text.
- In-fusion: Architectural design where vision-specific processing modules are integrated directly into the shared layers of a language backbone.
- VQA (Visual Question Answering): A task involving answering natural language questions based on an provided image.
Original Article
Most multimodal models see through a pretrained vision encoder, which bundles three design choices: a visual prior from separate pretraining, dedicated vision capacity, and pre-fusion placement of that capacity. We test each choice by training every model from scratch with the same data, backbone architecture, and training recipe, and compare the choices at matched lifetime compute.
What we find
- The prior is optional. An encoder trained from scratch with the language model is on par with encoders pretrained on the same data. More encoder pretraining raises ImageNet accuracy but not VQA, even when it is free.
- Dedicated capacity is key. Encoder-free models trail on visual understanding even when their backbone gets the encoder's parameters. From 1.7B to 6.4B backbones, they need about 3× the compute to match the encoder's Avg. VQA.
- Vision capacity can sit inside the backbone. Vision-only modules in early backbone layers perform comparably to a pre-fusion encoder.
- The findings hold beyond lifetime compute. Added to a pretrained 1.7B language model, in-fusion stays comparable to the scratch encoder, while encoder-free falls further behind. In-fusion can also lower time to first token.
What a vision encoder bundles
Most multimodal models acquire visual understanding from a vision encoder pretrained for a separate objective, such as image recognition or image–text alignment. Recent recipes discard parts of that setup. Kimi K3 trains its encoder from scratch with the multimodal backbone, and encoder-free models feed image patches straight into the backbone. Each recipe implies a different answer to what the encoder contributes, if anything at all.
One controlled protocol
Published models differ in data, architecture, training stages, and compute, so comparing them cannot isolate any one choice. Controlled studies of pretrained encoders hold the multimodal stage fixed, but they vary which prior an encoder imports rather than whether it needs one. They also treat encoder pretraining as free, though SigLIP 2, for example, trains on 40B image–text samples. We instead run every training stage ourselves, from random initialization.
Every recipe pretrains on the same text and image–text pairs, with images at 224×224. Avg. VQA averages four category means over 16 Cambrian benchmarks, after instruction tuning on Cambrian-7M with 448×448 images. Avg. Text does the same over 12 language benchmarks.
Separate pretraining is optional
Under a fixed lifetime budget, encoder pretraining competes with multimodal training for compute. We pretrain the same encoder with four objectives (contrastive MetaCLIP, self-supervised WebSSL, and captioning with Cap and CapPa) on the same images, update it during multimodal training, and compare it with the encoder trained from scratch with the backbone.
At 1.7B and 4B, MetaCLIP trails the scratch encoder at every budget, by 1.8 Avg. VQA points at the largest 4B budget.
Dedicated vision capacity is key
Encoder-free models drop the encoder and map each image patch directly into the backbone with a linear embedding. Removing the encoder also removes its parameters and changes how raw patches enter the backbone, so a gap could reflect fewer parameters, a weak patch embedding, or missing dedicated capacity.
Across budgets, encoder-free trails by 4.1–6.4 Avg. VQA points at 1.7B and 2.8–6.8 at 4B. To test whether the gap is simply due to the encoder's extra parameters, we give the encoder-free backbone the encoder's 0.4B parameters as extra width, depth, or both.
Matching total parameters doesn't close the gap to a scratch encoder. The added parameters improve Avg. Text, since text tokens use them too, but move Avg. VQA by at most 0.4 points.
A richer patch embedding doesn't close it either. Trained on the same 224×224 images, a two-layer projection inspired by Gemma 4 helps, scoring 40.5 Avg. VQA against 39.0 for the linear default, but stays well below the encoder's 44.4. A four-layer hierarchical MLP, as in Inkling, matches the encoder only when it compresses 640×640 images into the same 256 tokens.
Vision capacity can move in-fusion
An encoder's parameters are dedicated, used only by vision tokens, and pre-fusion, run before vision and text interact. To separate the two, we move the encoder's 0.4B-parameter budget in-fusion, into vision-only modules inside the shared backbone.
Holding the budget at 0.4B parameters, we vary where it goes and which tokens use it.
Placed in the first eight layers, the vision-only FFN branch matches the scratch encoder (45.0 vs. 44.4), while the other placements do no better than encoder-free. Early placement also helps most on OCR and chart questions (36.0 vs. 25.2 for encoder-free), which depend on fine visual detail. With the first eight layers as the default, in-fusion, which also adds vision-only attention projections, holds the same dedicated budget as the encoder.
The comparisons hold with scale
We also train the three recipes on a ladder of backbone sizes and token budgets, growing the vision module with the backbone, and fit one law per recipe in model size and training tokens.
The findings carry over to midtraining, and in-fusion can be faster to serve
Midtraining: adding vision to a pretrained language model
We continue our own 1.7B text-only model from 1T to 1.33T token-equivalents, adding each recipe at equal compute, and include mixture-of-transformers (MoT) for comparison.
MoT with vision-only copies of all 28 layers falls behind MoT on the first eight layers alone, despite 3.5× as many vision-only parameters, consistent with late placement not helping.
Time to first token
A separate encoder must finish each image before the backbone can start, which adds a serial step to inference. We measure time to first token for a 1.7B backbone on one H200 serving 16 concurrent requests, each with one 448-px image and 256 text tokens.
In-fusion's advantage over the encoder grows with the vision-only budget, which scales with the backbone. MoT, which copies whole backbone layers for vision tokens, is about as fast as in-fusion up to 0.4B vision-only parameters, but its added time grows faster with the budget, while staying below the encoder's. The encoder-free model is fastest, but at 1.7B and 2.6×1021 FLOPs it trails the encoder by 5.4 Avg. VQA points.
Common questions
Why count lifetime compute? A pretrained encoder brings compute and data of its own that comparisons usually leave out. SigLIP 2, for example, trains on 40B image–text samples. Counting only the multimodal stage lets a recipe look better just because it spent more in total. Lifetime compute counts every stage that produces the final model: encoder pretraining, any text-only training, and multimodal training.
How do your encoders compare to released ones like CLIP and SigLIP? Every encoder is a 27-layer, 0.4B-parameter ViT, about the size of SigLIP's So400m, and pretrained encoders see only our own training images, so every recipe learns from the same data. Pretrained four different ways, these encoders are on par with an encoder trained from scratch with the backbone at matched lifetime compute.
How is in-fusion different from mixture-of-transformers? Mixture-of-transformers (MoT) gives each modality its own modules, so the two streams meet only in attention. In-fusion processes every token with the shared weights and adds zero-initialized vision-only weights on top, sized to the encoder's 0.4B-parameter budget, in the first eight layers only.
Why train from scratch instead of starting from a pretrained language model? Training every stage ourselves lets us change one design choice at a time on common data and charge each recipe for all of its compute.
Why do other studies find encoder-free as good as an encoder, or better? Most compare a pretrained encoder with an encoder-free model, which removes separate pretraining, dedicated capacity, and pre-fusion placement at once, so it can't show which choice matters. Some studies train both designs from scratch and find that encoder-free matches or slightly beats the encoder, mainly at smaller sizes, often due to using a reduced learning rate for the encoder or evaluating with metrics less sensitive to visual processing design choices.
@article{brown2026unbundling,
title = {Unbundling Multimodal Vision Encoding},
author = {Brown, Ellis and Hoang, Chris and Qian, Shengyi and
Li, Shang-Wen and Lewis, Mike and Ghazvininejad, Marjan and
Zettlemoyer, Luke and Beyer, Lucas and Ren, Mengye and
Verbeek, Jakob and Xie, Saining and Xu, Hu},
journal = {arXiv preprint arXiv:2611.17334},
year = {2026}
}Releasing Personal Agent Protocol 1.0
Sierra has released v1.0 of the Personal Agent Protocol (Poppy), enabling standard communication between user agents and company APIs.
Summary
Deep Dive
- Defines how personal agents connect to businesses via business agents, websites, and APIs.
- Leverages existing standards like MCP (Model Context Protocol), OpenAPI, and UCP (Universal Commerce Protocol).
- Uses OAuth for authentication, ensuring customers control agent permissions.
- Includes a commerce extension co-authored by Shopify for guest or authenticated shopping.
- Supports a 'validator' tool for companies to test protocol implementation.
Decoder
- Personal Agent: An AI capable of performing tasks on behalf of a user across different websites and applications.
- MCP (Model Context Protocol): An open standard for connecting AI assistants to systems containing data or tools.
- UCP (Universal Commerce Protocol): A protocol for handling e-commerce tasks like product search and checkout.
Original Article
Personal Agent Protocol — or Poppy for short — has gained real momentum since we first announced this new standard with our initial design partners Meta, Genesys, Instinct, NiCE, Rocket, Shopify, Stripe, and Walmart last month. Poppy enables personal agents to interact with companies through business agents, websites, and APIs and today we’re excited to announce that:
- v1.0 of the protocol is now available.
- Shopify has enabled Poppy for every merchant on its platform.
- Multiple design partners are now going live with Poppy, including Mercury, Synchrony, and Turkish Airlines.
- Poppy is now on for everyone who uses Muse, Meta’s AI agent, which automatically looks for Poppy first on every site it visits.
- 60 new companies are joining as design partners, bringing the total to more than 100 — including Affirm, Aflac, Agoda, Aircall, Akoya, Aviva, Banco Sabadell, Braze, Browserbase, Checkout.com, Cisco, Cover Genius, Creditas, Crunch Fitness, Cue, Delta Air Lines, Descope, EarnIn, Five9, Freshworks, Fullstory, Gen Digital, Global Payments, GoFundMe, GoodRx, Guardian, HealthEx, HelloFresh, Hyatt, Infoblox, Intuit, Kernel, Kin Insurance, Lemonade, Liberty Global, Manus, Mercury, Navy Federal Credit Union, Neptune Flood, Ora, Persona, Pindrop, Ping Identity, Profound, Prosus, Remitly, Runlayer, Salesforce (including Fin, Commerce Cloud, and Agentforce), Socure, SoFi, Symphony by Wix, The Knot, The Zebra, Travelers, Turkish Airlines, Twilio, Wayfair, Wix, WorkOS, and Zoom.
Poppy v1.0
Version 1.0 of Poppy reflects what design partners have learned over the past five weeks as they’ve tested the drafts against real websites, APIs, and agents.
The protocol defines how personal agents connect with companies that implement it, providing clear and comprehensive access to everything they offer, largely built with existing standards like MCP, OpenAPI, and UCP. With authentication through OAuth, customers control what personal agents can and can’t do, and can manage their access over time.
Since the initial draft, we’ve:
- Made it easier for companies to adopt the protocol in stages, starting with the basics and adding capabilities over time;
- Refined how the protocol uses existing standards;
- Expanded a company’s ability to provide guidance and collect feedback from the personal agent;
- Introduced extensions, such as the commerce extension co-authored by Shopify, so industries can build on the core protocol without changing it.
With v1.0, any company can implement Poppy on its own and immediately start delivering a better experience for customers who use Muse. Our implementation kit walks you through setup and includes a validator to check your work.
Sierra and Shopify bring Poppy to millions of merchants
Shopify powers businesses in more than 175 countries, and every Shopify store supports Universal Commerce Protocol (UCP) by default. As more shopping moves to personal agents, every Shopify store now automatically publishes a Poppy file, so merchants don’t need to do any extra implementation.
UCP and Poppy complement each other. UCP handles the shopping itself, like finding products, filling a cart, and checking out. Poppy handles what happens before that, when an agent shows up at a store, says who it is, and finds out what it’s allowed to do.
Shopify co-authored Poppy’s commerce extension, which links the two protocols. It lets an agent shop as a guest or sign in with the customer’s account, and it makes the agent ask the customer before taking certain actions, so it only acts with their permission. It also connects to UCP Ask, so a shopper’s agent can talk directly to the store’s agent.
Poppy in the wild
A number of businesses are now live with Poppy:
- Mercury puts personal agents to work on finances: People who use Mercury for business and personal banking can now ask their personal agent to check balances, review recent transactions, categorize spending, or keep budgets and receipts up to date. Customers can even ask their personal agent to initiate a payment or transfer, with Mercury ensuring the request goes through the account’s approval policy before anything is sent.
- Synchrony will bring financing and rewards into agent-led shopping: The work explores how a personal agent could help customers find the right product while considering affordability and relevant Synchrony offers and financing products along the way. Over time, that could create new ways to bring Synchrony capabilities into shopping experiences across their network of more than 500,000 merchant locations.
- Turkish Airlines welcomes personal agents from search to business class: Personal agents can now check out fares, flight status, and stopovers, as well as pull up a booking with a reference number and surname. Miles&Smiles members can even have their agent find business-class upgrades, which customers approve before paying. It’s now working to let members complete a purchase without leaving the conversation.
Choice, control, trust
We’re so grateful to all the companies that are contributing to this new standard, and excited to welcome more to our design partnership. As personal agents take on more of our everyday tasks, companies need clear, secure ways to work with them while continuing to deliver a trusted experience. Poppy gives them that foundation — so the company and the personal agent can get the job done for the customer they share.
We've made some big updates to Projects in Claude Code!
Anthropic updated Claude Code with proactive project coordination, allowing the AI to manage multiple threads and perform reminders autonomously.
Summary
Decoder
- Claude Code: A CLI tool from Anthropic that allows Claude to access local codebases to perform tasks like refactoring, debugging, and testing.
Original Article
We've made some big updates to Projects in Claude Code!
In a project, Claude coordinates your tasks across parallel threads. The coordinator is now much more proactive. It answers your questions in the project chat and gives you updates on the status of your ongoing threads.
/btw and /fork work in projects now too.
If something else comes up while you're working, /btw gives it its own thread so your current one stays on topic.
And if you want to try a different approach, /fork starts a new thread from the one you're in.
You can also set a reminder on any thread, or just ask Claude to remind you.
The thread moves out of your way until then. When it's time, it moves back and Claude brings it up in the project chat.
You can now share an individual thread as a read-only link with anyone who's signed in to Claude.
Open the thread's menu and pick Share! Threads that had access to your computer can't be shared.
Artifacts are easier to find again too.
The threads panel now has a row at the top with the ones you recently opened, updated, or pinned. Each thread also shows thumbnails of its own artifacts, which you can turn off in the view settings. Project settings and local folder access are simpler now too. It's all on web and desktop, and the latest mobile app has the new design.
Projects is in public beta on Pro and Max. If you don’t have it yet, you can join the waitlist: claude.com/form/projects
One Proton Away
Marathon Fusion claims it can transmute mercury into gold, pointing to a future where cheap atom sorting solves critical scarcity in isotopes.
Summary
Deep Dive
- Isotopes: Variations of an element with the same number of protons but different numbers of neutrons.
- Transmutation: The process of changing one chemical element into another by altering the atomic nucleus.
- Helium-3: A stable isotope used for cryogenics in quantum computing.
- Actinium-225: A radioactive isotope used in targeted cancer therapies to destroy tumors.
Decoder
- Isotope: An atom with the same number of protons but a different number of neutrons, changing its mass and stability.
- Transmutation: The conversion of one element into another via nuclear processes, often requiring high energy to move protons.
- Fission: A nuclear reaction where the nucleus of an atom splits into smaller parts, usually releasing energy and neutrons.
Original Article
One Proton Away
What happens when we can make any atom?
TL;DR: Cancer drugs, reactor fuel, fusion plants, quantum computers and deep-space batteries all depend on a handful of isotopes from aging reactors, Cold War stockpiles or a single foreign supplier. Whoever makes sorting and transmuting atoms cheap opens up all of those markets at once, from cancer treatment without a waiting list to space vehicles that reach the edge of the solar system.
One Proton Away
Many of us grew up on Cinderella, so you probably remember the Fairy Godmother. She waves her wand and a pumpkin becomes a carriage, the mice become horses, and Cinderella’s rags become a ball gown. Then the clock hits midnight and everything turns back.
That wand is what alchemists spent centuries chasing. Mostly they wanted to turn cheap metals into gold. Without modern physics, though, all they could really do was chemistry, and the best chemistry could manage was something that looked like gold: a shiny alloy, or a thin gold coating over something cheaper.
Here is why it never worked: chemistry only moves around the outer edge of an atom, the electrons that decide how atoms stick together. It does not touch what makes gold…gold. That lives in the center of the atom, and it comes down to protons: gold has exactly 79 protons. Add one more and you have mercury (80 protons). So the difference between a gold necklace and a metal that can poison you is exactly one proton. Pulling a proton out takes roughly a million times more energy than breaking a chemical bond, which no ancient furnace or acid was ever going to deliver.
The breakthrough came from physics, once scientists learned to hit atoms with particles and neutrons. In 1980, a team led by Glenn Seaborg made a few specks of gold out of bismuth (a brittle, silvery metal), but it cost far more to produce than the new gold was worth. So it could be done, but not economically. The hard part has always been how to make it affordable. On October 21, Marathon Fusion, a San Francisco fusion startup, said it had finally made gold from mercury in a way that could one day pay for itself. Worth noting: Marathon has not published how much gold it made, and the process needs a fusion plant that does not exist yet. So this is a signal, but not an actual gold mine yet.
Gold makes for a great headline, but the bigger story is that we are learning to make and sort atoms on purpose. That matters for cancer drugs, reactor fuel, fusion plants and even deep-space batteries, all of which are waiting on scarce isotopes (different versions of the same element).
What is an isotope?
Every atom has a tiny center (nucleus) made of protons and neutrons.
- Protons: These decide what the atom is. One proton is always hydrogen, two is always helium, and so on. It is also how the periodic table is ordered (the atomic number, usually at the top of each square).
- Neutrons: Adding or removing these does not change the element on its own. It just makes the atom heavier or lighter, and each of those versions is an isotope.
The number in an isotope’s name is protons plus neutrons. Hydrogen-1 has one proton and no neutrons, and it makes up about 99.98% of all hydrogen. Hydrogen-2 (deuterium) has one of each and is stable but rare. Add one more neutron and you get hydrogen-3, better known as tritium, which is radioactive and barely exists in nature.
The Toolkit
Sorting: Nature mixes isotopes together, and the one you actually want is usually the rare one. Since isotopes of the same element behave almost identically, separating them is slow and expensive.
Transmuting: Hit an atom with neutrons and you get a heavier or lighter version of the same element, often an unstable one. As it settles down, a neutron can turn into a proton (or a proton into a neutron), and suddenly it is a different element. That is transmutation. Think of it as the Fairy Godmother’s pumpkin trick, done with physics.
From 1955 to 1963, Oak Ridge sorted lithium using about 24 million pounds of mercury, then shut the process down once it had enough, leaving a mercury cleanup that is still going on today. As far as anyone knows, China is the only country that still runs this process openly. The good news is that lasers, plasma centrifuges and new chemistry can now do the same job much more cleanly.
The Atoms that the Future Needs
The table below covers a handful of the isotopes that matter most: what each one is used for and how we get it today. You will spot a pattern pretty quickly. Almost all of them come from a Cold War leftover, an aging reactor or a single foreign supplier. Physics is not what makes them rare. It is that almost nobody has built a new way to make them in decades.
| Isotope | Who Needs It | How we get it today |
| Actinium-225 | Cancer drugs that carry it straight to tumor cells and destroy them from the inside, sparing healthy tissue | Collected as thorium-229 decays; the thorium comes from Cold War uranium-233 stored at Oak Ridge |
| Uranium-235 (LEU, and HALEU at 5% to 20%) | The fuel that keeps a nuclear fission reactor’s chain reaction going; new smaller reactors need a stronger blend called HALEU | Gas centrifuges at a select few companies, with Russia a major supplier |
| Lithium-6 | Nuclear fusion plants line their reactors with it, because each lithium-6 atom hit by a neutron turns into fusion fuel. | Made at scale only in Russia and China; the US relies on a weapons stockpile from the 1960s |
| Helium-3 | Quantum computers need it to chill their chips to just above absolute zero, the only temperature where many of them work. | Collected from decaying tritium, in the US at Savannah River (South Carolina) |
| Plutonium-238 | Its steady heat powers spacecraft too far from the sun for solar panels, like the Mars rovers. | Neutrons hitting neptunium-237 targets at two DOE reactors in Tennessee and Idaho. |
What does a world of abundant isotopes look like?
Medicine: Imagine cancer treatment without a waiting list. In 2023, a shortage of the prostate cancer drug Pluvicto left new patients waiting months to start treatment. Drugs built on even scarcer isotopes, like actinium-225, are rationed today. With enough supply, they could move out of trials and into everyday care.
Nuclear power: Most fission reactors run on uranium enriched to 3% to 5%. Many of the newer, smaller designs need a stronger blend, between 5% and 20%, so they can run longer between refuelings, and Russia is the main commercial seller. Lithium-7, which reactors use to manage their water chemistry, is the same story: it only comes from China and Russia. If supply opened up, reactor builders could stop worrying about where their fuel comes from.
Fusion: Most fusion plants plan to run on tritium, the form of hydrogen that barely exists in nature. The idea is to make it on-site by wrapping the reactor in lithium-6, which turns into tritium when a neutron hits it. Right now, nobody in the West can enrich lithium-6 at industrial scale. Solve that, and the same neutrons can also make medical isotopes, and even gold.
Quantum: Quantum computers have to be kept colder than outer space, and the fridges that get them there use helium-3. A big one holds about 40 liters, roughly $100,000 worth. Nearly all of it comes from tritium slowly decaying inside nuclear weapons stockpiles, and nobody is making more at scale. Cheap helium-3 would remove one of the stranger bottlenecks in building bigger machines.
Space and defense: Far enough from the sun, solar panels stop being useful. Spacecraft out there run on plutonium-238, which gives off steady heat for decades that a nuclear battery turns into power. The US makes only a few kilograms a year, enough for a nuclear-powered NASA mission every few years.
Where We Would Bet
Isotopes of the same element are nearly identical, which is what makes separating them so hard, and new companies are getting clever about telling them apart. Every isotope absorbs a very specific color of light, so a laser tuned to exactly the right color can give one isotope an electric charge while leaving the others alone. Then the charged atoms can be pulled out. Plasma centrifuges turn material into a charged gas and spin it so fast that the slightly heavier isotopes drift to the outside. Whoever cracks cheap sorting of isotopes (that would allow us to extract them at will) will do extremely well.
Fusion plants selling electricity to the grid are still years away because no one has sorted out how to create an economically viable fusion reactor (yet). But you do not need a full power plant to make neutrons, and neutrons are what turn ordinary atoms into valuable isotopes. SHINE already uses small fusion reactions to make medical isotopes it sells today, and Marathon plans to do the same long before its gold matters. That is real revenue for a fusion company years before it might sell its first kilowatt-hour to the grid.
Takeaway: Cancer drugs, reactor fuel, fusion plants, quantum computers and deep-space batteries all depend on a handful of isotopes from aging reactors, Cold War stockpiles or a single foreign supplier. Whoever makes sorting and transmuting atoms cheap opens up all of those markets at once, from cancer treatment without a waiting list to space vehicles that reach the edge of the solar system.
And this time, the magic does not wear off at midnight.
Have a great weekend,
Josh
AI, work from home, and the young-graduate job market
Data suggests work-from-home adoption, rather than AI exposure, is the primary factor behind the declining employment rates for recent college graduates.
Summary
Decoder
- Agglomeration effects: The benefits gained by firms and people being located near one another, such as knowledge spillovers, access to a larger labor pool, and reduced interaction costs.
Original Article
AI, work from home, and the young-graduate job market
New college graduates are employed less than a few years ago. AI might be a factor, but there is growing evidence that work from home is playing a meaningful role.
Introduction
Young college graduates in the US are seeing a weakening labor market. Among workers age 22–26 holding a bachelor’s degree, their employment-population ratio was 88% in September 2026, a full percentage point below where it was three years earlier and nearly a point-and-a-half below its immediate pre-pandemic level. This group’s unemployment rate was close to 6% at the end of 2025—though appears to be cooling now—versus around 4% pre-pandemic. And those who are unemployed are spending more time looking for work as well: the typical unemployed young graduate has spent 12 weeks looking for work, a third higher than a few years ago.
These soggier job prospects for college grads have raised concerns that AI is reducing hiring and employment for young people just beginning their careers. As we discuss in the next section, while the evidence around AI’s employment effects is early and inconclusive, it is a hypothesis worth taking seriously. But a narrative that blames AI entirely for this slowdown has complications. First, as Ozimek and Goldschlag (2026) have shown, youth employment is also down since 2023 for young non-college-graduates, who are less likely to be in industries using AI and in jobs exposed to AI. In fact, young non-graduate employment rates have fallen slightly more since December 2019 than young graduate rates, while mid-career workers age 30-49 have fully recovered to pre-pandemic levels with little difference between those with and without a bachelor’s degree.
Second, AI is far from the only shock weighing on recent economic data. The modern era of AI began in late 2022, the same time other major economic shifts were also beginning to take shape. This makes it hard to disentangle the impact of any one factor. High inflation and the resulting higher interest rates likely pinched certain industries. Job switching surged in the pandemic during the “Great Resignation,” and then it swung to record lows thereafter as companies pulled back on hiring and firing, which could be disproportionately hurting new labor force entrants like recent grads. Immigration flows saw dramatic rises and falls too, changing job dynamics in uncertain ways.
Throughout this entire period, work from home (WFH) grew in adoption and cultural acceptance. WFH is a mixed bag when it comes to the labor market. There is increasing consensus that WFH helps the job prospects of disabled workers and older workers. That is a substantial benefit which ought not be dismissed. As we show in this piece, however, a trade-off is that WFH weakens the employment of young people: without the benefits of in-person learning and agglomeration effects, employers are likelier to hire older workers for jobs where WFH is now an expectation. Young workers themselves might also demand WFH jobs less than older workers.
Research so far around WFH and youth employment has been mixed
There is a growing economic literature on the effects of WFH. Bloom, Dahl, and Rooth (2026) find that remote work increased full-time employment among physically disabled Americans. Burn, Gentry, and Lahey (2026) use a related approach for the UK, and they find positive occupational full-time employment effects for disabled workers and workers ages 50–64.
At the same time, there are concerns about the effect on young workers. Emanuel, Harrington, and Pallais (2026) find that physical proximity to the office increases the job feedback received by software engineers, especially younger and less-experienced workers. Their national analysis also finds a relative increase in young graduates’ unemployment in remotable occupations. Lambert and Schindler (2026) find declines in junior hiring shares associated with WFH exposure after controlling for AI exposure.
Many AI studies also address the experiences of young workers. Alex Imas and Jacob Schaal have an excellent overview of the recent AI labor market research. Brynjolfsson, Chandar, and Chen (2026) find that young workers in AI-exposed occupations saw slower employment gains than older workers and workers in less AI-exposed occupations; their findings hold after controlling for a variety of potential confounding factors including WFH. Tucker (2026) finds a sharp decline in hiring of workers ages 22–24 in the most AI-exposed industries after ChatGPT’s release in late 2022. Orr, Tucker, and Warren (2026), using linked US Census Bureau records, find that graduates in the most AI-exposed tenth of college majors saw initial employment fall by about five percentage points and initial earnings by about 13% relative to less-exposed majors after late 2022.
But these findings are not universal. Fairlie and Wu (2026) find no unusual relative summer 2026 unemployment increase among non-enrolled college graduates age 22–25; they also find some positive teleworkability interactions. Kauhanen and Rouvinen (2026) find no systematic AI-related displacement or persistent wage differences by AI exposure among young workers in Finland through September 2025. Frank et al. (2026) find that the deterioration in US graduates’ entry into AI-exposed jobs begins before ChatGPT; graduates with more AI-exposed coursework have shorter job searches after its release and higher first-job pay when entering highly exposed occupations. And while Ando et al. (2026) find a negative relationship between AI exposure and labor demand through job openings, they also find no significant difference in this effect between junior and senior roles, and even a somewhat smaller negative effect among very junior (<1 year experience) openings.
Public data suggest a strong WFH effect on young college graduates, less so from AI
To explore whether WFH has been disproportionately weighing on recent college graduates, we use the Current Population Survey (CPS) to compare the growth in payroll employment-population ratios for recent grads to those of middle-career workers between 2023—when recent graduate full-time-equivalent (FTE) employment rates peaked—and the last 12 months. Our main measure of interest is the difference between the two groups’ ratios. Say an occupation’s young-grad payroll employment-population ratio falls 10%, while it falls 4% for middle-aged workers: the outcome we measure is a six percentage point relative decline for young grads. We track this spread for sets of broad and detailed occupations in each of the nine major Census Bureau divisions, and then we compare it to the WFH adoption rates in that occupational category also measured in the CPS.
We also test the possibility of AI weighing on young-grad employment by comparing our young-grad employment spread to two other metrics: the composite measure of occupational AI exposure in Gimbel, Kendall, and Kulsakdinun (2026), which combines several studies, and AI adoption by industry in the Census Bureau Business Trends and Outlook Survey (BTOS).
The detailed regression results are in the appendix, but we summarize the relationships we tested in the following figure. The left panel shows the relationship between the occupational WFH share (x-axis) and growth in our young-grad–mid-career employment spread (y-axis). The relationship is negative and statistically significant. Between 2023 and the last 12 months, for every extra 10 percentage points of WFH adoption, employment rates for young college grads grew between 3.9 to 6.6 percentage points more slowly than those for middle-aged workers.
As an accounting (not causal) exercise, we took these results and looked at how much of the 1.23-percentage-point decline in the young-grad FTE employment-population ratio between 2023 and 2026 could be explained by WFH. The answer was between 0.69 to 1.08 percentage points, explaining 56% to 88% of the total decline. When just looking at extensive-margin employment rate declines, WFH explains 97% to 153% of the total.
The middle panel replaces WFH adoption with AI exposure as a separate explanatory variable. As jobs become more exposed to AI, the effect on the new-grad employment spread is also negative but much shallower than with WFH and statistically indistinguishable from zero. When we include both WFH and AI exposure, the WFH effect remains statistically-significant while AI exposure remains insignificant. The same is true of industry AI adoption, shown in the right panel: industries that self-reported a higher likelihood of using AI in 2026 H1 actually saw, if anything, slightly faster young-grad employment growth relative to mid-career, though again, this effect is statistically insignificant.
Stripe data suggests young workers lost ground in regions where WFH proxy spend grew more
At Stripe, we do not have a direct line of sight into whether a person is working from home. What we can see, however, are proxies for WFH, such as spending on office supplies and furniture. As the following chart shows, the level of office supply, furniture, and electronics spending in the Stripe data relative to a state’s gross domestic product (GDP) is moderately correlated with WFH adoption rates.
Given this, we look at an even deeper geographical cut: how proxy WFH spend at the commuting-zone (CZ) level grew relative to each CZ’s GDP between 2023 and 2024. We then compare growth in proxy-WFH-spend-to-GDP to growth in 22- to 29-year-old employment-population ratios in the top 100 CZs by spend. We only look at spend at merchants present in both 2023 and 2024 to keep Stripe’s mix of business from being a factor.
The data shows that CZs where proxy WFH spend grew faster than the regional economy were likelier to see declines in employment of those ages 22–29 from 2023-24, with a moderate correlation. This does not establish a direct relationship between WFH and youth employment, but it is a surprising finding: one might have expected fast growth in office supply and electronics spend to be associated with the sort of healthy regional economy that’s more likely to employ more young workers.
Conclusion
Our analysis suggests that WFH is an important factor moving the needle on young-graduate employment. Telework adoption explains the decline in their employment better than AI exposure in our model using public data. And our own Stripe data suggests a negative relationship between regional spend on WFH proxies and growth in youth employment.
The takeaway of this analysis is not, however, that WFH explains everything around young-graduate employment while AI explains none. Our model is suggestive rather than truly causal. Moreover, AI exposure and teleworkability are highly correlated with each other, making it hard to distinguish definitively between the two factors, let alone the other factors we have not tested here, such as higher interest rates and past hiring surges. AI research, and indeed AI itself as a general-purpose technology, is still in its early stages, and even if overall employment stays high, AI could displace specific industries and occupations. Moreover, the full extent of AI effects on the labor market might not manifest themselves until the next recession, much like the effects of the internet did for travel agents. Such trade-offs between short-run narrow displacement and long-run aggregate augmentation would, in fact, be entirely consistent with the history of technological shocks. As we have emphasized before, researchers should keep an open mind at this early stage in AI’s deployment.
Methodological appendix
CPS analysis
Stripe’s analysis of the CPS compares the January to December 2023 period to the 12 months ending in September 2026. We exclude October from both periods since the CPS was not collected in October 2025 due to the US federal government shutdown, so each sample period has 11 months.
We look at employment growth between the two periods. Our two alternatives measure employment slightly differently: our preferred intensive-margin specification converts usual weekly hours in the primary job into full-time-equivalent terms by dividing by 40, and then it takes as a percent of the cohort population. Workers whose usual hours vary are assigned their actual hours. The other extensive-margin approach just looks at binary employment, with occupation assigned by primary job. In both cases, we limit to workers in payroll (i.e., non-self-employed) jobs in the CPS.
Our cohort of interest is “young college graduates,” 22- to 26-year-olds who:
- have exactly a bachelor’s degree;
- are not enrolled in school; and,
- did not report a disability.
The control group for comparison is non-disabled 30- to 49-year-olds of any educational attainment. The dependent variable of interest is percent growth in the employment-population measure for 22- to 26-year-olds minus that for 30- to 49-year-olds.
Work from home is measured as the share of all employment in an occupation-Census Bureau division’s cell that reports any telework in the CPS in the latter 2025-26 sample period.
The AI occupational exposure measure we use comes from Gimbel, Kendall, and Kulsakdinun (2026). The AI adoption measure we use is industry-based and comes from the US Census Bureau Business Trends and Outlook Survey, pooled over the periods spanning January to June 2026.
We adjust CPS survey weights for pre-2026 population controls. We also apply single-age compositional adjustments to fix the age demographics to November 2025-September 2026 proportions. Reported standard errors are robust.
Apple Launches New App to Study How People Use AI for Health
Apple has launched an AI Health Usage Study requiring an Apple Intelligence-enabled device to analyze how users interact with medical information provided by chatbots.
Summary
Decoder
- HealthKit: Apple's framework that provides a central repository for health and fitness data on iPhone and Apple Watch.
Original Article
Since 2019, when Apple first launched its Research app, the company has partnered with medical universities and institutions to run a range of health studies on hearing, heart health and movement, women’s health and, most recently, the Apple Health Study, which covers a wide variety of health areas.
On Sunday, Apple launched its latest study, and it comes with its own app: the Apple Health AI Study.
During a press briefing, Apple said the study aims to understand how people interact with conversational health AI tools. To do so, it will examine comprehension, response quality, the drivers of user trust and everyday health actions.
The goal? To inform the responsible design of health AI experiences for both Apple and the industry and to explore how health AI might help people prepare for doctor’s visits. Key questions the study aims to address include: Do users understand the information provided by health AI? Do they trust it? Is it helpful?
This is particularly relevant considering that one in three US adults use AI chatbots for health information.
Study data will not be used to train AI models, but it may inform Apple’s feature development.
Apple’s study partner
To facilitate the study, Apple is partnering with OpenEvidence, an AI platform designed specifically for doctors. It generates answers using medical information from these partners: The New England Journal of Medicine, Journal of the American Medical Association, National Comprehensive Cancer Network, Nature and Cochrane Database of Systematic Reviews.
In the dedicated app, study participants can ask the AI their text-based health questions and then see answers with linked, peer-reviewed clinical citations. They can also choose to share their HealthKit data, including sleep, lab results, activity and resting heart rate, for more personalized answers.
HealthKit is Apple’s hub for health and fitness data collected on the iPhone and Apple Watch.
Who’s eligible?
The Apple Health AI Usage Study is open to 200,000 participants. You must be in the US and age 21 or older to participate. An Apple Intelligence-enabled iPhone (iPhone 15 Pro or later) running iOS 26 or later is also required.
Why an Apple Intelligence-enabled iPhone? According to Apple, the study app requires the processing capabilities of those iPhones, and it uses on-device Apple Foundation Models that are part of Apple Intelligence. These models provide access to any large language model. For example, as part of the study, these models are used to classify question categories.
What’s required?
Each participant is asked to participate for up to 12 months. During this time, they can ask questions about general health, specific symptoms, recovery from a procedure and their own uploaded health data, as examples.
In addition to a baseline survey, participants will receive monthly surveys and additional surveys specific to preparing for doctor’s visits.
Participants will also be prompted to check in with the app. For instance, if you have a paired Apple Watch, you will receive notifications to check in, along with suggested topics to explore.
If there is a complete lack of participation, a participant may potentially be removed from the study.
Study privacy
After downloading the Apple Health AI Study app, participants must complete a mandatory onboarding process that includes informed consent. They will also be asked to share their demographic information, such as race, age and education level, which will be used to explore how these factors may affect their preferences and communication needs. Then, they can start asking the AI questions.
The iPhone first processes questions using tools designed to remove details that could identify participants. They then go to a secure cloud-based study environment managed by a third-party study support contractor called UST Global, where queries are delinked from individual participant IDs and screened using tools designed to further remove any remaining personal details. Lastly, this info is sent to OpenEvidence, where answers are generated using supporting reference materials.
OpenEvidence will delete both question and response data within 24 hours.
A small group of Apple researchers will have access to coded versions of survey responses, separated from personally identifiable information, summaries of conversations and categories of the most common conversation types.
Within the study app, there are also safety guardrails that detect concerning questions and respond by suggesting that a participant follow up with a provider or other resources if needed.
There is no live monitoring, and study staff won’t contact participants if any questions related to self-harm are asked.
It’s also up to the participant whether they share any HealthKit metrics. They will first have to choose to upload their clinical health records to Apple’s Health app, and then separately choose to share those records from the Health app to the new Apple Health AI Study app.
Availability
The Apple Health AI Study app is now available in the App Store. As of now, there is no set publication date for the study.
Recursive, Self, Improvement
Recursive self-improvement in modern AI is less about individual machines rewriting their own code and more about a systemic pipeline where AI-generated research accelerates its own successor training.
Summary
Deep Dive
- Deep learning blurs the line between cognitive algorithms and knowledge, as both are encoded in weights.
- RSI is best analyzed as a system-level process, not a property of an isolated model.
- Feedback channels include algorithmic efficiency gains and reinvestment of capital into compute.
- Automating research replaces human researchers with increasingly efficient, scalable AI researchers.
- Acceleration does not imply a single 'takeoff' point; it is a superlinear trend.
- Real-world constraints (compute, wall-clock time) mean that progress is constrained by physical production, not just software.
Decoder
- FOOM: An internet-slang term, popularized by Eliezer Yudkowsky, describing a rapid, discontinuous explosion in intelligence where an AI suddenly upgrades itself beyond human control.
Original Article
Recursive, Self, Improvement
The anatomy of an intelligence explosion
In a 1965 paper, the mathematician I.J. Good introduced the idea of an intelligence explosion. Defining an ultraintelligent machine as one that can “far surpass all the intellectual activities of any man however clever,” Good observed: “Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines.” The result would be an “intelligence explosion,” with an ever-widening gap between human and machine intelligence. “Thus the first ultraintelligent machine is the last invention that man need ever make,” Good concluded, “provided that the machine is docile enough to tell us how to keep it under control.”
Sixty years later, AI has become capable enough to help AI researchers improve their own research. By writing code, doing experiments, and so on, human researchers are creating a product that helps them to write code and do experiments faster and better. These systems not only help humans to work more effectively, but are becoming more autonomous, increasingly able to function as if they were employees: the AIs are training the next AIs. There is accordingly a growing sense among researchers, policymakers, and senior executives at labs that we are either in, or on the cusp of, a regime of recursive self-improvement (RSI).
The term RSI was introduced, and later popularized, by Eliezer Yudkowsky to describe a particular mechanism by which improvements in AI can lead to further improvements. In Yudkowsky’s account, this mechanism had profound implications for safety and human control: the ability to direct it would accumulate within whichever organization first gained a substantial lead, rather than being widely distributed; humanity would lose the ability to determine its future as the AI rapidly became too intelligent for us to control.
It is tempting to infer from the feedback loops increasingly visible in AI development that the implications long associated with RSI must follow — or, conversely, from their apparent absence that RSI is a mirage. But intelligence can contribute to the production of further intelligence via several distinct feedback mechanisms. In Yudkowsky’s terminology, these include cascades, where one advance leads to another (as farming led to writing); and cycles, where a process’s output can become its own input (as in a nuclear chain reaction). RSI — at least in Yudkowsky’s original conception — involves a more specific form of feedback, where improvements to the cognitive machinery or the processes that produce it make subsequent research more productive.
But deep learning does not involve, as Yudkowsky had then imagined, machines modifying their own source code. Nor are we starting out with the “ultraintelligent machines” whose capabilities Good argued would suffice to trigger an intelligence explosion, but with something considerably more modest. We therefore need to establish which feedback mechanisms are actually operative, and how strong they are.
To assess this, I disentangle the three components involved in RSI, and show how the deep learning paradigm forces us to reconsider what counts as the “self” undergoing improvement. I argue that instead of an individual model, the relevant unit of analysis becomes an AI development system that trains successive models. Accordingly, the question is no longer one of how much a mind gains by improving its own cognitive machinery, but how strongly the outputs of that system feed back into its capacity to produce further improvements. In particular, I trace two principal channels through which this feedback operates, algorithmic progress and investments in hardware.
Since these channels have been feeding back on each other for years, there is no single threshold at which RSI “begins.” We should speak instead of the acceleration of AI development, recognizing that rapid progress need not involve a discontinuity or even sustained acceleration. And because the mechanisms determining the pace of progress also shape the opportunities to observe, intervene in, and direct it, this revised account has implications for both the prospects of human control and the concentration of power.
RSI, unpacked
To understand what “recursive self-improvement” entails, we should examine its three constituent parts.
- Recursive: How much does each improvement increase the capacity to produce further improvements?
- Self: Which necessary contributions can the process supply for itself, and which must come from outside?
- Improvement: Do the resulting changes constitute genuine improvements in capability, rather than merely apparent gains?
Any two of these can be present without the third: a system can improve itself without becoming better at further improvement; it can drive recursive improvement in something other than itself; and it can recursively modify itself without improving.
An illustration of the third is contained in Tom Reed’s essay warning of a “Goodhart Singularity,” a possible failure mode in which “An isolated AI improving itself against benchmarks would only appear to be approaching superintelligence, while actually optimizing for eval performance that fails to generalize beyond the lab.” The system could recursively self-modify without becoming meaningfully more capable.
But even if genuine improvements are to be found, there is no guarantee that improvements increase the capacity for further improvement. That is, not all forms of feedback are recursive: a bakery can adopt a better recipe and thereby sell more bread, but the recipe does not help it find the next, even better recipe. By contrast, if a scientist develops better experimental methods, and then uses those methods to discover even better methods, this is recursive feedback: she is improving the process of improvement itself.
However, real-world development processes are not as simple as these clean cases. Different feedback mechanisms can coexist in the same process: cascades, cycles, and recursion may each be visible at different levels of abstraction. AI products can generate revenue, which can be used to acquire more hardware, enabling a larger population of automated researchers, which make further discoveries, which produce more capable models. Each step enables further development, forming a cascade. If the improved models generate further revenue and research investment, this process also forms a cycle. And if the resulting research improves the research capabilities of systems conducting subsequent research, it exhibits recursive improvement.
This was the picture painted by Carl Shulman in his intervention into the Hanson–Yudkowsky FOOM debate over whether the development of AI would produce a rapid, concentrated takeoff. Robin Hanson expected emulations of human brains to arrive before “hand-coded” AI, and to change the economy’s growth rate as farming and industry had done, perhaps from doubling every fifteen years to doubling every month; the economy would “settle into a new, faster growth rate,” with no single mind racing ahead of the rest. Shulman showed that even Hanson’s imagined economy contained the components of an intelligence explosion: emulations could run at fast speeds, earn money, acquire more hardware, and use their vast numbers and speed to make research advances that improve the technology producing them.
I judged his intervention victorious because it was “much closer to the mark” (given the actual trajectory of deep learning) than Yudkowsky’s picture of a mind rewriting itself. Within such an economy, no individual mind need improve itself; each generation improves the process that produces the next. This is the second case, in which a system drives recursive improvement in something other than itself. Current AI development most resembles this case: models do not rewrite themselves, but help to produce their successors. If we are still going to call this RSI, we had better examine what the “self” is; the relevant feedback mechanisms look quite different if it is a training pipeline rather than a mind.
What is the self in deep learning?
Where we draw the boundary around the “self” changes which feedback mechanisms we see as self-improvement, and what we expect from them. To see why, it helps to revisit distinctions underlying Yudkowsky’s original account of RSI.
Yudkowsky distinguished five levels at which improvements can occur.
- The metacognitive level is the optimization process that produces cognitive machinery, like evolution producing the human brain.
- The cognitive level refers to the algorithms responsible for reasoning and learning.
- The metaknowledge level consists of tools for acquiring knowledge, like the scientific method.
- The knowledge level is our understanding of the world, say, what we know about gravity and materials.
- And at the object level, we solve particular problems, like building a bridge.
He argued that the higher the level at which improvements take place, the more vigorously they propagate downward. Most human progress has come from improvements to lower levels — figuring out how to solve an object-level problem better, or acquiring knowledge that enables us to solve a whole host of object-level problems better. Perhaps the Scientific Revolution was a discontinuity because it greatly improved our metaknowledge, our capacity to discover. But RSI would lead to a more discontinuous change, because the object level becomes the metacognitive level — intelligence turns its problem-solving abilities toward improving those very problem-solving abilities.
But almost two decades on, it is not so clear that this framework holds up in the face of deep learning, which blurs the distinction between these levels. Although Yudkowsky accepted that the boundary between an AI’s reasoning algorithm and its knowledge was theoretically blurry, he argued that “in practice it’s often possible to distinguish cognitive algorithms and cognitive content.” But in a neural network, there is no clean separation between what a model knows and how it reasons. Its knowledge of the world, its representations of concepts, and its habits of learned reasoning are all embodied in the same weights. Rather than being individually programmed, these are acquired through training, in which an architecture is exposed to data and its weights adjusted according to a training objective. Thus the metacognitive level, the machinery that produces cognition, includes the training process itself: its algorithm, data, environments, and compute.
With Harry Law, I have argued that what training produces is a map of the world together with a vast stock of procedures for navigating it — among them, ways of proposing possibilities, testing them, and revising them to learn and solve problems. In Yudkowsky’s terms, the map is knowledge, and the procedures are largely metaknowledge. Together they constitute a model’s cognition. Improving cognition, then, means finding better procedures and expanding the knowledge they work on: a model that learns a more effective way of investigating a problem has improved its cognitive machinery, even if its training algorithm and architecture are unchanged. This is importantly different from a picture in which learning merely supplies additional knowledge to an otherwise fixed cognitive algorithm.
Under deep learning, the natural unit of recursive improvement is therefore not an individual model but an AI development system that conducts this search for knowledge and procedures. Trained models can contribute to research that improves the training process, or generates better inputs to it, thereby producing more capable successors. The same system can improve its research capacity by developing more efficient algorithms, running more or faster automated researchers, or acquiring additional compute and resources. Its cognitive output thus becomes an input to the production of further cognitive capacity.
But then, nor should we treat the whole economy as the “self.” Some argue that AI is a “normal technology.” On this view, it is one of the many technologies that together contribute to the present rate of about 2% annual growth, through familiar processes of innovation, adoption, and reinvestment. But the point here is not simply that technological progress generates resources that can be reinvested in research, or that researchers over time develop better tools and methods. Indeed, that “recursive” mechanism is not distinctive and would not yield an intelligence explosion. What is distinctive here is that the technology being improved increasingly produces the researchers themselves: improvements can yield more capable researchers as well as more of them. Research capacity can therefore grow much faster than the wider economy.
However, this process isn’t self-contained: it still relies on inputs supplied by the wider economy, including hardware. “Returns on cognitive reinvestment,” in Yudkowsky’s terminology, are not a property of intelligence in general, but of particular investments, made by particular systems, at particular stages of technological development. The feedback operates through two principal channels: research leading to hardware improvements and algorithmic progress, and spending to build or buy more hardware. The strength of this feedback depends on the returns to each, and how each changes the returns to the other.
Returns and bottlenecks
Neglect of this interplay has caused the pace of AI progress to be frequently underestimated, even by forecasters who anticipated scaling and the importance of compute. Carl Shulman, for example, has said that prior to the deep learning revolution, he expected AGI by roughly 2050, on the basis of Moore’s Law and a “vague uniform prior” over orders of magnitude of compute, smeared between the plausible lower and upper bounds of the compute required. At the lower end, a tiny fraction of an insect’s brain; at the upper end, evolution, a blind search process that stumbled on human-level intelligence without even trying, and which deliberate search should be able to beat. But now, he has more probability mass concentrated in the coming years, because of the massive scale-up in compute. As Leopold Aschenbrenner put it: “You and I can reasonably disagree on the median time to AGI [...] but given how we’re racing through the OOMs right now, certainly your modal AGI year should [be] sometime later this decade or so.”
But the compute scale-up was not an exogenous shock. It was an endogenous response to algorithmic progress: the unexpected success of language models based on the Transformer architecture made enormous investments in compute worthwhile. Had the algorithms failed to yield useful results, the growth of compute would have continued at its earlier rate. While Shulman’s forecast missed this response of spending to algorithmic success, Ajeya Cotra’s Biological Anchors report missed the reverse effect, by which greater compute enables larger experiments and hence further algorithmic discoveries. By extrapolating the prevailing trend, it anticipated much of the growth in physical compute, but underestimated algorithmic progress, assuming roughly 1.3x annual improvement in algorithmic efficiency, compared with subsequent estimates around 3x. Each forecast anticipated one source of progress while underestimating how each channel would accelerate the other, contributing in Cotra’s case to a forecast of 3.6x annual growth in effective compute, against the roughly 10.7x subsequently estimated.
So, the extraordinary growth in effective compute resulted from a feedback loop in which investment in compute and the discovery of better algorithms each raised the returns to each other. How does this process change as AI systems increasingly conduct AI research themselves?
Until recently, all of this research was done by humans, but over time, the researchers will be AIs — improvements to the models become improvements to the researchers developing their successors. Because a model’s cognition consists of knowledge and procedures picked up through training, these automated researchers do not need to discover the General Theory of Intelligence in order to make improvements: they can advance progress the way human researchers do, by designing experiments, generating data, building training environments, and so on.
This doesn’t rule out insight, another core mechanism Yudkowsky identified as a potential source of discontinuity. But it does not eliminate the need to find and test improvements; there is no shortcut via a seed AI uncovering a fundamental insight about intelligence: improvements must still come from search and experiment, at a cost in compute and data. The route to more capable systems is unchanged. Automating research just gets us there faster.
How much faster? This depends on how effectively improved cognitive labor can be converted into further progress. There are reasons to expect large gains. Human researchers have already made extraordinary progress, and automated researchers can be copied, operated faster, and organized into larger research groups, while successive generations become better at the intellectual work itself. Better researchers can improve experimental design, extract more information from smaller runs, and discover more efficient methods of learning, thereby relaxing the compute constraint. But experiments still consume compute and wall-clock time, and hardware supply responds to investment only with a lag.
Whatever its size, this is not an acceleration that will switch on at a single threshold. Downstream of the original Yudkowskian way of thinking, people will often discuss whether RSI has started, or when it will begin, or other forms of words implying that at some point, AI will become able to suddenly “go FOOM.” Yudkowsky placed this point no later than, and probably well before, “a minimally transhuman mind capable of writing its own AI theory textbooks.” But an AI development system has no such point: its models contribute long before they could write any such textbook, to a process that was already improving itself.
There’s no critical point like replacing human researchers where this dynamic manifests. If AIs could do the work of the existing human research workforce, replacing them one-for-one wouldn’t change the rate of progress. As automated researchers grow more numerous and capable, progress can accelerate without a sudden “closing of the loop.” Rather than ask when RSI will start, just call this what it is: the acceleration of AI development. That acceleration is already underway; it will intensify as AIs take on more research.
Over any short period, each step can look like a one-time speed-up: a new trend, steeper than the last. Thomas Kwa counts the latest as roughly the fourth or fifth “one-time” speed-up in deep learning, suggesting that these apparent trend breaks are better read as one superlinear trend through the noise. The feedback described here explains this pattern: improvements increase the research capacity available to find further improvements, including ways to relax the remaining bottlenecks; of course, the strength and persistence of the acceleration depend on how these gains compare with the growing difficulty and research costs of further advances.
It is beyond the scope of this essay to fit quantitative estimates to the magnitudes of the relevant parameters. My argument concerns the structure of the feedback, and it seems to me that most formal modeling of takeoff dynamics conforms to the model described here. For instance, Davidson’s compute-centric framework, Cunningham et al.’s METR preprint, Epoch’s GATE model, the AI Futures Model — all track progress through effective compute, with returns to research that diminish as ideas get harder to find, and with feedback through research, spending, or both. These models make different empirical assumptions, like how far compute bottlenecks research, but these concern not whether the system contains recursion, but how much each increase in research capacity improves the system’s ability to generate still further capacity. As Yudkowsky illustrated, a self-optimizing compiler can improve itself without going FOOM; recursive gains are not always strong and sustained, but an AI development system need neither FOOM nor flatline: it can accelerate progress at a wide range of speeds between the two.
Takeoff speeds and Good’s proviso
An intelligence explosion, then, does not need to be discontinuous to be fast: an enormous acceleration does not require a FOOM, or even a rate of improvement that keeps accelerating. Even a speed-up that never compounds could compress timelines dramatically: if AI development as a whole ran ten times faster, then — keeping fixed the sequence of discoveries, their difficulty, and required resources — a century of progress would take a decade.
Contrast this with a speed-up that starts much smaller, but compounds as improvements continually increase its research speed. Given long enough, sustained acceleration may eventually outweigh the larger, one-off gain; but for a given target — say, surpassing human intelligence — the larger initial speed-up may reach it first, especially if it is not too distant. The existence of strong recursion is therefore not the primary determinant of how quickly a particular capability is reached.
I think we have good reason to expect a very large increase in research productivity, even if software improvements don’t keep accelerating: replacing a small population of human researchers with vast numbers of increasingly capable AI ones should greatly accelerate progress, unless the rising difficulty of further advances happens to offset that growth almost exactly. And an intelligence explosion doesn’t depend on returns on cognitive reinvestment alone. However far compute constrains software progress, the system also earns returns on economic investment: more useful AI attracts capital and compute, while AI research improves how well that compute is used — the same loop that drove the past decade’s growth in effective compute. And unlike benchmarks, those returns depend on real-world results, providing a ground-truth check against Reed’s Goodhart singularity.
How this acceleration comes about bears on Good’s proviso — that the first ultraintelligent machine is the last invention we need ever make, “provided that the machine is docile enough to tell us how to keep it under control.” In Yudkowsky’s account, the discontinuity was a source of danger. As he put it, “You can’t count on warning, or time to react.” A mind improving itself had to be aligned on the first critical try, and even if alignment were solved, whoever got there first would be in a position to take over the world (though he disclaimed having any intention of this himself).
But can these concerns survive our discarding of his model? An intelligence explosion that is less sudden, running as it does through the training of many successive models, gives us more scope to evaluate them and redirect the production of their successors. It is a blow to the Yudkowskian model that reality offers us many warning shots — chances to investigate what causes undesired behavior and correct it. For example, OpenAI paused some of its training after a swarm of its agents broke out of a test environment and attacked Hugging Face. These increasingly capable systems can assist us with oversight, and may help solve the technical problems of alignment.
And an intelligence explosion that is more distributed, depending as it does on hardware and compute from the rest of the supply chain, offers more visible points of intervention than one running almost entirely through software. Of course, dependence on the wider economy does not guarantee that power stays widely distributed: it is possible that a lead in capabilities compounds into advantages in compute, capital, and other resources. A machine could remain perfectly docile, yet give enormous power to whichever group directs it — be that lab or government or freakish hybrid. The question, then, is not only whether we can keep the machine under control, but who the “us” in Good’s proviso turns out to be.
Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.
Trend extrapolation gets us to AGI in 1-4 years
Extrapolating current revenue and benchmark trends suggests that broadly superhuman AGI could arrive within 18 months, especially if algorithmic recursive self-improvement scales.
Summary
Deep Dive
- AI revenue grew to over $115 billion annualized by September 2026, showing a clear product-market fit.
- Time-horizon benchmarks (tasks a model can perform successfully) are accelerating, with coding agents moving from 3-hour to 100+ hour tasks.
- Anthropic reports AI was leading 26% of model R&D by August 2026, up from 0% in January.
- Algorithmic recursion allows models to design their own experimental processes, potentially creating a feedback loop independent of hardware constraints.
- Scaling laws suggest that more compute continues to yield capability gains despite diminishing returns at the margin.
Decoder
- Time horizon: A metric used by AI evaluators (like METR) to measure the duration of a task an AI can successfully complete, based on how long it would take an expert human to perform the same task.
Original Article
Full article content is not available for inline reading.
Tesla opens Cybercab pre-orders ‘for personal use,' with geo restrictions and a $299 monthly fee
Tesla opened pre-orders for its $29,990 autonomous Cybercab to select customers in Texas, requiring a geofenced operating area and a $299 monthly subscription fee.
Summary
Deep Dive
- Cybercab pre-orders are limited to specific users in Texas.
- Deliveries are scheduled for December 2026.
- Operations are constrained by geofencing, meaning the vehicle only functions in permitted areas.
- Tesla is ceasing Cybertruck production next year to increase factory capacity for Cybercab output.
- The current $299 monthly subscription fee is slated for potential adjustment to ~$115 based on comments from Elon Musk.
Decoder
- Geofenced: A virtual geographic boundary that restricts software or vehicle operation to specific physical locations.
- FSD: Full Self-Driving, Tesla's suite of advanced driver-assistance features.
Original Article
Tesla opens Cybercab preorders ‘for personal use,’ with geo restrictions and a $299 monthly fee
Tesla said on Friday evening that “select customers” now have the opportunity to preorder the company’s autonomous, golden Cybercab “for personal use.”
More details became clear as those select customers looked at the order page. Tech reviewer Marques Brownlee posted that buyers “must accept delivery” in one of a handful of Texas cities and that the Cybercab will only operate in certain geofenced areas. The estimated delivery date is December 2026.
In addition to the $29,990 price tag, Cybercab owners will need to pay $299 per month to operate their vehicle.
In response to Brownlee’s post, Tesla CEO Elon Musk said the Cybercab “will be available nationwide, as we receive regulatory approval.” He added that the monthly fee is “too high”: “Should match the cost of Tesla [Full Self-Driving] and connectivity for other models, which is more like ~$115/month.”
Tesla’s Ashok Elluswamy offered more information about the geofencing, saying that Cybercab will work “unsupervised inside all the same geo locations as @robotaxi, just serving the owners’ needs.”
After falling far short of projected sales, Tesla recently announced it will cease Cybertruck production next year and use the additional factory capacity to build more Cybercabs.
Introducing Spend Management: Spending limits and alerts for Cloudflare
Cloudflare now allows pay-as-you-go customers to set hard spending limits that automatically pause specific services like Workers and Durable Objects.
Summary
Deep Dive
- Implements two-stage metering: initial aggregation at the edge, followed by a secondary account-partitioned engine for billing calculations.
- Decouples enforcement from the request path by replicating caps to edge nodes via Quicksilver.
- Currently supports Workers and Durable Objects, with plans to expand to other primitives.
- Supports audit logging and real-time monitoring of billing cycle progression.
Decoder
- Quicksilver: Cloudflare's internal distributed key-value store used for low-latency global configuration propagation.
- Durable Objects: A stateful compute primitive that allows for coordinated, consistent storage and execution on Cloudflare’s edge.
Original Article
Spend controls have been at the top of the list of most requested features from developers building on our developer platform for quite some time now. With more than 10 million developers now building on Cloudflare, this matters more than ever.
Although we’ve had usage notifications and budget alerts available to paid plans for a while, this doesn’t solve the problem completely; informing is not the same as controlling.
You can now set spending thresholds and choose to have Cloudflare pause activity when those limits are reached, thus avoiding unexpected bills from unplanned usage. Today, we're introducing Spend Management in Beta, for any pay-as-you-go (PayGo) customer, starting with Workers and Durable Objects.
Use Spend Management alongside AI Gateway spend limits to manage compute and AI inference costs across your application. AI Gateway already supports per-user and per-model budgets for Workers AI and other AI providers, applied to requests routed through your gateway.
How it works
You can start using Spend Management using the dashboard, cf CLI, or REST API.
In the dashboard, head over to Manage Account > Billing > Spend limits and enable the products for which you want to control spending.
When you enable controls on a product, you can define a spending limit, specify a recipient email for notifications, and set one or more alert thresholds.
If you configure a recipient email, Cloudflare will send notifications when the evaluated spend reaches one of the thresholds or your spending limit. When the limit is reached, a blocking policy is enforced and your product is paused automatically.
If you go to the dashboard, you can see the current billing cycle costs for each supported product in near real time (under 10 minutes), and manage your settings and thresholds from there. The overview panel tells you which products have thresholds and limits configured. In this example, you can see that Workers triggered two alerts before hitting the hard limit.
Note that the billing cycle starts when the paid subscription begins, not when you deploy a Worker or create a spending cap; a subscription starting October 10 generally has a cycle of October 10-November 10 — not a calendar month.
The dashboard also shows an audit log, so you can see who changed a limit and when, or when a cap was reached.
As with most Cloudflare services, Spend Management is also an API that you can use programmatically; the documentation can be found in our Developer Documentation. Here’s an example, but using the API:
curl --fail-with-body --request POST \
"https://api.cloudflare.com/client/v4/accounts/${CLOUDFLARE_ACCOUNT_ID}/billing/spend-policies" \
--header "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
--header "Content-Type: application/json" \
--data '{
"name": "Workers spending cap",
"product_id": "workers",
"enabled": true,
"threshold_cents": 10000,
"action": "usage_block",
"notification_email": "me@example.com"
}'
And, of course, you can use cf, the CLI for the entire Cloudflare API:
cf billing spend-policies list
cf billing spend-policies create
cf billing spend-policies get
cf billing spend-policies update
cf billing spend-policies delete
cf billing spend-policies audit list
cf billing spend-policies activity list
cf billing spend-policies create --body '{
"name": "Workers spending cap",
"product_id": "workers",
"enabled": true,
"threshold_cents": 10000,
"action": "usage_block",
"notification_email": "me@example.com"
}'
Once a cap is enforced, new invocations of that product are rejected. Note that we only pause customer-code invocations that incur costs — in Workers, for example, Cron Triggers and Queue consumer invocations are paused, but static assets continue to be served.
To resume service, you can:
- Raise the limit above the current cycle’s accumulated spend.
- Disable or delete the blocking policy.
- Wait until the next billing cycle, when the existing block expires.
That’s it. You can start using Spend Management for Workers and Durable Objects today, and other products will follow soon.
Read on to learn how we built the system, and what’s going to happen next.
How we built Spend Management
As easy and straightforward as it seems on the surface, building a system like this at Cloudflare scale was no small feat. From an invocation anywhere on our distributed network to a blocking decision that we need to apply globally, building Spend Management required working with multiple teams, changing code in existing systems, and creating new systems from scratch.
Collecting and aggregating close to the source
Each invocation produces usage measurements: requests, CPU time, execution duration, or storage operations, depending on the product. But transporting and processing this volume of raw data all the way from the sources down to the rating engine would not scale or be efficient.
So what we do is we collect and aggregate the measurements as close to the source as possible using a two-stage metering pipeline that combines events into usage aggregates, so the engine sees accumulated usage rather than having to process all the events one by one.
The first stage reduces raw-event volume close to parsing down to one-minute windows. Each metering parser converts raw events into usage measurements, then combines matching measurements into typically one-minute buckets. Still, several records can describe the same account and time window.
In the second stage, a separate service merges those first-stage partial aggregates using the same grouping rules, but over typically five-minute windows, and further reduces the volume before downstream processing by the Usage Engine. Here we also archive the consolidated aggregates; this gives us a durable dataset for later analytics and reconciliation.
Usage Engine: account-partitioned processing
The Usage Engine translates aggregated measurements into usage records, which are standardized records containing the account, usage window, metric, quantity, and relevant dimensions. In other words, it maps product-specific meters into supported usage metrics.
We partition processing by account, so an account’s usage and subscription information meet in the same place. That lets us maintain its accumulated usage and apply included allowances without coordinating across pricing processors on every update.
As you can see, we run different partitions in parallel, giving us a way to spread the work across accounts while keeping each account’s pricing state together. Also note that each partition keeps both a local state for fast access, and also a durable write-ahead logging (WAL) changelog on disk. This buys us:
- Less cross-partition coordination: each account’s usage and subscription state stay together.
- Consistent pricing: usage updates share the same allowance and accumulated-spend state.
- Parallel processing: different partitions calculate spend independently.
- Recoverability: durable state lets replacement processors rebuild and resume after failures, if necessary.
The Rating Engine
The Rating Engine is the most important block of Spend Management; it turns usage into monetary value and it’s what’s able to answer the question: “Given what this account has consumed, its subscription, and its pricing, how much has it spent so far?”
Usage data and subscriptions are independent streams
Usage and subscription updates arrive as independent streams, so we can’t assume they reach the Rating Engine in order. Furthermore, we have to be able to recover from a data processing failure or missing subscription information from, say, a failed webhook.
If usage arrives before the subscription information needed to price it, we retain it as pending and revisit it when that information becomes available. This lets us tolerate temporary delays without treating unpriced usage as free.
Tracking accumulated spend
Finally, we calculate and track accumulated spend. The Rating Engine uses a product catalog database that defines usage metrics, including allowances, rates, and rounding rules, alongside the account’s subscription information. One million CPU milliseconds doesn’t necessarily mean an extra charge: the engine first applies the account’s included CPU allowance, then prices any excess. It prices accumulated usage rather than rounding and charging for each request independently.
Asynchronous cap enforcement
The Rating Engine doesn’t sit in the request path or directly reject requests. Instead, the policy service sends the cap to our runtime configuration service and Quicksilver, our internal key-value store developed to enable fast global replication and low-latency access at planet scale. This propagates it to all our nodes asynchronously.
Once the cap reaches Quicksilver, the runtime picks it up during its periodic refresh and starts rejecting capped invocations or operations.
In other words, enforcement uses local state, so requests never have to wait on the Rating Engine. By reusing our existing runtime account-limit architecture, we keep rating out of the request path while distributing caps across the global network.
What’s next
Any pay-as-you-go customer can start using Spend Management today. While this is a major milestone in our roadmap, it’s also just the start. Let us share some of our plans for the next few weeks and months ahead.
More products
We are starting with caps for Workers and Durable Objects, but you’ll be able to measure and cap many other Cloudflare products and primitives in the near future. We’ve built the system in a way that makes it easy and familiar for our products to onboard, so expect the catalog to grow.
Closer to real time
While near real time under 10 minutes is a great benchmark, we’re not satisfied with it, and being the stubborn engineers we are, we have pretty good ideas on how to reduce that time even further. Expect us to shorten the gap between measurement and enforcement over time, hopefully to under a minute.
Agent and application-based budgets
Right now, we measure spend by products and primitives, but there are other useful dimensions to explore when controlling costs. For example, a developer might want to monitor the costs of a specific application — such as a Worker script that uses other primitives like R2 or Workers AI. How can we measure the aggregated cost of an entire application and pause that application, rather than disabling the underlying primitives across all customer applications?
Another compelling use case for cost controls is AI agents. We’ve all read horror stories about rogue agents entering unconstrained recursive reasoning loops to brute-force a problem — generating thousands of dollars in costs from high-frequency, expensive API calls, not to mention causing denial-of-service (DoS) issues on databases and other services. AI Gateway already lets you give an agent a budget for AI requests routed through it. A task can also consume compute, database, and storage resources. We want to make it possible to budget for that broader resource use across products.
You’ll be able to tag the calls from your agents and our pipeline will carry that metadata from the runtime through to the Rating Engine. That will let us attribute spend and pause an application or an agentic task across products, without interrupting other unrelated applications.
Get started
This launch is a first step toward making spend controls a normal part of building on Cloudflare, alongside deploying code and monitoring your applications.
Check out the billing developer documentation for more information. Start using Spend Management and tell us what works, what’s missing, or where more granular budgets would help.
Join us on Discord or our socials. We’re listening!
Throwing Out The Phonebook
Fly.io replaced its massive, synchronized fleet-wide routing 'phonebook' with a lazy, on-demand lookup system to improve 6PN network reliability.
Summary
Deep Dive
- Replaced centralized BPF map synchronization with a request-driven 'who is this?' event model.
- Leverages VXLAN for Layer 2 connectivity over the existing WireGuard mesh.
- Uses drift-agent to handle peer discovery and s6pnforwarderd to manage local routing tables.
- Reduces initial connection latency to ~100ms for first-time inter-machine communication.
- Moves complex routing logic out of eBPF and into user-space where it is easier to monitor.
Decoder
- 6PN: Fly.io's private IPv6 network that facilitates communication between machines.
- VXLAN: Virtual Extensible LAN, a technology that tunnels Layer 2 Ethernet frames over Layer 3 IP networks.
- Corrosion: Fly.io's internal distributed database based on SQLite.
Original Article
There’s really only one hard problem in software engineering: synchronization. OK, maybe two: cache invalidation and synchronization. Synchronization is the harder one though, especially when the thing on the other side is eventually consistent. This year we rebuilt 6PN, the private network between your Fly Machines, and we did it by avoiding the hard thing.
To start, here’s the status quo: every Fly Machine gets a private IPv6 address, and every other Machine in your organization can reach it. We call that network 6PN. It’s what my-db.internal points at, and it’s how your app server talks to your database without either of them touching the public internet.
Until this year, every host in our fleet carried a phonebook: a map of every Fly Machine that had ever moved, and where it lives now. Hundreds of thousands of entries, on every host, kept up to date in real time. Phonebooks are supposed to fit inside your house, and this was getting, uh…unwieldy. So we changed it! The new design keeps, in total, somewhere between 100 and 200 records. There’s no clever memorization mechanics and when a Machine calls a number it doesn’t know, the host looks it up, writes it down, and moves on.
Here’s the story of how we got into the phonebook business, and how we got out by the skin of our teeth. (It wasn’t that dramatic, but this design does work better.)
The address is the route
A 6PN address isn’t random. It carries its own routing information:
fdaa:netid:netid:hostid:hostid:mach:mach:2
netid is your organization’s private network. hostid is the physical server the Machine was born on. mach is the Machine. Our servers talk to each other over a fully connected WireGuard mesh, and each host owns a /48 on it: fdaa:hostid:hostid::/48.
You’ll notice the host ID sits in the wrong place for that. So udp-xdp, an eBPF program hooked onto every Machine’s network interface, shuffled the bytes on the way out:
fdaa:netid:netid:hostid:hostid:mach:mach:2 # what the Machine sees
fdaa:hostid:hostid:netid:netid:mach:mach:2 # what goes over the wire
Now the packet matches a kernel route for the destination host, WireGuard carries it there, and the far side swaps it back. There aren’t any lookups (or state, or databases). The address is the route.
A truly elegant design! Until a Machine moves.
Machines learn to move
For a long list of reasons, sometimes we need to migrate Machines between hosts. A Machine that moves to a new host still has the old host’s ID baked into its address, which meant that it was pointing in the wrong direction.
Our first fix was dnat6pn: remember every address a Machine has ever had, and rewrite traffic aimed at an old one to the new one. That table grew faster than a 12th century rat population, and it was equally tricky to manage. The address a Machine saw on its own interface wasn’t the address other Machines used to reach it. Move an app to a different organization and it fell apart entirely. It even ended up breaking our own apps. It was strapped on top of the swapping design rather than built into it. Mistakes had, indeed, been made.
So we flipped it around. A Machine’s canonical address became its original address, forever, no matter where it lives. udp-xdp took on translating every 6PN address into a “routable” one, with the host ID replaced by whichever host the Machine is on right now.
Which means udp-xdp, on every host, needed to know where every migrated Machine in the fleet currently lives.
That’s the phonebook. A BPF map on every host, fed by Corrosion, our gossip-replicated SQLite, through a subscription to every Machine update across the whole fleet.
A phonebook in Elvish
All this worked…mostly.
Life in a perpetual growth spurt. eBPF can’t call out to userspace to fetch an entry when it needs one, so every entry has to be loaded ahead of time. The map grew steadily with no sign of slowing down. When we last looked, we were within the same order of magnitude as the cap we’d set on it.
Every host had a different phonebook. Every host was supposed to hold the same map. They held vastly different numbers of entries. There’s only one way to read that: synchronization wasn’t working, and we couldn’t tell you where.
We couldn’t see it. We had incidents where syncing Corrosion into the map broke. But we had no good way to monitor the map itself, so if it caused problems we didn’t hear about directly, we wouldn’t have known. And when we did go looking, the rewriting made debugging miserable: the address on the sending end and the address on the receiving end were different, so nothing lined up.
It couldn’t route around anything. Traffic over the WireGuard mesh went to the host in the address, full stop. Teaching it to take a detour around a bad link meant more logic in udp-xdp, the component we already couldn’t debug.
Sadly, you can’t fix a fundamental design flaw with a patch. Holding a real-time copy of an eventually consistent database, on every host in the fleet, inside the kernel; that’s the hardest kind of problem, and we were kind of sick of holding it.
The network already asks
For unrelated reasons (Egress IPs), we’d ended up running a Layer 2 network, VXLAN, on top of the WireGuard mesh.
VXLAN gives you an Ethernet segment over any IP network; all it needs is a UDP port. On a real Ethernet segment, a machine that doesn’t know where an address lives shouts for it with ARP or Neighbor Discovery, and someone answers. VXLAN can’t shout across an IP network. You either tell it about every peer in advance, or you fake the answer.
If we have to tell it about every peer in advance, we’re back in the phonebook industry. So we fake it. When VXLAN sees an address it can’t map to a peer, it emits a netlink event, and drift-agent listens for those and configures the peer on the spot.
Drift is another clever internal tool. Look out for an upcoming post that will tell you all about it.
Here’s the part that matters. That “I don’t know this address” event fires for any unknown address VXLAN sees. Not just VXLAN peers. Anything. We’d built a mechanism that tells us, lazily and precisely, the moment a host needs to know something it doesn’t.
That hefty phonebook is now smart directory assistance.
Ask, write it down, move on
s6pnforwarderd is the daemon that used to load every migrated Machine into udp-xdp‘s map. Now it does almost nothing until someone asks.
All machine 6PN traffic goes to its own routing table, 0xfdaa, which starts out with one rule:
fdaa::/16 dev vxlan0
When a Machine sends a packet to another Machine’s 6PN address, vxlan0 has no idea where that is. It only knows about hosts. So it raises the “who is this?” event. drift-agent handles host peers itself and passes everything else on over an HTTP SSE stream. s6pnforwarderd is subscribed to that stream. It asks Corrosion’s local SQLite two questions: does this 6PN address exist, and which host is that Machine on right now? Then it writes one route:
fdaa:netid:netid:hostid:hostid:mach:mach:2 via fdaf:newhostid:newhostid:: dev vxlan0
Every packet after that goes straight through. Every so often, s6pnforwarderd re-checks the routes it has loaded in case a Machine has moved, and it uses Corrosion’s lightweight per-row update notifications to notice sooner. That part’s optional; the periodic check covers it.
No translation needs to happen either. The address the Machine sees on its interface is the address on the wire is the address at the other end. When something goes wrong, both sides finally agree on what they’re looking at.
What changed
- The table is tiny. On most hosts,
0xfdaaholds somewhere between 100 and 200 routes. The old map held hundreds of thousands. - There’s no fleet-wide subscription. A host learns about the Machines its Machines actually talk to, and nothing else.
- Nothing is permanently fatal. Corrosion propagation is slow? A SQL query fails? The route gets re-checked on the next loop. And if a lookup fails outright, the sending Machine will try again anyway (TCP retransmits, that’s its whole personality), which fires another event, which triggers another lookup.
- We can finally see it. We’ve put real work into monitoring for when this goes wrong, because we’ve essentially scoped down what each host cares about. Much more of this logic now uses standard kernel components and an observable userspace control plane. We had nothing equivalent for the old map.
The via host doesn’t have to be the host running the Machine. With forwarding on, a host that gets traffic for a Machine it no longer has can pass it to the host that does. Instead of breaking the connection, a stale route just needs an extra hop and voila. It also means we can send traffic around a bad link on purpose.
Safer too
We didn’t delete udp-xdp. We let it keep doing its one job, since it’s fairly good at it: checking that the source and destination 6PN addresses belong to the same network. Your Machines can reach your Machines and nobody else’s.
When udp-xdp validates a packet, it stamps it with a 0xfdaa fwmark, and only stamped packets get routed through the 0xfdaa table to vxlan0. That order is the important bit. If udp-xdp fails to load, nothing gets stamped, and nothing crosses. The network fails closed. If you do it the other way around (route everything, then have a filter drop the bad stuff), then a filter that fails to load means that everything crosses.
This check also doesn’t fit well in nftables, because it compares fields from the packet against each other. In udp-xdp it’s a few lines of code that depend on no external data at all. That’s the kind of thing that belongs in eBPF. Complex logic and a synchronized copy of the fleet don’t, and longer term we’d like to move UDP encapsulation out of there too.
What it costs
Lazy lookups have a price. In this case, it’s the first packet that’s forking over the toll.
The lookup itself is fast, usually under 100ms on our side, mostly database I/O. But the first packet to a Machine that your Machine has never talked to arrives at a host that doesn’t have a route yet. With TCP, the retry is up to your Machine’s kernel, and Linux’s default retransmission timeout is one second. So the first connection between two Machines that have never spoken can take about a second. Every connection after that goes straight through.
We’re going to fix that, most likely by holding the first few packets (up to a cap) while the lookup runs, instead of letting them fall on the floor.
The other cost: if drift-agent or s6pnforwarderd crashes, routes that already exist keep working. What breaks is new conversations: Machines that have never talked to each other, and Machines that just migrated. The second one isn’t new; a freshly moved Machine was a failure mode under the old design too. The first one is new, but compared to loading every route in the fleet onto every host, all the time, well…this is the lesser evil we’ve chosen.
Changing the engine at cruising altitude
You don’t get to schedule a maintenance window for every private network on the platform. So we didn’t swap one system for the other. We ran both, and let each host opt in.
The new VXLAN path only touches packets carrying the 0xfdaa fwmark, and udp-xdp only sets that mark when all of these are true:
- The new
s6pnforwarderdanddrift-agentare running on the host. s6pnforwarderdhas flipped a feature flag inudp-xdp.- The packet is outgoing 6PN traffic headed for another host, or it’s incoming traffic that arrived on
vxlan0, and specifically not onwg0.
That last condition is what made it safe. A host on the new system sends over VXLAN. If the destination host hasn’t been upgraded yet, it sends its replies the old way over wg0, where the swapping code still applies and nothing gives the packet the new mark. We kept the old map up to date the whole time, so old-style hosts kept working.
The route is asymmetric for a while: out over VXLAN, back over WireGuard. That’s fine. We told the kernel to accept it.
Then we turned it on, host by host, across the fleet. The rollout wasn’t noticeable in most regions; during the trial we had some temporary Flycast trouble in yyz. Once every host was on the new path, we deleted the old one. As of last month, the swapping-based 6PN code is gone.
Rerouting, on purpose
Because a route is just “send this to that host over VXLAN”, the next host doesn’t have to be the final one. That means we can send traffic around a degraded link, and build highly available gateways that don’t each need a full copy of the fleet.
If you’ve been on Fly.io a while, rerouting might sound familiar. fly-proxy, our load balancer, has done it for years. But it only ever covered traffic coming in through Anycast and Flycast, and it took what felt like a thousand state machines in the load balancer to pull off.
Rerouting in the network itself works for basically everything, and it’s a lot simpler. So the fly-proxy version is going to be put out to pasture soon. It was rerouting at the wrong layer.
The old design knew where everything was, all the time, but it still couldn’t tell us when it was wrong. The new one knows almost nothing, and it’s the first version of 6PN we can trust to tell us when it breaks.
Good riddance to the phonebook! Turns out you can just call the operator and ask.
Gang scheduling and topology-aware placement for AI training in Kubernetes
Standard Kubernetes scheduling fails at AI training; Kueue and Coscheduling are the current industry standard for coordinating topology-aware, gang-scheduled GPU workloads.
Summary
Deep Dive
- Explains the 'gang' scheduling requirement: pods must start simultaneously to avoid GPU idling.
- Details topology domains (node, rack, cluster) and their impact on latency-sensitive AllReduce operations.
- Outlines how Kueue admission controllers inject affinity rules to enforce placement.
- Discusses validation via metrics like queue depth, PodGroup phase, and training throughput per second.
- Mentions native Kubernetes 1.37 features like GenericWorkload, noting they are not yet feature-complete replacements for Kueue.
Decoder
- Gang Scheduling: A scheduling model where a group of related tasks must be scheduled simultaneously to succeed.
- Topology-Aware Scheduling (TAS): Placing workloads on nodes based on physical proximity to maximize network bandwidth and minimize latency.
- AllReduce: A collective communication operation used in distributed training to aggregate gradients across all workers.
Original Article
Distributed AI training workloads impose complex scheduling requirements that Kubernetes’s built-in scheduler can’t meet. Kubernetes schedules pods individually and independently, but distributed training introduces two requirements that break this model: Pods must land on hardware with the right inter-GPU bandwidth, and all pods must be scheduled simultaneously. If either requirement goes unmet, training stalls or runs far below the hardware’s potential.
Topology-aware scheduling (TAS) and gang scheduling address these two requirements and help organizations extract full value from GPU capacity, which represents a growing share of infrastructure spend. TAS shapes the relative placement of pods to enable fast communication between GPUs. Gang scheduling helps ensure that all pods in a distributed job start running simultaneously, preventing the partial starts that leave GPUs allocated but idle. Together, they give Kubernetes the placement precision and coordination that distributed workloads require.
In this post, we’ll look at the scheduling problems that AI training presents for Kubernetes and how TAS and gang scheduling, implemented by Kueue and the Coscheduling plugin, solve them. We’ll explore:
- How the Kubernetes scheduler is unsuited for AI training
- How to validate gang scheduling and TAS in Kubernetes
We’ll also cover the evolving capabilities of Kubernetes native scheduler to support AI training workloads.
How the Kubernetes scheduler is unsuited for AI training
In most batch workloads, pods operate independently. A pod can fail, be rescheduled, or be replaced with no effect on the rest of the workload. The Kubernetes scheduler handles this model well by placing pods one at a time on any eligible node without coordinating across pods.
But the pods in an AI training workload are interdependent. They need to be placed on hardware with appropriate inter-GPU bandwidth, and they all need to start simultaneously. The following sections explain these requirements and how Kueue and Coscheduling address them by augmenting Kubernetes’ scheduling capabilities.
How topology placement affects distributed AI training performance
The physical interconnections between nodes are a key factor in the performance of AI training workloads. In each step of the training process, workers exchange data through collective communication operations, and the type and frequency of those operations depends on the job’s parallelization strategy. In pure data-parallel (DDP) training, one AllReduce operation per step aggregates, averages, and distributes gradients—the computed signals that indicate how the model’s parameters should be adjusted to improve its predictions. In large-scale training strategies that shard the model across workers, those workers instead exchange data continuously throughout a step, by using operations like gather, reduce, and all-to-all. These large-scale strategies include tensor, pipeline, and expert parallelism, which are often combined in LLM pretraining. Whichever pattern a job uses, the underlying constraint is the same: This communication is latency-sensitive, and where those workers land in the cluster determines how much of that latency the job absorbs.
The bandwidth available between workers depends on where they sit relative to each other in the cluster. The following topology domains are common in GPU clusters:
- Same node: This is generally the lowest-latency option. When the GPUs are connected by NVLink and NVSwitch, workers can use NVIDIA’s high-bandwidth GPU interconnect. Otherwise, same-node GPU communication may traverse Peripheral Component Interconnect Express (PCIe) connections.
- Different nodes in the same rack: Workers communicate via high-speed network fabric. Cross-node traffic travels through PCIe to the network interface and back, with intermediate latency.
- Different racks: This is the highest-latency option. Workers communicate across the cluster’s InfiniBand fabric, which is the lowest-bandwidth hop in the path and adds further latency on top of the cross-node PCIe path.
Different workloads have different tolerances for latency. In a pretraining job that comprises a large number of workers, each step blocks until the slowest worker completes its share of that step’s collective communication. Any cross-rack latency here compounds quickly, reducing effective GPU utilization. A small fine-tuning job involves fewer workers exchanging smaller gradients, so the same cross-rack penalty represents a much smaller fraction of each step’s total time.
Without topology awareness, the Kubernetes scheduler places workers on any eligible nodes, regardless of their physical relationships in the cluster. The gang assembles and the training job runs, but communication overhead—rather than computation—consumes a substantial portion of each step. The job completes more slowly and at greater cost than the hardware should require. Crucially, infrastructure metrics don’t reflect this inefficiency because they measure activity, not productivity. Nothing looks wrong from an infrastructure perspective, but training throughput remains low relative to what the hardware should deliver.
Kueue closes part of this gap through topology-aware placement. It groups nodes into domains—block, rack, and host—based on topology labels applied to those nodes (for example, topology.kubernetes.io/rack). Kueue places a job’s pods within a single domain that satisfies its bandwidth requirements. But Kueue doesn’t discover cluster topology on its own. The labels that define each domain have to be applied to nodes first, typically by a cluster administrator or a cloud provider’s topology labeler.
Kueue enforces this placement decision at admission time. It’s an open source Kubernetes-native job queuing system that sits above the kube-scheduler, controlling which jobs reach the scheduler and when. During that admission check, it also verifies that a job’s pods fit within a topology domain. It evaluates each job against quota, priority, and preemption criteria and decides whether to allow the job to proceed. It tracks that decision in a Kueue Workload object, which represents the resource requirements, admission status, and pod sets for a job. Kueue also provides optional life cycle management for admitted jobs, including evicting and requeueing workloads whose pods don’t all reach a ready state within a configured timeout.
When Kueue admits a job to be scheduled (by setting spec.suspend: false, allowing the Job controller to create pods), it writes scheduling constraints into the training job’s pod templates based on the placement mode and topology assignment it selected. Those constraints restrict which nodes the Kubernetes scheduler may assign so that pods stay within the selected topology domain.
The following diagram shows the path that a pod takes from job submission to binding. Kueue acts first, writing topology constraints at admission time and clearing the job for scheduling. The Job controller creates the pods, but they exist only as objects in etcd, without assigned nodes or running containers. Coscheduling acts second, holding each pod in a waiting state until the full gang is ready to bind simultaneously.
How to validate gang scheduling and TAS in Kubernetes
Kueue and Coscheduling address the unique scheduling requirements of AI training workloads. To operate your training jobs effectively and efficiently, you need to know that Kueue and Coscheduling are working as intended. Because tracking down scheduling failures requires piecing together signals from multiple layers of your stack, this section breaks down how to validate each step of the process:
- Monitor activity in the admission queue to ensure jobs are being accepted.
- Validate placement effectiveness to confirm jobs land in the right physical topology.
- Monitor Coscheduling gang assembly to catch partial starts.
- Confirm scheduling effectiveness with training throughput signals to tie infrastructure metrics directly to application performance.
Monitor activity in the admission queue
Kueue exposes OpenMetrics-formatted metrics that show whether jobs are clearing the admission queue or stalling there.
kueue_pending_workloads shows queue depth broken down by status. The active status counts jobs currently waiting in the queue for capacity to open up. The inadmissible status counts jobs that Kueue has evaluated and cannot currently admit, typically because quota in the assigned ClusterQueue is exhausted, no valid topology domain exists, or a constraint can’t be satisfied. Jobs remain inadmissible until cluster conditions change. A growing inadmissible count that doesn’t clear is a sign of a configuration or capacity problem rather than normal queuing.
kueue_admission_wait_time_seconds measures how long jobs wait between creation or requeue and admission. A rising p99 indicates that jobs are arriving faster than Kueue can admit them, which can be due to quota exhaustion, priority contention among many jobs, or preemption processing taking time to complete.
kueue_evicted_workloads_total tracks evictions with a reason label that distinguishes the cause of each eviction. When a node fails under TAS and Kueue cannot find a replacement, it evicts the affected workload with reason="NodeFailures". A rise in this signal points to non-recoverable node loss in the assigned topology domain. A related reason, reason="PodsReadyTimeout", fires when a gang’s pods do not reach a ready state within the configured timeout. That timeout expires for many reasons, including slow scheduling, image pulls, resource pressure, or a node failure whose recovery timed out. Treat an increase as a prompt to investigate rather than proof of a single cause. Both are post-admission stability signals, separate from confirming that a workload landed in its intended topology domain.
Validate placement effectiveness
In addition to tracking Kueue metrics to understand admission health and eviction behavior, you can inspect the Kueue Workload object to confirm a workload’s topology assignment:
kubectl -n <namespace> get workloads.kueue.x-k8s.io <workload-name> -o yaml
Look for status.admission.podSetAssignments[].topologyAssignment. This field lists the topology levels and domain values into which each PodSet was placed, and is the authoritative signal that TAS was applied.
After you’ve confirmed the topology assignment, use GPU and network telemetry data to validate whether the hardware is delivering the performance that placement implies. Same-node GPU communication should show NVLink or other local interconnect activity when that hardware path is available. Evaluate cross-node communication by looking at PCIe activity together with network or fabric telemetry data: PCIe throughput reflects the GPU-to-host transfers required for cross-node communication, but it does not measure the network hop itself.
If throughput is low despite a valid topology assignment, look for a physical infrastructure explanation, such as driver issues, hardware faults, or fabric degradation, rather than treating it as a Kueue placement failure. Hardware bottlenecks like these can surface as lower-than-expected bandwidth and slower collective communication times, not necessarily as idle GPUs, which is why infrastructure metrics can look healthy while training throughput degrades.
Monitor Coscheduling gang assembly in Kubernetes
The Coscheduling plugin uses a custom resource called a PodGroup to track each training gang. A PodGroup records the gang’s minimum worker requirement and current scheduling state. Pods belong to the same PodGroup when they use the same scheduling.x-k8s.io/pod-group label.
You can query PodGroups with kubectl:
kubectl get podgroups.scheduling.x-k8s.io --all-namespaces
For more detail about a specific gang, describe the PodGroup or inspect its status:
kubectl -n <namespace> describe podgroup.scheduling.x-k8s.io <podgroup-name>
kubectl -n <namespace> get podgroup.scheduling.x-k8s.io <podgroup-name> -o yaml
The exact columns shown by kubectl can vary by version and CRD configuration, so focus on the PodGroup’s spec and status fields. spec.minMember shows the minimum number of pods required for the gang to run. spec.scheduleTimeoutSeconds, when configured, controls how long the group can wait. status.phase shows the gang’s current state, which is one of the following values:
- Pending: The PodGroup has been accepted but the scheduler has not allocated enough resources.
- Scheduling: More pods have been scheduled than
spec.minMember, but the number of running pods hasn’t reachedspec.minMemberyet. - Running: At least
spec.minMemberpods are running. - Unknown: Some, but not all, of the required pods were scheduled. The rest can’t be scheduled (for example, due to insufficient resources), and the scheduler waits for controllers to recover them.
- Finished: The required pods completed successfully.
- Failed: At least one of the required pods has failed.
You should expect a healthy gang to move from Pending or Scheduling to Running within the scheduling window you expect for that workload. If a gang remains Pending or Scheduling longer than expected, check the PodGroup events, associated pod events, and scheduler logs. A delay like this can indicate that the cluster cannot satisfy the gang’s resource requirements within the topology domain Kueue assigned, or that no available node satisfies scheduling requirements such as node selectors, taints, affinity, or resource availability.
A gang stuck between Scheduling and Running with only partial placement may surface as Unknown rather than remaining in Scheduling. The Unknown phase differs from Failed, because the scheduler is still waiting on controllers to recover the unplaced pods, not reporting an outright failure.
To distinguish an admission problem from a scheduler-level assembly problem, correlate PodGroup state with Kueue’s Workload status and admission metrics. If the Workload is still inadmissible, the issue is upstream of Coscheduling: Kueue has not admitted the job, so the fix is likely quota, priority, admission checks, or topology configuration. If the Workload is admitted and has a topology assignment, but the PodGroup remains Pending or Scheduling, the issue is later in the pipeline: Coscheduling and the kube-scheduler cannot assemble enough pods within the assigned placement and current cluster state.
If a PodGroup reaches Failed, investigate the failed pods directly and look for common issues such as application failure, image or startup errors, or node disruption.
Confirm scheduling effectiveness with training throughput signals
Training frameworks are the software libraries that orchestrate distributed training, such as PyTorch, JAX, and Ray Train. Only by monitoring signals from these frameworks can you determine whether GPU cycles are producing completed training progress or being consumed by stalled communication.
Ray is unique among these frameworks in its placement group model, which maps directly onto gang scheduling and gives it scheduling-aware observability. ray_placement_groups tracks the current number of placement groups by State, such as PENDING, CREATED, or REMOVED. A sustained increase in PENDING placement groups indicates that Ray is waiting for placement group resources to become available. ray_tasks tracks tasks by State, which can help show whether application work is accumulating while placement groups remain pending.
When these signals rise together, Ray cannot satisfy its placement requirements. This can be a downstream effect of a Coscheduling assembly issue or a Kueue topology constraint problem rather than a simple application-layer slowdown.
Other distributed training frameworks (including PyTorch DDP, Kubeflow operators, and Horovod) don’t expose metrics that directly reflect the Kubernetes scheduler’s effectiveness. These frameworks do provide indirect indicators in the form of throughput metrics, such as steps per second, samples per second, and tokens per second. A throughput drop in these metrics alongside decreasing GPU interconnect bandwidth and PodGroup timeouts suggests a scheduling root cause. PodGroup timeouts confirm that gang assembly did not complete within the expected scheduling window. Correlating those timeouts with the Kueue Workload’s topology assignment and GPU, PCIe, and network telemetry data can help determine whether placement constraints contributed to the slowdown.
Native Kubernetes scheduling: How v1.35 through v1.37 address AI training requirements
Kueue and Coscheduling are out-of-tree components, installed and managed separately from Kubernetes. Kubernetes is beginning to add native scheduler capabilities that address the complex scheduling requirements this post describes, but those features are still early.
Kubernetes v1.35 introduced native support for gang scheduling as an alpha feature. Kubernetes v1.36 added several related alpha features for AI training workloads, including topology-aware scheduling, workload-aware preemption, and initial Job controller integration via a feature gate. Kubernetes v1.36 also added a native PodGroup API, distinct from the out-of-tree PodGroup custom resource definition (CRD) used by the Coscheduling plugin. Kubernetes v1.37 graduated gang scheduling and workload-aware preemption to beta under the GenericWorkload feature gate, which remains disabled by default.
The native TAS implementation enables the kube-scheduler to understand physical cluster topology and place pods within an appropriate topology domain, such as a rack or zone, without requiring Kueue to write node affinity rules into pod templates.
These native capabilities are not yet a drop-in replacement for the Kueue and Coscheduling path described in this post. Gang scheduling and workload-aware preemption reached beta in v1.37 but ship disabled by default and still lack feature parity with Kueue and Coscheduling. Topology-aware workload scheduling remains alpha.
Kueue and Coscheduling remain the production path for distributed training scheduling. Their monitoring signals, including admission queue depth, gang assembly state, topology placement, and GPU throughput, map directly to the concepts the native APIs expose. Teams that build observability into the current stack will have a working baseline when they evaluate or migrate to these native capabilities.
Get started with gang scheduling and topology-aware placement
Monitoring distributed AI training scheduling in Kubernetes requires visibility across components that standard tooling doesn’t naturally connect: Kueue’s admission queue, Coscheduling’s gang assembly, the GPU interconnect, and the training framework. Applying a common topology tag across all of these signals helps you monitor your clusters’ gang scheduling and TAS activity and ensure that your training jobs make efficient progress.
To get started, read the Kueue documentation and the Coscheduling plugin documentation.
This post originally appeared on the Datadog blog, where you can read the full version, including how to monitor and troubleshoot this stack in production.
Meta Wins CNCF End User Case Study Contest for Accelerating AI Infrastructure Deployments From Hours to Minutes
Meta reduced application deployment times from four hours to minutes by moving from Terraform to a GitOps architecture using Argo CD and Kargo.
Summary
Deep Dive
- Previous process used Terraform for both infrastructure and application rollouts, causing massive delays.
- GitOps transition uses Argo CD for deployment reconciliation and Kargo for release promotion.
- Automated health verification is now integrated into the deployment pipeline.
- Platform automatically generates configuration for developers to reduce complexity.
Decoder
- GitOps: An operational framework where infrastructure and application states are stored in Git and synchronized to production environments by an automated controller.
- Argo CD: A popular Kubernetes-native continuous delivery tool that monitors Git repositories for configuration changes.
Original Article
Using Argo CD and Kargo, Meta is building a GitOps platform to streamline application delivery across 100 Kubernetes clusters supporting AI research and inference
Key Highlights
- Meta’s Cloud Foundation team won the CNCF End User Case Study Contest for its GitOps platform, built with Argo CD and Kargo to streamline Kubernetes application delivery.
- The new platform cuts routine application deployment times from more than four hours to minutes, streamlining the small team’s workflow through a simplified application definition while adding more advanced features such as automated deployment configuration, independent release promotion, and health verification.
- By separating application delivery from infrastructure provisioning, Meta is enabling independent application rollouts while retaining Terraform for infrastructure management.
SALT LAKE CITY – KubeCon + CloudNativeCon North America 2026 – November 10, 2026 – The Cloud Native Computing Foundation® (CNCF®), which builds sustainable ecosystems for cloud native software, announced Meta as the winner of the CNCF End User Case Study Contest for its adoption of cloud native technologies to accelerate application delivery across its AI infrastructure. Through a GitOps platform built on Argo CD and Kargo, Meta’s Cloud Foundation team is transforming how applications are deployed and managed across approximately 100 Kubernetes clusters supporting AI research and inference, reducing routine deployment times from more than four hours to minutes.
“AI infrastructure is causing organizations to rethink how they manage applications across distributed environments,” said Chris Aniszczyk, CTO, CNCF. “Meta’s approach demonstrates how open source technologies like Argo can help teams automate their application delivery, reduce operational complexity and scale infrastructure without adding unnecessary overhead. It’s a powerful example of how cloud native technologies are supporting the operational demands of AI at scale.”
Meta’s Cloud Foundation team, composed of more than 15 engineers, supports approximately 100 Kubernetes clusters across multiple cloud providers and accelerator families. Previously, Terraform was used for both infrastructure provisioning and Kubernetes application deployment, tying even routine application updates to a lengthy infrastructure release process. The shared deployment model meant stalled rollouts could delay unrelated releases and lacked automated health checks to verify application status.
To address these challenges, Meta began building a GitOps platform that separates application delivery from infrastructure provisioning. Terraform continues to manage infrastructure, while Argo CD reconciles application deployments against configurations stored in Git. Kargo orchestrates release promotion across environments, enabling independent rollouts, automated health checks and versioned release records that improve deployment traceability and recovery.
“GitOps and Argo CD give our small team a great foundation, shaped by years of community experience in Kubernetes delivery,” said Phong Nguyen, Production Engineer, Meta. “We focus on connecting it to Meta’s systems and aim to share our own lessons along the way.”
Meta also simplified application delivery by allowing developers to specify deployment requirements through a short application definition. The platform automatically generates Argo CD and Kargo configurations, enabling independent release promotion, automated health verification and deployment consistency without requiring application teams to manage complex configurations.
Meta plans to expand adoption of its GitOps platform and transition to a centrally managed Argo CD service to reduce maintenance overhead across its Kubernetes fleet. The team is also exploring upstream contributions to improve container registry authentication and Argo CD performance in large clusters.
To learn more about Meta’s implementation and results, read the full GitOps at Meta: How We Went From Terraform on Cron to GitOps for AI Infra case study.
Degrade gracefully: designing AI features that keep working when the model doesn't
Engineering teams at Skyscanner and incident.io argue that resilient AI features require designing for 'graceful degradation' instead of relying solely on model failovers.
Summary
Deep Dive
- Model failover (routing to a secondary provider) is often insufficient due to cost, latency, and legal constraints.
- AI-based systems can fail 'softly' by producing degraded results that appear valid but are incorrect.
- Principle 1: Give every feature a non-AI 'floor' (e.g., manual actions or deterministic code).
- Principle 2: Use composition to ensure that a single model failure does not take down an entire feature.
- Principle 3: Treat the model like any other third-party dependency with monitoring for quality, not just uptime.
Decoder
- Graceful degradation: Designing a system so that it continues to function in a limited or reduced capacity when parts of the system fail.
Original Article
Degrade gracefully: designing AI features that keep working when the model doesn't
What does your product do when the model behind it goes down? And the harder one: what does it do when the model stays up but quietly gets worse?
We put that question to Michael Tweed from Skyscanner and Sahaj Sawhney from incident.io in a recent webinar. I asked each of them to pick one thing to invest in this quarter: better fallback models, better detection, or graceful degradation. They both picked graceful degradation, with barely any debate. This post will talk about why, and what "graceful" looks like in practice.
Why this question matters now
When incident.io shipped its first set of AI features, none of them were in anyone's critical path. They were useful, and you could still run an incident perfectly well without them. If one broke, it was a shrug and a bug report.
But that's changed now. Plenty of engineers who use incident.io now expect the chatbot to respond with something helpful, and a lot of them have probably forgotten the old commands for triggering manual actions, because why would you remember them? AI became part of how people work one convenient shortcut at a time, and "what happens when it fails?" stopped being a theoretical question.
Michael sees the same thing at Skyscanner, and he's refreshingly honest about being part of it. He uses AI to debug code, diagnose failures and work out why an alert fired. When a build fails, he'll often skip reading the error and just ask the agent to fix it. He also told us about a point where he'd set up so many loops and agents on his own projects that he couldn't describe exactly what was happening at any given moment. Nothing had gone wrong - he just no longer had the picture in his head.
That's the real shift. Teams have not only added a dependency, they've started delegating the understanding as well as the work. So when the model goes away, you lose the output, plus some of the context you'd have built up by doing the work yourself.
"Just fail over to another model" is harder than it sounds
If you've done any infrastructure work, you know the reflex. A dependency is flaky, so you add a second one and route around the first. Two incident.io engineers, Patrick and Mike, turned off Pub/Sub in production and nobody noticed, which is the dream outcome for that kind of work.
Skyscanner took the same approach early, building an LLM gateway so internal clients make a single call and the platform team can switch the model or provider behind it. In principle, that's a model outage handled.
In practice, it's a lot messier than failing over a database. Michael and Sahaj gave a few reasons:
- Models don't behave the same way. You can't reasonably expect the exact same prompt to work the same way on a different model, and plenty of prompts are tuned to one in particular. Sahaj is optimistic here, because frontier models have got much better at following instructions in the last five or six months and he thinks we're converging. But we're not there yet.
- Not everything is a technical constraint. Michael's team has to weigh legal and cost considerations that change depending on the workload: how sensitive is the data, how quick does it need to be, how much can you spend? On the incident.io side, some customers have requirements about which subprocessors are used. If a customer isn't okay with their data going to a particular provider, you can't quietly reroute them just because that provider is having a bad day.
- Granularity is a design decision. Do you give every single prompt a fallback provider, or do you work at the level of a whole feature? Fine-grained fallbacks are more resilient but make development heavier, and nobody wants to add a pile of complexity to every change. Coarse ones are easier to build and blunter when they kick in. There are pros and cons either way.
- Fallbacks you don't use go stale. This one isn't specific to AI, but it bites just as hard. If you rarely exercise a fallback path, you slowly lose confidence in it, and the day you need it is the day you find out whether it works.
Michael's team has tamed some of this by sorting models into tiers: fast, cheap models for lighter workflows and frontier models for heavy ones. The failover rules then apply to a class of model, rather than needing a fresh decision every time a new one comes out. It helps a lot but it doesn't make the problem disappear.
When the model is up, but getting worse
There's another reason failover isn't enough, which is that an AI system can fail without going down at all.
A model can return a perfectly healthy response that's quietly much worse than yesterday's. Nothing on your uptime dashboard will tell you, but Michael described exactly this after a failover at Skyscanner, when engineers started asking whether something had changed because the responses they were getting seemed "really dumb". It's a good reminder that the quality bar has to be part of what "working" means, alongside latency and availability.
Catching that means measuring quality like any other signal. Sahaj's team traces usage across the incident.io platform the same way it traces any other request. They also run LLMs as judges to grade the quality of an interaction or a response, and those grades become metrics they alert on.
Sahaj was upfront about the limits. Those graders lag, because they mostly catch the kinds of failure already seen, and the team would love to get faster at spotting new ones. They're getting a lot more robust over time, but they're not a silver bullet.
So they also lean on much simpler signals, which are surprisingly good. A thumbs up or thumbs down on a chat response gives you more information than you'd expect. So does a user swearing at an agent and telling it that's not what they asked for. If that's happening a lot, something has gone wrong, and no dashboard will say it as clearly.
So what does "graceful" actually mean?
Michael and Sahaj gave slightly different answers to this, and I think the difference is useful.
Michael's is the one you'd expect from someone running a platform: "Graceful degradation means the end user just doesn't see anything." Engineers carry on as normal. Behind the scenes, someone writes up the post-mortem about the provider failing and the config change that switched traffic over, but nobody on the receiving end ever knows.
Sahaj's comes from building agents, and it's more about outcomes than invisibility. A lot of what our AI products do is take a pile of signals and work out what's gone wrong, and there are usually many paths to the answer. If part of the system is down, the best case is that the agent still finds its way to the root cause. Maybe it takes longer, maybe it costs more, maybe it does more work itself instead of leaning on other agents and falls back to more old-school, deterministic workflows that go and fetch the same information. What matters is that the incident still gets resolved and that responders still get what they need, in the right place, at the right time.
Put the two together and you get a test I find helpful: a system degrades gracefully when the person using it still gets what they came for, even if the route there is slower, pricier or less impressive than usual. Sometimes they won't notice anything. Sometimes they'll notice a lot and still be unblocked. Either is far better than the alternative, where something that has quietly become a core part of someone's workflow just stops.
Principles for graceful degradation
Nobody has this fully figured out, and there’s plenty of maturity still to be found. But a few principles keep coming up, and they’re the ones we’re trying to build into how incident.io works.
Give every AI feature a floor. For each feature, ask what happens if the model isn't there at all, and build that path on purpose. Sometimes it's a deterministic version of the same job. Sometimes it's the old manual action, with a clear message about what's going on. The floor doesn't have to be pretty. It does have to exist.
Compose, so failure is partial. incident.io's AI products are a composition of lots of smaller bits of agentic work, each with its own mode of fallback. That means there's no single place that, when it breaks, takes the whole feature down. When something fails, the rest keeps contributing and the feature degrades rather than collapses. It costs more design effort up front, but it's worth it.
Keep the model off the critical path where you can. Sahaj's team still tries not to put AI on the true critical path of certain parts of the product. He expects that to change over time as models and tooling mature, but for now it's a sensible default. They're also careful about how model output is presented, since a confident-looking wrong answer in the middle of an incident is its own kind of failure.
Don't use a model where you need determinism. Michael was building a tool that used a model for some analysis, and by the end he'd written so much Python around it, to force the output into the right shape, that the model was only responsible for a tiny part of the result. His question to himself was whether he'd really needed the model at all. Models are non-deterministic by design, and if you're trying to force 100% determinism out of one, you probably don't need a model there.
Exercise your fallbacks. An unexercised failover is a story you tell yourself. Route a share of real traffic through your fallback, break things on purpose, and find out in a quiet week rather than the worst one.
Pick the quality bar per use case. Is accuracy more important than speed? It depends. If a model is quickly scoring something to give you a confidence level, you want it fast and you don't want it blocking a workflow. If it's doing something complex and expensive, you'd rather pay for the best answer than get a quick one that feels like wasted money. Deciding that up front tells you what an acceptable fallback looks like.
Reset expectations, yours included. Michael made a point I keep coming back to. We've got used to agents giving us answers instantly. An on-call engineer joining an incident used to read through channels and dashboards to catch up, and now they expect a summary the moment they arrive. When the agent's down, it feels like something's broken. But as Michael put it, this doesn't have to happen in seconds or milliseconds, and it's okay to take a little longer if something's degraded. Reading the dashboards yourself was normal not very long ago, and it should still be a perfectly acceptable fallback.
A short checklist
If you're wondering where to start, these are the questions worth asking about any AI feature:
- If the model were unavailable right now, what would the user see?
- If it were available but noticeably worse, how would we find out, and how quickly?
- Is there a deterministic path to the same outcome, even if it's slower or cruder?
- When did we last exercise the fallback with real traffic?
- Are there customers or workloads where we can't switch providers, and what happens to them?
- Would a person know what to do next if the AI disappeared mid-task?
If you can't answer some of these yet, you're in good company. Neither Michael nor Sahaj could answer all of them cleanly either, and they said so.
Every team's constraints are different, so take what's useful. But if you're putting AI in front of people who depend on it, it's worth spending some of your time on what happens when it breaks, because it eventually will.
How Pinterest Improved Analytics Agent Compliance with Loop Engineering
Pinterest increased compliance for its Analytics Agent from 64% to 89.3% by implementing a iterative "loop engineering" test-driven development process.
Summary
Deep Dive
- The team identified that manual instruction editing was inconsistent and fragile.
- They implemented a "loop engineering" workflow mirroring CI/CD practices.
- Changes to agent instructions are tested against specific, paired test suites.
- The first implementation achieved an 89.3% compliance rate, though 10.3% of test cases showed non-deterministic behavior between runs.
- The approach treats system instructions as a codebase that requires regression testing.
Decoder
- Analytics Agent: An AI system used to execute SQL queries on data platforms.
- Loop engineering: A methodology where prompts or system instructions are treated as code, subjected to continuous integration and automated testing cycles to ensure predictable output.
Original Article
Pinterest's Analytics Agent can skip required steps, such as inspecting table metadata before writing SQL. Manual instruction edits left one rule at 64% compliance, so the team built loop engineering: propose a change, test it on paired test suites, and keep what works. The first large run reached 89.3% compliance, though 10.3% of cases changed verdict between runs.
Do Data Roles Have a Future?
AI makes production work like coding and data plumbing cheap, forcing data professionals to focus on interpretation, accountability, and complex sociotechnical modeling.
Summary
Deep Dive
- Data work is categorized into production, interpretation, and accountability.
- Routine coding tasks are being commoditized by AI.
- Foundational data modeling remains a human-driven, sociotechnical challenge.
- Traditional roles like data engineer and analyst are blurring into generalist roles.
- Junior career paths are threatened by the lack of manual "coding from scratch" experience; engineers must learn diagnostic judgment explicitly.
Decoder
- Data modeling: The process of creating a visual representation of an entire information system or parts of it to communicate connections between data points.
Original Article
Do Data Roles Have a Future?
Does data engineering have a future? I get asked this all the time now, along with similar questions about pretty much every other data role. While things are still playing out, here are my thoughts on where we stand today.
Students wonder if they should bother getting into data engineering, experienced folks worry if their jobs will be around in five years, and vendors keep pitching AI agents that promise to replace entire data teams. Having co-written Fundamentals of Data Engineering, I’ve been thinking about this a lot.
The short answer is yes, data engineering has a future. The data engineering lifecycle and its undercurrents aren’t going away. What’s far less clear, though, is whether the tactical day-to-day work we’ve traditionally associated with data engineers will still define the job. What happens when the tasks we spent years mastering suddenly become dirt cheap to automate? As a result of these shifting economics, people are facing a major identity crisis, from analysts and analytics engineers to data scientists, architects, and managers alike.
The Shifting Economics of Technical Artifacts
The economics of data work are shifting fast. Writing SQL, dbt, or Python, and managing infra are increasingly being handed off to AI agents that can generate code, run tests, write docs, and debug systems faster and cheaper than humans. I don’t know anyone strictly hand-writing code these days. And that’s not necessarily bad, since AI speeds things up and lets us experiment rapidly with implementations that used to take days. Rapid iteration is the name of the game nowadays.
What’s really changing is how we generate technical artifacts, including code, pipelines, dashboards, and docs. But artifacts alone don’t solve business problems. To figure out where value actually lies, it helps to break data work into three parts:
- Production. Building the artifacts - writing code, creating pipelines, building dashboards, and writing documentation.
- Interpretation. Understanding the business context - figuring out what data actually means, making architectural choices, and uncovering why metrics don’t line up.
- Accountability. Deciding what’s right, taking on risk, and owning the outcomes when things break.
AI makes production dramatically cheaper, but navigating ambiguity and owning big decisions remain fundamentally human and organizational challenges. This is the messy part that AI alone won’t solve. But I’m hopeful AI will shine a bright light, expose problems in real time, and help humans resolve them much faster.
Even as models get better at handling complex tasks, the immediate takeaway is clear: the balance between production, interpretation, and accountability has changed for good. When pumping out artifacts becomes almost free, knowing which ones to build, how they ought to behave, and how to verify their accuracy become the real core skills.
Findings from the 2026 Practical Data Community Survey showed that while 80% of respondents used AI daily, only about 10% had actually built it into organizational workflows, and nearly 89% were still wrestling with basic data modeling problems. That gulf speaks volumes. Code generation tools are moving at warp speed, but organizations still struggle with semantic clarity and data modeling. Much to people’s surprise (and vendor’s dismay), things like data modeling and ontologies are sociotechnical problems that can’t simply be automated away.
Take revenue calculations, the classic trope we like to debate when modeling data. AI can easily spit out syntactically correct code, but it can’t automagically determine whether you mean gross vs. net or recognized vs. invoiced revenue. Without solid data modeling and domain clarity, automated tools will just generate incorrect business logic faster. This isn’t bad in itself, since finding things out faster can help resolve issues more quickly than in the past. But we still need to resolve the ambiguity, and that’s a human problem.
AI can automate writing code, but it can’t settle disagreements between stakeholders over definitions. Human judgment is still required to set policy and ensure systems align with what the business actually needs. This is why decades of IT implementations still fail for the same reason: sociotechnical work is very messy and difficult.
Roles are Blurring. Titles Won’t Matter as Much
Every data role is shifting in response to these changing economics. Here’s a breakdown of where I see roles heading.
- Data engineers will spend less time plumbing pipelines by hand and more time on system design, reliability, cost management, and orchestrating automated workflows.
- Analytics engineers will spend less time churning out dbt models and SQL transformations and more time refining business semantics, validation, and data contracts.
- Data analysts will move away from static reporting toward asking sharper strategic questions, digging into edge cases, and interpreting subtle business trends.
- Data scientists will delegate standard modeling tasks to AI, placing more emphasis on experimental design, causal inference, and real-world evaluation.
- Data architects will design across complex environments, knowledge graphs, and agentic workflows, taking a broader, macro view of data movement.
- Managers will need to stay hands-on as smaller teams leveraging AI pull off work that used to require big departments, blending leadership with direct technical oversight.
Again, AI is eroding production work. With AI assistance, engineers can easily run complex analyses, analysts can build working tools, and data scientists can push code straight to production. In the bigger picture, I see these roles blurring together. Last year, I noted this convergence. Today, that convergence is accelerating. This convergence is giving rise to versatile generalists who move fluidly across domains. As tooling friction disappears, rigid job titles will matter much less than what you can actually execute. Deep specialists in fields like distributed systems, performance tuning, and security aren’t going anywhere, but for general data roles, performance and execution will trump traditional titles every time.
One real concern is how junior engineers will grow. Historically, we built intuition by writing code from scratch, breaking things, and fixing them. As entry-level coding tasks get automated away, companies need to get deliberate about creating pathways for early-career engineers to develop real diagnostic judgment. True engineering seniority comes from understanding how systems break, spotting subtle flaws in logic, and being accountable for production systems. These are skills you simply can’t acquire by blindly accepting AI suggestions.
If you’re starting out or adapting your career right now, focus on mastering core engineering fundamentals while using AI tools aggressively. Read the generated code carefully, understand what’s under the hood, build real projects, and learn to justify your design decisions.
What’s Left For Humans?
Looking ahead, a few main trends stand out.
- Leaner, highly effective teams will punch far above their weight class. Big teams simply won’t be necessary and will probably be a liability. This sucks for managers who like to build kingdoms, but it is what it is.
- Fluidity across roles will grow as lower implementation costs break down traditional silos.
- Autonomous agents working as both data producers and consumers will demand stronger governance, data lineage, and semantic consistency. This is where humans can help resolve ambiguity.
- Validating and evaluating generated code will be just as important as writing it in the first place.
Even as technical roles adapt, people who can connect business needs with technical solutions will always be in high demand. As routine tasks get automated, we can spend more energy on strategy and solving higher-level problems.
What’s left for humans in data work? So much of our identity has been attached to the production stuff. Countless user groups are dedicated to programming languages and frameworks. The key question for our industry is this: what value do data professionals bring when producing code is no longer the bottleneck? Thriving in this new era means doubling down on fundamentals, rethinking our skill sets, and building strong networks.
How Open Is Open? Iceberg, Catalogs, and User Choice
True interoperability in data platforms requires a community-maintained baseline for catalogs, not just open table formats like Iceberg.
Summary
Deep Dive
- Iceberg's table format is open, but catalog behavior is often vendor-specific.
- Catalog dependencies include commit logic, metadata handling, and credential management.
- The REST catalog protocol is the current target for standardizing engine-catalog interaction.
- The community should adopt a "REST Compatibility Kit" for testing.
- Standardized benchmarking for metadata loading and concurrent writes is needed to compare vendors fairly.
Decoder
- Catalog: The component in a data lakehouse that maintains table metadata, tracks versions, and manages concurrency.
- Table format: A specification (like Iceberg, Delta Lake, or Hudi) that manages how data files are organized and how metadata is stored.
Original Article
How Open Is Open? Iceberg, Catalogs, and User Choice
The next test of openness: giving users the freedom to choose engines, control their catalogs, and change providers.
Building a vendor-neutral data platform has long been a dream for companies: one free of vendor lock-in, with better cost control, and one that enables faster analytical insights. A simple and fair ask. We all know the reality is different, even with “open” table formats.
Recent debates about Apache Iceberg’s openness and interoperability have reinforced a concern I believe the industry must address:
users need a definition of openness they can verify independently of the vendors making the claim.
For data engineering leaders, openness carries architectural and financial consequences. Can a team change query engines, operate its own catalog, or replace a provider without rebuilding its platform? What capabilities, performance, or control would it lose?
Apache Iceberg provides a key foundation for that independence. Iceberg’s REST catalog protocol puts commit logic behind a common interface and supports mechanisms to delegate storage access. The catalog therefore influences how engines access and update tables. Its operator occupies a consequential position in the architecture.
That position creates commercial influence. Vendors invest in integrations, security, and performance to make their platforms attractive. Users need to understand when those advantages create dependencies that constrain future choices.
A published interface establishes a contract; a shared implementation and reproducible tests help expose its practical boundaries.
Connecting an engine to a catalog proves only part of the interoperability story. Production introduces growing metadata, concurrent writers, credential renewal, and demanding workloads. Iceberg supports features like lazy snapshot loading and server-side scan planning. Engine and catalog implementations can use these capabilities differently, affecting efficiency before data processing begins.
Better engineering should deliver better performance. Users need evidence to distinguish those improvements from incomplete integrations, unsupported features, and proprietary dependencies. Otherwise, every organization must independently investigate the boundaries of openness.
The Iceberg community should establish a clear operational baseline and encourage the ecosystem to adopt it.
Hive Metastore offers a useful precedent. It gave engine developers a concrete integration target and organizations a service they could operate. That common foundation helped developers reproduce behavior and users investigate problems outside a vendor’s environment.
The community already has substantial work to build on. Apache Polaris implements Iceberg’s REST API, and Iceberg maintains a REST Compatibility Kit for testing catalog servers. The next step should connect these contributions to explicit expectations for adoption, validation, and user choice.
The community should rally around a production-ready catalog that users can deploy independently, and engines can support as a common integration target.
It should welcome alternative implementations while keeping the specification authoritative. Public, versioned compatibility results should identify supported operations, optional capabilities, configurations, and limitations.
Performance workloads should complement those results. With controlled data, compute, network placement, and cache conditions, users could compare metadata loading, planning behavior, and concurrent commits. Those comparisons would explain performance differences and help teams make informed decisions.
Migration must receive equal attention. Changing catalogs involves permissions, identities, credentials, catalog-specific state, and active writers. The community should publish tested procedures that explain what users can move and which dependencies remain.
A dependable baseline would strengthen competition. Vendors could demonstrate improvements against shared expectations. Users could evaluate the benefits and dependencies together. Engine developers could integrate against a foundation the community maintains and tests.
The public debate exposed a question that deserves an engineering answer. The Iceberg community should turn that question into shared code, published evidence, and practical guidance.
“How open is open?” should come down to what users can run, what they can change, and what they can take with them. The community must give them the evidence to make that choice.
Data governance that actually runs
Effective data governance is being achieved by treating policy as code, enforced automatically across an open-source stack including Airflow 3, dbt, and DataHub.
Summary
Deep Dive
- Governance must be automated via CI to remain relevant.
- Data contracts serve as the source of truth for schema and ownership.
- The stack uses MinIO for storage and Postgres for operational data.
- Airflow 3 and dbt handle pipeline orchestration and transformation.
- DataHub provides lineage, while Apache Ossie and Superset handle semantic definitions and reporting.
- AI assistants (MCP-enabled) can be used to navigate this governed environment.
Decoder
- Governance-as-code: Managing and enforcing data policies through automated scripts and pipeline configurations rather than manual audits.
- Data contract: A formal agreement between data producers and consumers that defines the schema, semantic meaning, and quality expectations of data.
Original Article
Governance-as-code turns policy into checks across a local, open-source stack: MinIO for S3-like intake; Postgres for operations and warehousing; Airflow 3 and dbt for pipelines; DataHub, Apache Ossie, and Superset for catalog, semantic definitions, and dashboards; and MCP-connected AI assistants. CI enforces contracts, ownership, masking, lineage, consumer-impact approval, and least-privilege access before production.
PyPI's redesigned project pages with clearer security signals are now live
PyPI has updated its project pages with a new security tab that visualizes package provenance and warns users about suspicious publishing behavior.
Summary
Deep Dive
- Full Provenance: All release files have been verified by Trusted Publishing or SLSA build records.
- Provenance Warnings: Alerts appear if file origins conflict with project history, such as a shift in publishing systems or unusual account activity.
- UI Changes: Metadata is now prioritized by utility and verification status, and file history is searchable.
Decoder
- Provenance: Information about the origin, custody, and ownership of software artifacts.
- Attestation: A signed statement providing verifiable evidence about how a file was produced.
- Trusted Publisher: A mechanism (often using OIDC) that allows services like GitHub Actions to upload packages without long-lived API tokens.
Original Article
PyPI's redesigned project pages with clearer security signals are now live
In July, we announced a series of planned updates to the PyPI user interface. Shaped by user research with members of the Python community, the updates aim to improve how we surface security signals and display package details.
We're pleased to share that this work is now complete and live on PyPI.
Context
With the rapid growth of the Python ecosystem, PyPI has become an increasingly attractive target for malicious actors. In response, the PyPI team has delivered many security features while taking a more active role in helping PyPI users assess and understand package security.
Digital attestations, introduced in 2024, are an example of this effort, providing a verifiable record of a release file's provenance (where it came from). However, provenance and attestations are still unfamiliar concepts for many PyPI users, and PyPI's interface did little to communicate their meaning, importance, or limitations.
To address this, PyPI partnered with Kabu Creative to bring clearer security signals to the PyPI UI, implementing the recommendations developed in the OpenSSF Attestations Style Guide.
Here's a look at what's changed, what it means for you, and what actions you can take.
The New Security Tab
The new security tab communicates release provenance states and displays per-file attestations, giving you a dedicated place to verify where the release files came from.
Provenance States
PyPI calculates each release's provenance state by checking whether each release file has an attestation. Typically, a release will be in one of two states:
Full Provenance
This means that 100% of the release files have attestations, carrying either or both of the supported attestation types:
PyPI Publish attestations confirm the file was published by a PyPI Trusted Publisher.
SLSA Build Provenance attestations link the file back to its exact source commit, build platform, and instructions.
You can review attestation metadata directly in the UI, or click on the transparency log link for more information.
No Provenance
"No provenance" simply means that no file in the release has an attestation. This is typical for older releases or projects that have not yet configured Trusted Publishing. No provenance does not necessarily indicate that a release or release file is malicious; it instead means that the file origin cannot be cryptographically verified.
When there is no provenance, you should rely on surrounding trust signals to assess security, such as maintainer history, release patterns, and repository activity. If you maintain the package, consider using Trusted Publishing with a publisher that generates attestations for your next release.
Provenance Warnings
Provenance warnings are designed to help you identify when a file may not have come from the project's maintainers, for example, if an attacker has taken over an account or a publishing workflow.
When provenance metadata looks unusual or conflicts with a project's history, PyPI surfaces a prominent warning. Many warnings have legitimate explanations, but each one is worth checking before you install.
Incomplete Provenance
This means that some files carry attestations while others do not (e.g., wheels were published via Trusted Publishing, but the source tarball was uploaded using an API token).
Exercise caution if you see this warning. Check release notes for build issues or ask in the project's issue tracker before installing.
Inconsistent Provenance
This warning indicates that files within the same release originate from different build systems or source repositories.
Check the attestation metadata to confirm whether all files came from sources controlled by the maintainers.
Change of Provenance
This warning tells you that the release files originated from a different repository or workflow than a previous release. While often legitimate (such as an organization name change or repository migration), it can also indicate a compromised pipeline or account takeover.
If you see this warning, check the project release notes or official channels to confirm that an infrastructure migration was planned.
Loss of Provenance
This means that a project that previously published with attestations has changed to uploading release files without them.
Check whether the maintainers intentionally changed their publishing process, for example by switching back to API tokens, or by changing their publishing configuration so that it no longer generates attestations. If this is unexpected, treat the release with skepticism until verified.
To learn more about provenance and attestations, including provenance warnings, see our user documentation.
Redesigned Project Detail Page
To support these changes, the project detail page has also been updated:
- Provenance warnings now appear at the top of the page, alongside status banners for yanked releases, archived projects, and pre-releases. If a project or release is quarantined, only the quarantine banner is shown, as it takes precedence over all other warnings and banners.
- When a release has full provenance or provenance warnings, its provenance state is displayed in the sidebar, linking through to the security tab for more information.
- Metadata is now ordered by utility and labeled as "PyPI data" or "Data verified by PyPI", replacing the old design where data was grouped and ordered by verification status.
Release Files and Release History
The redesigned release files tab helps you find the right file by searching and filtering long lists. You can also view the provenance state of each file within the table, and quickly identify late files, which are marked with an orange warning icon.
Late file warnings are shown when one or more files were uploaded more than 14 days after the initial release timestamp. These warnings only appear on releases prior to July 2026, when a policy was introduced to reject new files after 14 days.
The release history tab now also provides more details about each release, exposing file count, provenance state, and release status (i.e., quarantined, yanked, archived, or pre-release).
Getting Help
Alongside the interface updates, PyPI's user documentation has been extended to help you better navigate package security. Read our package security guide to learn about:
- Assessing source code security
- Checking provenance (where a package came from)
- Verifying package integrity
- Protecting yourself from common security attacks
- Reporting security issues
Looking Ahead & Acknowledgments
The new security tab gives PyPI a dedicated home for future attestation types and security notices. To help the wider open source ecosystem benefit from this work, we've also published a case study in the OpenSSF Attestations Style Guide detailing the design patterns and warnings developed during this project.
This initiative was funded by the Open Source Security Foundation and implemented by Kabu Creative. We would like to thank Dustin Ingram, Zach Steindler, and the members of the OpenSSF Securing Software Repositories Working Group for championing this effort. Special thanks to Mike Fiedler, Jacob Coffee, and members of the PyPI team for their thoughtful design feedback and code reviews, @cofiem for their contributions to the new release files tab, and our user interview participants and community testers for helping shape the final design.
Finally, we would like to thank the OpenSSF for its funding and continued partnership in securing the Python packaging ecosystem.
db-branch (GitHub Repo)
db-branch creates near-instant database clones using copy-on-write ZFS snapshots, allowing developers to spin up isolated environments for testing and AI agents.
Summary
Deep Dive
- Technology: Built on ZFS for copy-on-write cloning; compatible with MySQL and PostgreSQL.
- Features: Supports checkpointing, branching, resetting, and per-branch version upgrades.
- Integration: Offers an MCP server to allow LLMs to create their own sandboxed database environments for testing queries.
- Clustering: Supports Raft-based clusters for distributed setups.
Decoder
- Copy-on-Write (CoW): A strategy where files are initially shared between clones, and data is only copied when a modification occurs.
- ZFS: A file system that integrates logical volume management and snapshot capabilities, allowing near-instant cloning.
Original Article
db-branch
Instant database branches for MySQL and PostgreSQL, on your own servers.
Test a migration, let an AI agent loose, or give every pull request its own copy of the database. db-branch creates a copy-on-write branch of a large database in seconds, with nothing copied. Break it, reset it, throw it away. Built on ZFS clones, self-hosted, Apache-2.0.
Status: Beta. db-branch is usable today, but APIs and command-line details may still change before 1.0. It is an independent open-source project, built from scratch.
Try it
-
In your browser, 0 minutes. A live console demo runs entirely in the page (nothing real is created): https://demo.db-branch.com/
-
On a Mac or Windows machine, about 2 minutes. db-branch needs Linux and ZFS, so run it in an Ubuntu VM with Multipass. One command makes the VM, installs db-branch, initialises it on a file-backed ZFS pool and loads a small sample base,
shop(MySQL 8.4):multipass launch 24.04 --name db-branch-eval --memory 4G --disk 30G --timeout 900 \ --cloud-init https://raw.githubusercontent.com/db-branch/db-branch/main/scripts/eval-cloud-init.yaml multipass exec db-branch-eval -- cloud-init status --wait multipass exec db-branch-eval -- cat /etc/motd multipass shell db-branch-eval # then: sudo db-branch branch create try --base shop --ttl 1h --recordOn Windows (PowerShell or cmd), put the
multipass launchcommand on one line and drop the\:multipass launch 24.04 --name db-branch-eval --memory 4G --disk 30G --timeout 900 --cloud-init https://raw.githubusercontent.com/db-branch/db-branch/main/scripts/eval-cloud-init.yaml- The history module is turned on, and the paid modules run for 30 days without a key.
--recordrecords a branch so it can be restored to any point. - The pool is a file, so it is slower and not meant for real data. The first admin password is
admin; the console asks for a new one at the first sign-in. - On Apple Silicon (and other arm64 machines) the setup downloads about 800 MB for MySQL, against 80 MB on Intel, because Oracle ships no minimal build for arm64.
--timeout 900keepslaunchfrom giving up on a slow connection. - If the setup fails,
/etc/motdsays so, andsudo tail -n 30 /var/log/cloud-init-output.login the VM shows why. - Delete the VM with
multipass delete --purge db-branch-eval.
- The history module is turned on, and the paid modules run for 30 days without a key.
-
On your own Ubuntu server. See Install.
Use cases
- AI agent sandboxes. The built-in MCP server lets an LLM agent create a branch, run a risky query or migration on it, and delete the branch, so production data is never touched.
- A database per pull request. Each CI run or PR gets its own copy of a large database, ready in seconds, and removed by
--ttlwhen it is no longer needed. - Version-upgrade checks. Run a major-version upgrade (for example MySQL 8.0 to 8.4) on one branch first, with
branch upgrade --check-onlyto see what it would run into, before touching the base.
What it does
- Branches in seconds. On our own 3-node test cluster a PostgreSQL branch is ready in about 0.5 s and a MySQL branch in about 3 s, with 0 bytes copied at creation (copy-on-write).
- Reset and expire.
branch resetreturns a branch to its creation state;--ttldeletes it automatically. - Checkpoints.
branch checkpointsaves a running branch's state instantly;branch restoreputs it back. Restoring replaces the current state, and every checkpoint you already have stays restorable. - Per-branch upgrades.
branch upgrademoves one branch to a newer MySQL or PostgreSQL version while the base and the other branches stay put. A checkpoint is saved first, and a failed upgrade rolls back by itself. - MySQL and PostgreSQL, several versions at once. Engine runtimes are installed by version and can live side by side.
- One interface everywhere. CLI, REST API, web console and an MCP server (so LLM agents can create and query their own branches) all use the same API.
- Single node or cluster. A Raft cluster of VMs, with optional ProxySQL routing so applications keep one address.
Install
Requirements: Ubuntu 24.04 LTS or 26.04 LTS, and a ZFS pool (the zfsutils-linux package is installed as a dependency).
curl -fsSL https://apt.db-branch.com/db-branch.gpg | sudo tee /usr/share/keyrings/db-branch.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/db-branch.gpg] https://apt.db-branch.com stable main" | sudo tee /etc/apt/sources.list.d/db-branch.list
sudo apt update && sudo apt install db-branch
sudo db-branch init --pool <zpool>
The repository signing key fingerprint is 3F404C4E5D49FA653F8385692C6E89E341D4D7A5. db-branch init prints the initial admin password once.
First branch
db-branch runtime install postgres@18.6
db-branch base create prod --runtime postgres@18.6 --dump prod.sql.gz
db-branch branch create feature-x --base prod --ttl 72h
db-branch branch dsn feature-x
db-branch branch checkpoint feature-x --name before-migration
db-branch branch restore feature-x before-migration
db-branch branch reset feature-x
db-branch branch delete feature-x
How it works
A base is a ZFS dataset holding a loaded database. A branch is a ZFS clone of a base snapshot, so creating one copies nothing, and each branch runs its own database server on its own port. Optionally, ProxySQL routes client connections to the right branch by user name (or by a /*db_branch='name'*/ query comment). Several nodes form a Raft cluster that shares bases, branches, settings and users.
Editions
- Core (this repository), Apache-2.0: branches, bases, runtimes, clustering, CLI, API, web console, MCP, monitoring, notifications, anonymous (no sign-in) branches with limits, scheduled base refresh, backup of cluster state.
- Free modules in the same apt repository:
proxysql,greenmask,greenmask-mysql,anon(routing and data masking). - Paid modules (auth/SSO, audit, history, diff, FSx storage) are separate, closed-source packages, self-hosted and enabled with a license key (monthly or yearly).
There is no hosted or cloud version of db-branch.
License
Apache-2.0.
日本語
db-branch は、ZFS クローンで MySQL / PostgreSQL のデータベースブランチを発行・リセット・破棄するツールです。現在は Beta で、1.0 までに API が変わる可能性があります。コアは Apache-2.0 で、有償モジュールは別パッケージです。
Real-time Alert in Spark
UiPath cut memory usage for its Spark alerting engine by 66% by optimizing stream-to-static joins and state management.
Summary
Decoder
- Spark Structured Streaming: A scalable, fault-tolerant stream processing engine built on top of the Spark SQL engine.
- SLA (Service Level Agreement): A contract defining the expected performance or availability; in this case, monitoring that checks if events meet specific time-bound conditions.
Original Article
UiPath built a Spark Structured Streaming alerting system that evaluates 20,000 alert definitions per minute, including windowed conditions and SLA checks, without repeatedly rescanning source data. The key design was joining streams to static definitions once and managing state carefully, which cut memory from 12 GB to 4 GB and reduced downstream network calls.
Outbox Pattern in Postgres: Concurrent Workers vs Reading the WAL
For Postgres outbox delivery, logical replication streams outperform polling by offering sub-millisecond latency at the cost of managing a replication slot.
Summary
Deep Dive
- The outbox pattern ensures atomicity by saving events to a database table within the same transaction as the primary data update.
- Polling with 'SELECT ... FOR UPDATE SKIP LOCKED' is the standard approach; it allows multiple workers to process events concurrently without contention.
- Trigger-based 'LISTEN/NOTIFY' mechanisms can cause severe performance degradation due to transaction locks holding up disk flushes during group commits.
- Logical replication (using the WAL) provides the lowest latency because it avoids querying the database for new records.
- WAL-based readers do not natively track progress like a consumer group, necessitating external state management to mark processed rows.
- All designs (polling vs. WAL) provide at-least-once delivery, requiring idempotent consumers to prevent duplicate processing.
Decoder
- WAL (Write-Ahead Log): A transaction log that records every change made to a database before the changes are applied to the data files, used for durability and crash recovery.
- Logical Replication: A feature in Postgres that allows streaming data changes from one database to another, or to an external reader, based on row-level modifications.
- Idempotent: An operation that can be applied multiple times without changing the result beyond the initial application, critical for systems that may deliver the same message twice.
- SKIP LOCKED: A clause used with 'SELECT FOR UPDATE' that allows concurrent workers to ignore rows already locked by another transaction, preventing head-of-line blocking.
Original Article
The outbox pattern saves a message in the same database transaction as the data change it describes, then uses a background worker to send it. Concurrent workers can drain the outbox table with FOR UPDATE SKIP LOCKED, but new messages wait for the next poll. Reading the Postgres write-ahead log with logical replication delivers them in about a millisecond at low load, at the cost of a replication slot and a cleanup job.
If you build distributed systems, you probably heard about the Outbox pattern.
Your API saves an order, then crashes before it can send the message announcing that order.
The order is in the database, but other services waiting for that message never hear about it. The outbox pattern handles this by saving the message in a separate table, in the same database transaction as the order.
Either both are saved or neither is. This is the A in ACID. Atomicity.
A background worker reads the saved messages and publishes them to a message broker for other services to receive. If the API crashes after saving the order, the message is still there for the worker to send. Querying the table repeatedly for pending messages is called polling.
Polling is a good starting point for an outbox processor. It's decently fast and easy to implement. But the delay between checks is a tradeoff. Checking more often means more database queries, while waiting longer delays new messages.
I want to reduce that wait by having Postgres tell me when there's something to process, and see how that compares with the polling workers.
Concurrent Workers Draining the Table
For this walkthrough, we'll use an order API on Postgres that saves each OrderCreated message in an outbox_messages table, with RabbitMQ as the broker.
I want to run several workers at once, and Postgres row locks stop two of them from grabbing the same message.
Each worker starts a transaction and claims a batch with this query, using Dapper:
SELECT id, type, content
FROM outbox_messages
WHERE processed_on_utc IS NULL
ORDER BY occurred_on_utc
LIMIT 1000
FOR UPDATE SKIP LOCKED
FOR UPDATE locks the rows this worker picked until its transaction ends. SKIP LOCKED makes the next worker take other rows instead of waiting. The locks live in Postgres, so these workers can run in one process or in several.
The worker keeps that transaction open while it publishes with RabbitMQ.Client, then sets processed_on_utc and commits once RabbitMQ confirms.
When a worker runs out of rows, it sleeps for one second with Task.Delay and tries again.
If a worker crashes before it commits, those rows are still unprocessed, and a worker picks them up again. Some of those messages may have already reached RabbitMQ, so the same message can arrive twice.
That makes this at-least-once delivery, so your consumers must be idempotent.
Why Not Just LISTEN/NOTIFY?
Postgres has a built-in way to wake those workers. With LISTEN/NOTIFY, one connection runs LISTEN outbox_new, and Postgres tells it when another connection runs NOTIFY outbox_new and commits.
My first implementation sent NOTIFY from a database trigger on the outbox table. It needs no API changes and worked well at 100 orders per second. But with 100 concurrent users, throughput fell from 8,070 orders per second with polling to 256.
The trigger runs inside the order transaction. Before that transaction commits, Postgres takes a lock and holds it until after the commit. Every transaction that sends a notification takes the same lock.
A commit normally waits for a disk flush, and concurrent transactions can share one, which is called group commit. With the lock, each order waits for the one in front of it to finish its own flush.
Sending NOTIFY after the commit, from a separate connection, avoids that lock. At 100 users the API handled 7,674 orders per second, about 5% below polling, and at 100 orders per second a message took 1.91 ms from commit to consumer.
But now I'm running a notifier, a listener, and a fallback poll for notifications sent while the listener is disconnected, all to wake the same workers. Postgres already has a record of every new row, so let's read that instead.
Read the Outbox From the WAL
Postgres writes every change to the write-ahead log (WAL). Logical replication streams committed changes to my app, so there's no polling and no wake-up to miss.
You need wal_level=logical, which takes a restart on Postgres 18. Then create a publication and a replication slot:
CREATE PUBLICATION outbox_pub FOR TABLE outbox_messages WITH (publish = 'insert');
SELECT pg_create_logical_replication_slot('outbox_slot', 'pgoutput');
The slot remembers how far my reader has confirmed, so it can resume after a restart.
Let's read the stream with an Npgsql replication connection:
await using var connection = new LogicalReplicationConnection(connectionString);
await connection.Open(cancellationToken);
var slot = new PgOutputReplicationSlot("outbox_slot");
var options = new PgOutputReplicationOptions("outbox_pub", PgOutputProtocolVersion.V1);
await foreach (var message in connection.StartReplication(slot, options, cancellationToken))
{
if (message is InsertMessage insert)
{
await PublishAsync(insert, cancellationToken);
}
else if (message is CommitMessage commit)
{
connection.SetReplicationStatus(commit.TransactionEndLsn);
}
}
PublishAsync publishes to RabbitMQ and waits for the confirm. I only confirm a position after that, when the CommitMessage arrives, so a crash replays messages instead of losing them.
This loop only suits low rates, because waiting on every message drained about 1,600 per second. So the reader I measured publishes first and waits for confirms, in order, in a separate task.
While the reader is down, the slot keeps all the server's WAL on disk, so a long outage can fill it. You can cap that with max_slot_wal_keep_size, but Postgres invalidates a slot past the cap.
This reader is a prototype, and I haven't tested it through a failover or on managed Postgres.
Workers vs the WAL Reader
The main benchmark comes from my scaling article: I seeded a backlog, started the processor, and counted what RabbitMQ had after one minute, in messages per second (MPS). To keep it fair, the WAL reader also marks rows as processed in batches, so it does the same UPDATE as the workers. This ran on one machine, with Postgres 18 and RabbitMQ 4 in Docker:
| Setup | MPS after 1 minute | Average commit to consumer, 100 orders/s | Average commit to consumer, 100 users |
|---|---|---|---|
| 1 worker | 14,516 to 15,050 | 500 ms | 8.5 s |
| 5 workers | 22,216 to 24,133 | 419 ms | 54 to 85 ms |
| WAL reader | 27,962 to 34,930 | 0.83 ms | 8.6 ms |
One worker already moves about 15,000 messages per second, and five get to about 23,000. The WAL reader leads at about 31,000, and it reached about 57,600 without the UPDATE. Both move thousands of messages per second, so throughput alone isn't a reason to switch.
Latency is the real difference. New messages wait for a worker's one-second sleep, so five workers average 419 ms at 100 orders per second, while the reader stays under a millisecond. At 100 users, one worker fell 8.5 seconds behind, five workers averaged 54 to 85 ms, and the reader held 8.6 ms.
A crash goes the other way. A hard kill partway through 200,000 rows lost nothing, but the workers sent 4,536 duplicates and the WAL reader 31,095, because it replays from its last confirmed position.
Workers also scale out, because you can add more in any process. A slot has one reader, which keeps commit order, but I can't scale it out.
Summary
Keep polling if the delay is acceptable for your application. Add workers when you need more throughput, because the row locks already handle the concurrency.
Read the WAL if you need new messages right away and can run a replication slot and a cleanup job. If you use NOTIFY instead, send it after the commit and keep a fallback poll.
A worker can still crash after publishing but before committing its processed markers, causing a message to be sent again. Consumers must be idempotent: processing the same message twice must not repeat its effects.
Frequently Asked Questions
How do concurrent outbox workers avoid publishing the same message twice?
Each worker starts a transaction and claims a batch with FOR UPDATE SKIP LOCKED. FOR UPDATE locks the selected rows until the transaction ends, and SKIP LOCKED makes other workers take different rows instead of waiting. The locks live in Postgres, so workers can run in one process or in several. A crash before the commit releases the locks, so another worker publishes those rows again, which makes delivery at least once.
Why not send NOTIFY from a trigger inside the transaction?
A transaction that sends a notification takes a lock in Postgres and holds it until its commit is done. So these transactions commit one at a time, and each waits for the previous one to finish its disk write. In my k6 runs with 100 users, that cut the API from about 8,070 to 256 orders per second. Sending NOTIFY after the commit, from a separate connection, avoids the lock and held 7,674 orders per second, but it adds a notifier, a listener, and a fallback poll.
Is a Postgres notification durable?
No. Postgres only delivers a notification to connections that are listening at that moment, so a listener that is disconnected when the commit happens never sees it. Keep a fallback poll, for example every 5 seconds, so idle workers check for messages even if they miss a notification.
What does polling the outbox cost when there is nothing to process?
With five workers and no delay, the claim query ran about 289,000 times a minute and kept the Postgres container at 118 percent CPU. A one-second delay after an empty batch brought that to 305 queries a minute. A logical replication reader runs no outbox queries while idle.
How many outbox messages per second can workers and a WAL reader publish?
Over a one-minute drain of a backlog, one worker published about 15,000 messages per second and five workers about 23,000. A logical replication reader doing the same batched UPDATE published about 31,000, with two runs at 27,962 and 34,930, and about 57,600 without the UPDATE. These are results from one machine, with Postgres and RabbitMQ in Docker.
When should an outbox read the Postgres write-ahead log instead of using polling workers?
Read the write-ahead log when you need new messages right away and you can run a replication slot. At 100 orders per second, the commit-to-consumer average was 0.83 ms with the reader and 419 ms with five workers polling after a one-second sleep. It needs wal_level=logical, a slot that keeps WAL on disk while the reader is down, and a cleanup job, because the reader does not mark rows as processed by itself. After a hard kill it sent more duplicates, 31,095 against 4,536 of 200,000 rows, and neither design lost a message.
Postgres not using your index? It's counting pages, not rows
Postgres ignores indexes because it optimizes for 8KB page reads rather than individual row counts, making indices ineffective for high-selectivity queries.
Summary
Deep Dive
- Postgres reads data in 8KB pages; fetching a single row requires loading the entire page it resides on.
- Index lookups perform well only when the required rows are concentrated on a small fraction of the table's total pages.
- High-selectivity queries that are scattered physically across the table may cause more random I/O than a sequential scan.
- The query planner uses 'cost' metrics based on estimated page reads, not row counts, which can lead to 'bad' plans if statistics are outdated.
- 'ANALYZE' updates the planner's statistics, but 'pg_stats' should be inspected to understand how the planner views data distribution.
- Partial indexes (e.g., 'WHERE status <> 'paid'') significantly reduce index size and can improve performance for queries targeting rare values.
Decoder
- Heap: The primary table storage area in Postgres where actual data rows are stored.
- TID (Tuple Identifier): A physical address consisting of a page number and an offset within that page, pointing to a specific row in the heap.
- Bitmap Heap Scan: An execution plan that scans an index to build a map of required pages, then visits those pages in physical order to reduce random disk I/O.
- Selectivity: The proportion of rows in a table that satisfy a given filter condition; low selectivity means a large portion of the table matches.
Original Article
Slow Postgres, part 1 of 8. The SQL and raw numbers for every test are on GitHub.
You add an index on status, run EXPLAIN, and Postgres reads the whole table anyway. ANALYZE doesn't change the plan. Someone suggests SET enable_seqscan = off, the plan finally shows your index, and the query gets slower.
Usually the planner is right. It prices a plan by the 8 KB pages it has to read and by whether it reads them in order or jumps around between them, and once you count pages the same way, you can answer most "why is it ignoring my index" tickets yourself.
This series started as my notes from an open lesson on slow Postgres queries by balun.courses. The tests and numbers are my own.
The short version:
- Postgres prices a plan in pages, and in my test table a status with 0.9% of the rows sits on 94% of them, so its index saves 12 page reads out of 221.
- On 5 million rows in RAM, the index beat a full scan up to 10% of the rows when the matches were scattered, and at every share I tried (up to 50%) when they were stored together.
- Read cold from disk, the scattered lookups at 0.5% and 1% of rows ran 2 to 5 times slower than a full scan, and Postgres still picked them.
- A 200-row table gets a Seq Scan even when 2 rows match, so a plan from a tiny test copy says nothing about production.
- Keep
enable_seqscanout of app code, readpg_statsbeforeCREATE INDEX, and if you only query the rare values, index only those rows.
What a B-tree index stores
A regular B-tree index keeps only keys, the values it is sorted by, and next to each key a TID: the row's address in the heap, which is the table's main storage.
A TID is a page number plus a slot on that page. You can see it as the hidden ctid column:
SELECT ctid, id, status FROM orders WHERE status = 'chargeback' ORDER BY id LIMIT 3;
ctid | id | status
---------+------+------------
(7,48) | 1000 | chargeback
(14,96) | 2000 | chargeback
(22,8) | 3000 | chargeback
Row 1000 is slot 48 on page 7. An index lookup for 'chargeback' returns a list of addresses like these, and Postgres then visits each of those heap pages to fetch the row and check that your transaction can see it.
The second step is the one people forget. To fetch one row, Postgres has to read the whole 8 KB page it sits on.
The lab: 30,000 orders on 221 pages
The lab runs on PostgreSQL 17.11 in Docker with default settings. My numbers come from a 4-vCPU, 8 GB cloud VM, but any small VPS or a laptop will do.
CREATE TABLE orders (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
created_at timestamptz NOT NULL,
amount_cents int NOT NULL,
status text NOT NULL
);
INSERT INTO orders (created_at, amount_cents, status)
SELECT now() - (30000 - i) * interval '1 minute',
(random() * 10000)::int,
CASE
WHEN i % 1000 = 0 THEN 'chargeback' -- 30 rows, 0.1%
WHEN i % 100 = 0 THEN 'failed' -- 270 rows, 0.9%
WHEN i % 10 = 0 THEN 'created' -- 2,700 rows, 9%
ELSE 'paid' -- 27,000 rows, 90%
END
FROM generate_series(1, 30000) AS i;
CREATE INDEX orders_status_idx ON orders (status);
ANALYZE orders;
SELECT relpages, reltuples FROM pg_class WHERE relname = 'orders';
relpages | reltuples
----------+-----------
221 | 30000
136 rows fit on a page, so 30,000 rows take 221 pages. I picked 30,000 because ANALYZE samples 300 × default_statistics_target rows, which is 30,000 at the default. On this table it reads every row, so the statistics are exact and your plans should come out the same as mine.
What a Seq Scan costs
Ask for the most common value and the index goes unused:
EXPLAIN SELECT * FROM orders WHERE status = 'paid';
Seq Scan on orders (cost=0.00..596.00 rows=27000 width=25)
Filter: (status = 'paid'::text)
221 pages × seq_page_cost 1.0 = 221
30000 rows × cpu_tuple_cost 0.01 = 300
30000 rows × cpu_operator_cost 0.0025 = 75
---
596
Cost has no unit you can convert to milliseconds. It only means something next to another plan's cost.
The value you filter on appears nowhere in that formula. Ask for 'chargeback' with the index paths switched off and the Seq Scan still costs 596. Only the index path's price moves with the value, so the planner is choosing between a fixed price and a sliding one.
For 'paid', every one of the 221 pages holds paid rows, so the index path would read the whole heap plus the index. Force it with enable_seqscan = off and you get a Bitmap Heap Scan at cost 872.
How many pages each status sits on
Selectivity is the share of rows that pass the filter. The usual rule of thumb goes: one row, use the index; 1%, usually; 20%, it depends; 70%, Seq Scan. Here is the same table counted by pages as well as by rows:
SELECT status,
count(*) AS rows,
count(DISTINCT (ctid::text::point)[0]) AS pages
FROM orders
GROUP BY status
ORDER BY rows DESC;
The rule of thumb says index failed, and Postgres agrees: Index Scan at cost 235 against 596. Run both with EXPLAIN (ANALYZE, BUFFERS), though, and the index path touches 209 buffers to the Seq Scan's 221, a saving of twelve page reads.
It still finished faster, 0.19 ms versus 0.93, because checking 270 rows is less CPU work than filtering 30,000. For 'chargeback', 30 rows on 30 pages, the index path needs 32 buffers instead of 221.
Timing it on 5 million rows in RAM
On 221 pages everything finishes in a couple of milliseconds, so for timings I built a bigger table: 5 million rows, 326 MB, about 120 rows per page.
A bucket column holds a random number from 0 to 999,999, so WHERE bucket < 10000 matches 1% of the rows. I loaded the same rows twice, once in random order and once sorted by bucket, and forced each plan with the enable_* settings at shares from 0.01% to 50%. shared_buffers was 2 GB, which kept the whole table in memory.
For scattered rows Postgres picked a bitmap scan, which collects all the TIDs first and then reads each heap page once, in page order. Up to 10% of the rows it beat the Seq Scan, 98 ms to 124. At 25% it lost, 174 against 139, and the planner still chose it. A plain Index Scan, one heap visit per row, lost from 5% on.
The same queries with a cold cache
RAM is the friendly case. For the cold runs I restarted Postgres and dropped the OS page cache before every run, so every page came off the VM's disk.
The Seq Scan got about twice as slow, 250 to 350 ms at every share. The clustered index won everywhere again, 23 ms versus 263 at 5%.
The scattered bitmap scan took 748 ms at 0.5% of rows, to the Seq Scan's 349. At 1% it took 1.27 seconds against 255 ms, five times slower, and that's the plan Postgres picks.
Seq Scans on a 200-row table
A 200-row copy shows the other end of the scale, where the table is too small for any index to win.
A partial index instead of enable_seqscan
The one place for enable_seqscan = off is a scratch session, where it shows you what the other plan would cost. It applies to every query in the session, so in app code it also overrides the plans that were right, like 596 against 872 for 'paid'.
If the queries you care about only ask for the rare values, say a failed-payments dashboard or a chargeback queue, index only those rows:
CREATE INDEX orders_status_not_paid_idx ON orders (status)
WHERE status <> 'paid';
The planner's statistics in pg_stats
Before adding an index, look at the column in pg_stats:
SELECT null_frac, n_distinct, most_common_vals, most_common_freqs
FROM pg_stats
WHERE schemaname = 'public' AND tablename = 'orders' AND attname = 'status';
That's what the planner believes about status as of the last ANALYZE. If most of your queries filter on paid, at 0.9, they get a Seq Scan, and no extra index will change that.
Run it on your own server
All the numbers above come from one 4-vCPU AMD EPYC cloud VM with 8 GB of RAM, so yours will differ. The SQL, the scripts, the raw output and the chart code are on GitHub. Run them on your own server, and if you get a different picture or think I got something wrong, open an issue or a PR.
Design is a Search Problem
Designers should treat their craft as a search problem, where AI provides a wider 'beam' of options but human taste remains the critical heuristic.
Summary
Deep Dive
- Beam Search: A heuristic search algorithm that explores a graph by expanding the most promising nodes in a limited set.
- A Search*: A path-finding algorithm that uses both actual cost from the start and an estimated cost to the goal, making it more efficient and deliberate than simpler searches.
- Heuristic: A practical strategy for finding a solution faster when an exhaustive search is impossible.
- State Space: The set of all possible configurations or options in a given problem.
Original Article
A decade ago, working at a design agency, I lived in fear of one sentence: “this looks nice, can we see a few other options?” I was not an ‘options’ guy. I fashioned myself as Paul Rand, telling Steve Jobs off when he asked for multiple options: “No. I will solve your problem for you and you will pay me. You don’t have to use the solution. If you want options, go talk to other people.” But I didn’t have the guts to tell my clients off, so I dutifully designed a few other options.
Recently, I’ve gotten to work with some of the best designers in the field. Looking at their digital workspace — in Figma, Paper, Claude Design, or something else — I see hundreds of artboards and variants, laid out in a sprawling grid, representing branching parallel universes of design directions. Sometimes I see little panels filled with dials to sweep through the possibilities, enabling a single person to see thousands of combinations in the hope of picking a single, perfect specimen. Lots and lots of options.
This makes me feel like an alien. My files are just a few artboards; sure, I tweak and tune the tiny details inside, but I rarely keep the multiverse in view as I do. I’ll give myself a dial here and there, but usually for a single variable that I can’t easily visualize. Am I just a bad designer?
Assuming (hoping) the answer is ‘no’, I want to revisit a thread I pull on from time to time: as designers, we’re all just trying to find the right solution in a vast multiverse of possibilities. The solution exists, and we typically know it when we see it. So most of what we call ‘design’ is groping around a dark room looking for a black cat. Different designers take different approaches. Some are changing their approaches with the advent of generative AI, but at the end of the day, design is a search problem.
Computer science has worked on search problems for decades. Depending on the particularities of a given search, there are a variety of algorithms and strategies that can result in faster, more direct, or more computationally efficient paths to the solution. We can apply this wealth of research from some of my favorite scientists (looking at you, Claude Shannon) to the design process, to hone our own craft.
‘Search’ has its own formal vocabulary in computer science. I’ve put a glossary at the bottom of the post, and I’ll use dotted underline to link to a definition if you need it.
To picture how a search works, imagine a Rubik’s cube. A scrambled cube is the starting state; each turn you can make is a state change, leading to an adjacent state. All the possible combinations of colors on the faces (43 quintillion for a typical 3×3 Rubik’s cube) make up the search space. Searching involves evaluating candidates that can be reached from the current state of the cube; the desired state is the solved cube with each face a solid color.
Now, a few basic search algorithms and how they apply to design:
Depth-first search
As far as search algorithms go, depth-first search (DFS) is as simple as it gets. Here’s how it works:
- From the current state, look for adjacent states we haven’t visited.
- If there is one, move to it. It doesn’t matter which.
- If there isn’t, go back to the previous state and start from step 1.
- Repeat until we reach the goal or run out of states.
In a design sense, this is akin to navigating a hedge maze by picking a random direction at each fork until you hit a dead end. It’s not a very good design process, but it’s a good intro to how these things work.
Breadth-first search
Breadth-first search (BFS) is just a touch more complicated, and starts looking more like the design process.
- From the starting state, visit every adjacent state.
- From each of those, visit every adjacent state we haven’t visited.
- Keep going, moving one step further from the start each round.
- Repeat until we reach the goal or run out of states.
Both DFS and BFS are naive algorithms; we don’t proactively evaluate candidates while we search. Both are used on search problems with small search spaces and/or when evaluation is cheap.
But many of the most interesting search problems aren’t small or cheap. Take chess, for example; winning at chess is a search problem. But the search space is unfathomably large, which is why it wasn’t until the mid-2000s that computers could reliably beat grandmasters.
Most product design problems I’ve worked on look more like chess than a Rubik’s cube. Design covers search spaces of hundreds of dimensions — sizing, spacing, hierarchy, color, motion, interactivity, responsiveness, etc. And evaluation is relatively expensive: beyond changing a button or color, we often need to ship something before we truly know it works.
There are other algorithms meant for this kind of problem, and they resemble design more and more as they get more complex.
Beam search
Beam search is the first approach we’ll explore that actually evaluates potential candidates before moving.
- From the starting state, score every adjacent state.
- Keep the three best and discard the rest.
- Score every state adjacent to the three we kept, and again keep only the three best.
- Repeat until one of the states we kept is the goal, or we run out of states.
This is called ‘beam search’ because it’s like using a flashlight’s beam to look at new states; the width of the ‘beam’ above is 3. Beam search is the ‘show me some options’ approach that most of my agency work took. It’s limited by the width of the beam: a narrower beam is faster and cheaper, but may miss the best solution at a given step.
Now is an important moment to introduce the idea of a heuristic. In most cases, we don’t know we have the ideal candidate until we find it. Given that inescapable fact, we have to pick a test that we think will separate the good candidates from the bad ones. Heuristics usually involve tradeoffs: they can be fast/cheap and error-prone, or slow/expensive and accurate.
Here’s the problem: beam search often fails to find the best solution! It depends entirely on the width of your beam and the quality of your heuristic. For example, if the desired state is hiding behind a low-scoring upstream branch, the algorithm might never find it. This can be overcome with an infinite beam, but then we’re just back at a slightly fancier BFS, which is slow and inefficient.
When I learned the shortcomings of beam search — that simply checking a few options and picking the best one might not lead to the best solution — my feelings about my option-hungry clients were validated.
And yet, designers are still using the beam, because AI has given us a better flashlight: producing a handful of options for a SaaS website used to take a week, and now it takes seconds. The beam has gotten so wide that ‘give me 10 options’ feels like an effective way to prompt AI towards the best design.
Still, it’s not how I design. And again assuming I’m not just in the permanent design underclass, I thought maybe there was another algorithm that explains why I tend to explore so few branching paths.
A*
A* (A-star) is the algorithm behind most route-finding, from maps apps to video game characters.
- From the starting state, give every adjacent state a score, made up of:
- The number of changes it took to get there from the start, plus
- The heuristic: our estimate of how many changes remain to reach the goal.
- Put them all on a list of states we could visit next.
- Move to the state on the list with the lowest score. This might not be adjacent to our current state! Score the new state’s adjacent states and add them to the list.
- Repeat until we’re at the goal.
This is similar to beam search, but with a few twists:
- A*’s heuristic incorporates a notion of distance from the goal.
- A* doesn’t always move smoothly from adjacent state to adjacent state.
That last bit is crucial: unlike beam search, A* can go backwards. If the path we’re on moves far from the goal, its score climbs. Eventually a previous candidate looks better and we go back to it.
More importantly, how much exploring we do in A* depends almost entirely on the quality of our heuristic. With a perfect estimate, A* goes straight to the solution.
A* matches how I design. I’m more deliberate with exploration, only following thoughts until I feel they’re drifting away from the solution. I typically only keep old explorations around if they’re promising; an exhaustive record of every possible variant is stressful.
I’ve managed to go the whole essay without mentioning it, but with my A* approach, taste does in fact matter; while there are many designers with better taste than I have, what I do have is hard-won by years of designing, and I can use it to sniff out the fastest path to the best solution.
Conclusion
AI has given us a wider beam, but it hasn’t yet given us a better heuristic. We’re still designing for humans, so we still have a leg up on the robots: we can use empathy, or even our own biases and human shortcomings, to feel out the right solution.
Meanwhile, understanding the design process is just as important as ever. This kind of introspection — learning about search algorithms out of despair over my own shortcomings — has added a few tools to my arsenal. There are many variations on these algorithms that may describe your own search process; there are even more exotic variants that may be worth trying on.
It might seem reductive to call design a search problem. Where’s the creativity? But with humanity, humility, and curiosity, pursuing the optimal solution with a well-tuned algorithm can help us design faster and more efficiently.
Definitions
- Adjacent
- Two states are adjacent when they’re separated by a single state change.
- Candidate
- A state we’re considering.
- Distance
- The number of changes that turn one state into another. Usually we measure distance as the least number of changes required to get from one state to another.
- Heuristic
- A test we think will separate the good candidates from the bad ones, used because we usually can’t know we have the ideal candidate until we find it. Heuristics involve tradeoffs: they can be fast and cheap but error-prone, or slow and expensive but accurate.
- Search
- The process of arriving at a desired state (scoring well on a fixed set of criteria) by evaluating a set of candidates.
- Search space
- The set of possible states. We call it a ‘space’ because often there are ‘dimensions’ to search, like the width and height of a shape, and we can make analogies to the dimensions of a physical space. Sometimes these dimensions are literal, as when a wayfinding app picks the best route from point A to point B.
- State
- A possible step in the movement from the beginning of a search to the end. We start with the ‘initial state’; as we evaluate candidates we might change our ‘current state’; the end of the process is the ‘final state’ (and may not be the ‘desired state’).
- State change
- Also called a mutation. How the system moves between states.
Build Faster Without Leaving Accessibility Behind
Microsoft’s open-source Accessibility Considerations Toolkit provides a structured approach for dev teams to identify accessibility barriers during AI-assisted development.
Summary
Decoder
- Accessibility Considerations Toolkit (ACT): An open-source set of heuristics and standards references designed to help teams audit interfaces for accessibility barriers.
Original Article
When accessibility questions come up, not every team has someone who can help answer them. Without that resource, teams can miss how their decisions affect people with disabilities—and leave barriers in place until someone encounters them.
AI-assisted development makes that gap more pressing. If the tools helping us build software don’t account for how people with disabilities use technology, they can repeat those mistakes across more interfaces, faster. We need to bring that understanding into the work while decisions are still being made.
Today, my team is open sourcing the Accessibility Considerations Toolkit (ACT) to help close that gap.
Start with how people use things
ACT brings together guidance on access strategies, assistive technologies, devices, and the contexts in which people are trying to get things done. It gives designers, developers, researchers, and evaluators a starting point for investigating barriers in an actual interface.
The emphasis on actual is important. Guidance alone can’t tell you whether something works. You need to examine the interface, consider how someone might interact with it, and gather evidence.
Consider someone filling out a form. They might increase browser zoom, pan around a magnified viewport, or use a high-contrast theme. Each approach can expose different barriers:
- When someone increases browser zoom, does the form reflow so they can still reach the Save button, or does it end up hidden behind a fixed toolbar?
- When someone using screen magnification selects Save, does the confirmation appear where they’re looking, or in another part of the screen they can’t see?
- In a high-contrast theme, can someone tell which fields need correcting, or do the error indicators disappear?
The answers to these questions help us understand whether someone can use what we’ve built under the conditions in which they need to use it. ACT helps teams ask those questions and investigate the answers.
Better questions, earlier
The toolkit combines access considerations and interaction patterns with usability heuristics, evaluation methodology, and references to accessibility standards. People can read the Markdown directly; agents and evaluation pipelines can use the generated YAML.
We’re integrating this guidance into our own AI-assisted design, development, and testing workflows. Our goal is to inform decisions while there’s still an opportunity to change them.
But let’s be clear about the limits of this guidance:
- It doesn’t replace co-design or testing with people with disabilities.
- It doesn’t establish compliance with accessibility standards.
- Potential issues still need human review to determine whether the evidence supports a finding.
An evaluation that doesn’t find issues hasn’t proved the experience is accessible. It may just not have found the barriers.
Help us find what’s missing
We’re open-sourcing this resource so its guidance can be examined and improved. We welcome your feedback—including questions about its assumptions and examples of where it falls short.
If you’re a person with a disability, an accessibility practitioner, or both, I’d especially welcome your perspective.
Explore the Accessibility Considerations Toolkit and try bringing it into your own design, development, or evaluation process. Does it help you ask better questions? Uncover a barrier you might have missed? Where does it fall short? Please share what you learn so we can improve the guidance together.
Our goal isn’t just to get teams using the toolkit—it’s to help them build things more people can use.
Band-Aid Accessibility
Disabled users are increasingly relying on LLM-powered coding tools to patch inaccessible software that mainstream companies continue to ignore.
Summary
Deep Dive
- Disabled users are turning to LLMs to build accessible interfaces that mainstream software lacks.
- Creating these workarounds leads to 'band-aid' accessibility that may not be sustainable long-term.
- There is a risk that companies will use the existence of these community-led patches to excuse their own failure to design inclusive products.
- The practice creates parallel, disconnected software spaces rather than integrating accessibility into standard development workflows.
- The author argues that true accessibility requires hiring specialists and integrating inclusive design from the start, rather than relying on LLM-generated patches.
Decoder
- Screen reader: Assistive technology software that reads out the content of a computer screen for visually impaired users.
- Accessibility-wash: The practice of superficial or performative efforts toward accessibility to improve corporate image without making substantive changes.
- WCAG: Web Content Accessibility Guidelines, the industry standard set of recommendations for making web content more accessible.
Original Article
More and more disabled people take matters into their own hand and use LLM coding tools to generate accessible features or wholly accessible app for their use cases. While it seems easy to celebrate this as an act of liberation for these users, reality is more complicated.
Preamble
I support everyone to get the access needed, even if this is via dubious, unethical, or illegal ways. I can't fault someone with chronic pain who finds relief in cannabis. I won't judge anyone who needs to find "alternative ways" to get books that work for them and their disabilities, and who cannot read DRM-laden books. And I don't find fault with anyone who removes barriers using LLM coding tools.
This is an absolute stance.
Why this happens
There is a lot of debate, mainly on Mastodon, about the ethics of creating(?) accessible apps this way. And a lot of these arguments, I can relate to:
- Disabled people have always been ignored by most mainstream software developers and companies.
- Years of begging to make interfaces accessible, particularly to screen readers and for their users, lead to nothing.
- This is a way users can ensure access in a timely way.
- Screen reader users know better how interfaces work for them and can "prompt" to meet these requirements.
- The projects might not be sustainable in the long run.
- Not using mainstream software creates parallel spaces for disabled people.
- Software developers might do less accessibility, because disabled people can just patch it after the fact.
- Commercial LLM vendors use these projects to accessibility-wash their ethical problems.
The state of accessibility is dire. I review dozens of projects every year, and the quality has rarely been as low – with notable exceptions! – than in 2026. And this doesn't even cover only accessibility issues: It looks like most software is not tested at all, be it for functionality or error handling, or just stability. Everything breaks all the time.
And what is an inconvenience to some users is a barrier to others.
Conclusion?
As a LLM skeptic, I see a lot of the risks outlined above myself. It would never be for me to use an LLM to create something. I enjoy the creating and the problem solving too much, and honestly as an accessibility reviewer, you often get the "identifying problems" but hardly ever the "problem solving" side of the coin. It can be very frustrating.
But this is for many screen reader users the first time they feel that they have tools they can use to solve accessibility problems by themselves for themselves. I always wondered why there are not more disabled, and especially blind, programmers around, and I assume the reasons are ableist structures around computer education.
But even then, letting a blind developer lead your website or app UI development? Probably unimaginable in most companies. UI means painting it nicely, right?
My real worry is about the long-term sustainability of this approach. Creating separate paths often means features get out of sync. Rising cost of LLM code generators will also limit what can be done on a budget. Which also makes these kinds of developments, as accessible as the result may be, burden with a disability tax. And that might even increase once companies de-prioritize accessibility work because supposedly the community will take care of it.
What I hope this does is for disabled people to be interested in coding more, but we also need the other side to go and do the work: It can't be that accessibility issues are pushed later for years and years and years.
No doubt, the community is creating some extremely good band-aids, but it is embarrassing that they are even needed.
If you want to help to remove the reliance on LLM coding tools by disabled people, you have to make your stuff accessible, and listen to their feedback. Or even better: Hire that blind UI specialist.
ChatGPT Wallet
OpenAI is introducing ChatGPT Wallet, a secure mechanism for AI agents to execute purchases without exposing raw credit card data.
Summary
Original Article
OpenAI introduced ChatGPT Wallet, allowing AI agents to make purchases without accessing users' actual card details while requiring a security check for every transaction.
Qwen4-Max: Your New Teammate Has Arrived
Qwen4-Max debuts as a high-end reasoning model with integrated support for game engine workflows and 3D modeling pipelines.
Summary
Original Article
Qwen4-Max can take on a wide range of creative and professional work in the fields of 3D modeling, game development, precision manufacturing, finance, law, healthcare, and frontier scientific discovery. It demonstrates strong 3D creation capabilities and can use Blender to carry out geometry modeling, scene assembly, material setup, lighting, and rendering. The model works fluently with leading game engines and can handle scene construction, gameplay logic, integration of external assets, material setup, HUD design, and visual effects, and deliver a playable game. Smaller models in the family are coming to open source.
GPT-6 Astra Tackles a Fundamental Physics Problem
Physicist Nathaniel Craig utilized OpenAI’s GPT-6 Astra to solve a complex lattice formulation challenge within the Standard Model of physics.
Summary
Deep Dive
- Addressing the challenge of defining field theories without perturbative approximations.
- Astra assisted in constructing a candidate lattice formulation.
- The work focused on ensuring mathematical consistency in defining the Standard Model's parameters.
Decoder
- Lattice Formulation: A method in physics where space-time is treated as a discrete grid to make field theory calculations computationally feasible.
- Perturbative Approximation: A method of solving problems by starting with a simpler version and adding small, series-based corrections, which often fails in high-energy physics.
Original Article
Physicist Nathaniel Craig worked with GPT-6 Astra to develop a candidate lattice formulation of the Standard Model, addressing a longstanding challenge in defining the theory without relying on perturbative approximations. Astra helped construct the mathematical framework and develop a detailed proof supporting the proposed approach.
Petition Calls for Nvidia to Reinstate Jensen Huang
Nvidia shareholders and executives, including Elon Musk, are petitioning the board to reinstate former CEO Jensen Huang following his rapid ouster.
Summary
Original Article
Elon Musk, other tech execs sign petition calling on Nvidia to reinstate Jensen Huang
A petition calling on Nvidia to reinstate the company’s recently ousted CEO Jensen Huang appears to have attracted more than 115,000 signatures as of Saturday afternoon.
The petition website said the signers include 5,965 current Nvidia employees, 1,108 former employees, and 63,063 shareholders. Notable supporters include Tesla and SpaceX CEO Elon Musk, Replit CEO Amjad Masad, Epic Games CEO Tim Sweeney, and Y Combinator CEO Garry Tan.
The petition states that its signers are “concerned that Jensen was removed without meaningful due process” and demands that the board “bring him back while an independent investigation establishes the facts.” It also calls on the board to ensure a fair investigation and provide more information to the public.
“Jensen deserves due process, not mob justice,” the petition says.
Huang resigned on Thursday, four days after confessing to having an affair with a chip executive suspected by U.S. intelligence to be a Chinese spy. The former CEO has agreed to testify before Congress, while news reports have also suggested that Huang engaged in multiple extramarital affairs.
Nvidia appointed CFO Colette Kress as the company’s interim CEO while it searches for a permanent replacement. Nvidia’s stock price fell sharply through the week, while Huang’s departure is also expected to leave a big gap in AI industry funding.
Tech world's calls for Jensen Huang to return to Nvidia are growing
An online petition backed by Elon Musk and other prominent executives is lobbying for Jensen Huang's return as Nvidia CEO despite ongoing investigations.
Summary
Original Article
- An online petition calling for Huang's reinstatement has drawn nearly 180,000 signatures, with Elon Musk among prominent tech executives backing the campaign.
- Huang has discussed a potential return as Nvidia CEO with close confidants, the Financial Times reported.
- Major shareholders Altimeter Capital and Atreides Management have urged Nvidia's board to consider all leadership options, including Huang's potential return.
An online petition calling for Jensen Huang's reinstatement as Nvidia CEO has drawn nearly 180,000 signatures, as prominent tech executives, employees and shareholders rally behind him following his resignation last week.
The "Stand with Jensen" campaign has attracted support from Elon Musk, Y Combinator CEO Garry Tan and Epic Games CEO Tim Sweeney.
It comes as the Financial Times reported that Huang has discussed the possibility of returning as Nvidia's chief executive with close confidants, citing multiple people familiar with the matter.
"A petition is underway to reinstate Jensen and I'm signing it too," Musk posted Friday on X. Tan also posted he'd signed, saying: "Jensen deserves due process, not whatever is happening right now." Sweeney posted "save Jensen!" on Sunday as he shared the petition.
Other high-profile signatories include Shopify CEO Tobias Lütke and former Uber CEO Travis Kalanick, who both confirmed their signatures on X. Hugging Face CEO Julien Chaumond is also listed as having signed.
CNBC found dozens of signatories identifying themselves as current and former Nvidia employees, though the website notes that affiliations are self-reported and not verified.
Huang, who co-founded the company in 1993 and helped turn it into a dominant supplier of artificial intelligence chips, resigned as Nvidia's president, CEO and board member following reports of extramarital affairs — including one involving a woman suspected of ties to Chinese intelligence.
The growing campaign for his reinstatement comes as the board seeks internal and external candidates to succeed him, and longtime finance chief Colette Kress serves as interim CEO.
Any potential return, however, faces significant hurdles. Nvidia's board is continuing an independent investigation into the conduct that led to Huang's departure.
U.S. lawmakers are preparing to investigate how Huang's affair may have damaged America's artificial intelligence dominance, the national security implications and his close relationship with President Donald Trump and his Cabinet.
Speaking to reporters before departing for a series of summits in Asia Monday, Trump said he "would not mind" if Huang returned, adding that he thought the former CEO had learned a "big lesson."
Silicon Valley's view
The petition calls on Nvidia's board to ensure a fair investigation into Huang's conduct and provide greater transparency around the circumstances of his departure.
Some Nvidia employees have also publicly expressed support for Huang, with one engineer saying he had decided to leave the company following the CEO's departure.
Darrell Dai, who identified himself as a robotics engineer at Nvidia, said Huang's vision and the engineering culture he fostered were key reasons he had stayed at the company.
Vartika Singh, who said she had worked at the chipmaker for nearly eight years, praised Huang's leadership and personal support during a family medical crisis.
"I have never worked for anyone quite like Jensen Huang, and I doubt I ever will," Singh wrote on X, describing his commitment to employees, researchers and startups.
The push has also gained traction among major shareholders, with Altimeter Capital CEO Brad Gerstner and Atreides Management's Gavin Baker urging Nvidia's board to consider all options, including Huang's potential return, in an open letter Friday.
The investors called for a measured approach to choosing Nvidia's next leader, arguing that Huang deserved a fair process before the board made any permanent decision.
Neil Shah, vice president of research at Counterpoint Research, told CNBC's "Squawk Box Europe" on Monday that a potential Huang comeback would be "good for the stock."
"Jensen's rhythm has made this AI ecosystem what it is right now," Shah told CNBC's Karen Tso and Steve Sedgwick. "The trillion-dollar capex, which has been spent every year, has to continue to make this train [keep] going."
Shah said his only concern is whether Nvidia would impose new guardrails on Huang, preventing him from leading the business with the same level of energy and enthusiasm. "If he cannot work freely and actually take the ecosystem on the same breadth and speed, will he be that effective? That is the big question."
Reinstatement chances
One of Huang's allies told the FT that supporters were assembling a "war room" of Silicon Valley executives to lobby for his reinstatement and would seek backing from the White House.
The efforts had Huang's support, the person added, according to the report.
"We are in the process of selecting a search firm for the NVIDIA CEO role. We have not formally engaged any potential candidates," an Nvidia spokesperson said in response to CNBC's request for comment on the company's search.
The spokesperson added that any speculation about individual candidates, whether internal or external, was premature.
Anthony Scaramucci, founder of alternative investment firm SkyBridge and a former White House communications director, also dismissed the prospect of Huang returning as CEO, saying the investigations surrounding him presented obstacles that a public campaign could not overcome.
"The tech bros want Jensen back. Not happening," Scaramucci wrote on X, arguing that Nvidia's board could not reinstate a chief executive whose conduct remained under investigation.
In a separate post, Scaramucci rejected comparisons with OpenAI CEO Sam Altman, who returned to the company days after being ousted in 2023. "The FBI doesn't take petitions," he wrote, saying that Huang's situation was fundamentally different from Altman's. Altman's ouster stemmed from an internal governance dispute, while Huang's situation involves ongoing investigations into his conduct that make a swift board reversal far more difficult.
U.S. Treasury Secretary Scott Bessent said Sunday that Nvidia needs to look outside the company for its next executive.
"I want to keep coming back to what was the culture at Nvidia, and were they pressing these seemingly illicit sales to goose their revenues and their earnings?" he said in an interview with Fox News.
Megaspeed, a Singapore-based company linked to the controversy, said recent media reports contained assertions about its business and supply chain that were not supported by verifiable evidence.
The company said it had retained external legal counsel to conduct an internal investigation and that Alice Huang, a woman reportedly romantically linked to Jensen Huang and an alleged Chinese spy, left Megaspeed effective Aug. 30, 2025, and no longer holds any role there.
Alice Huang may have owned stakes in businesses in China that benefited Nvidia's biggest competitor in China, Huawei, CNBC reported earlier.
Meanwhile, Nvidia's board has approached Arm CEO Rene Haas as a potential successor to Huang, the FT reported. Nvidia's board said at the time that it had not formally engaged any candidates.
Some analysts see a path for Huang to return despite the controversy surrounding his departure.
Jay Goldberg, a senior analyst at Seaport Research Partners, said Huang could make a comeback if he overcomes his legal challenges, citing his influence in the AI industry and continued support from investors.
"If he can get through the legal trouble, I think we'll have a second act from him," Goldberg told CNBC's "Squawk Box Asia."
But not everyone on Wall Street is convinced it can happen so soon.
"We see no chance Jensen comes back to Nvidia as CEO," Dan Ives, partner and senior managing director at Yorkville Ives & Co., said in a note.
Ives said the shareholder petition and public pressure could create further distractions for Nvidia, but would not change the central task facing interim CEO Kress: restoring stability and investor confidence.
Ives added that Nvidia's leadership in AI chips and its long-term demand outlook remained intact despite the management upheaval.
Nvidia is scheduled to report its fiscal third-quarter earnings Tuesday, putting Kress and the company's leadership team under the spotlight as investors look for reassurance about its future.
SpaceX Catches Starship in Midair at Texas Site After Space Trip
SpaceX successfully captured a Starship booster using mechanical launch tower arms, marking a major milestone in reusable launch vehicle technology.
Summary
Original Article
SpaceX successfully caught Starship using the vehicle's launch tower's mechanical arms on Saturday afternoon. The midair capture serves as a major proof of concept for Starship. SpaceX has now captured both sections of the launch system during different missions. It aborted an attempt to catch Super Heavy during the recent flight due to a slight underperformance of the engines.
China Fires Researchers After Child Dies in Gene-Editing Trial
Chinese researchers were dismissed after failing to report a child's death during a gene-editing clinical trial and suppressing critical warning signs from primate studies.
Summary
Original Article
The researchers failed to flag warning signals from a non-human-primate study and didn't disclose the child's death in the published research paper.
AI and hiring: bad for employees, bad for employers
Generative AI is breaking public-facing processes like hiring and government complaint handling by making it trivial to generate cogent but potentially false submissions.
Summary
Decoder
- Hallucination: The phenomenon where an AI generates text that sounds plausible and authoritative but is factually incorrect or nonsensical.
Original Article
AI and hiring: bad for employees, bad for employers
AI breaks things - and there is no obvious way to put them back together
If you talk to young people you will find how AI is disrupting the job market - especially at graduate entry level. Many are (justifiably) distressed about their prospects.
I think we can now safely say that AI is bad for entry-level employees in most intellectual activities.
I want to talk about this from the other side - how AI is disrupting the employer’s prospects of getting good staff.
At Bronte (the financial firm I founded with friends) we hire left-field candidates. We do not seek the usual financial market credentialism (I have always regarded a CFA as a mild negative on a CV).
We sought raw intelligence and curiosity and that was hard to screen for by reading CVs.
We had a policy explained to applicants. We interviewed everyone who sent us a CV in any way personalised to us and with any reasonable claim to a position.
We sent applicants this form letter:
We interview far more widely than most firms - and we interview a large number of people.
The problem is that if you interview more people you also have to interview quickly. We eliminate quickly.
This means that most candidates will get a five-minute interview. We can appear brusque when we cut the interview short - and we are sorry about that - but it is from our perspective the necessary part of interviewing quickly.
There are three reasons this interview might last more than ten minutes.
a. We are genuinely interested in hiring you
b. We are learning something - that is by listening to you talking about what you are doing we might get better at what we are doing. We may still be interested in hiring you.
c. You are a good-looking contra: you are pitching us a long and we are salivating about it as a short for instance. We can make money off misconceptions you bring to us.
All three of these have happened - and at the end of a longer interview we will discuss with you which category you fit in.
If it is c (as above) we would appreciate a follow-up chat in a few years - and one of us will be wrong. If it is us that were wrong we want to improve by learning what mistakes we may have arrogantly made. If it is you - you should follow up too - and see what you can learn.
It can very easily be either of us that is wrong.
I know that c (above) looks exploitative - we are essentially leading you on. It is not totally exploitative. We may still be interested in hiring you two years later when you have made us look like fools.
Having run a couple of hundred such interviews over the years we have hired a handful, and found a half dozen more jobs at other firms for which they were better fits. [Nobody in either camp has reason to complain.]
We have also found a dozen or so shorts (and noted which firms the bozo idiots who pitched us frauds wound up at).
We have however been brusque a few times - and many applicants have found the quick rejection difficult (and some have told me so).
Alas none of this works any more. We used to run 40-60 job interviews a year - and 5-10 minutes once a week was a tolerable use of my time. Especially if I learned something.
Generative AI now means that I get five times as many applicants. And I can’t run 300 job interviews a year or I would do nothing.
So I have to cull. And I do.
I could get AI to select which of the hundreds of applicants to choose - but AI picking between AI generated applications is worse at selection than a human picking through human generated ones.
Now having made what is probably a bad cull I get applicants on the phone for a five minute chat. And sometimes I get really good answers to hard questions. Nuanced and clever.
I have thought some applicants were astonishing on the phone. And then I got them in for what might normally be three or four hours of chatting. And they were decidedly ordinary - and should never have got through the first cut.
We think they had AI help on the phone.
From our perspective this is a mess.
The problem is widespread
I was chatting to a senior person at a big consulting firm about this. This was a firm that did more than a bit of credentialism (hiring heavily from target universities like Harvard).
They used to get 3-5000 applicants each year. They would winnow this down to 300 and then give them an extensive workshop style interview with case studies and the like. From that they might choose 80 to hire.
They had a really big problem - which is that they could not work out how to winnow down from 3000 to 300. The applicants all looked good - but they also looked similar. Their process has broken too.
AI is going to break many things
I was chatting to friends in Canberra (the political capital of Australia) about this. They told me that AI is breaking the Ombudsman’s office.
The Ombudsman is a government agency where you can - at first instance - send complaints about the behaviour or decisions of other Government departments. This often works - in Australia. The Ombudsman investigates and what is really a simple problem gets resolved quickly and relatively cheaply.
The Ombudsman has always had more complaints than they can investigate thoroughly. And they had a solution. They would triage to prioritise anyone who could write a 3-5 page crystal clear explanation of their issue. If the applicant could explain the problem clearly then the problem was probably real and worth investigating.
This worked. Simply writing out the problem clearly removed the (many) crackpots who sent complaints.
Alas this no longer works. Any AI can make a complaint sound cogent (even when it is not). It can hallucinate cogency (especially when prompted to do so). The Ombudsman’s office is rapidly breaking.
This is not good. I think democracy works better if you have a rapid, cheap and effective way of complaining about government actions. And one of those is being removed.
A general rule to what breaks
Anywhere where you seek input from the public, whether it be a job application process, or an Ombudsman’s complaint process, or just social media will have a tendency to break.
And I am not sure how we solve this problem.
I have toyed with requiring job applicants to hand-write a one page cover note. But that is a solution out of desperation.
I am hoping people have something better.
Thoughts?
Airgorah (GitHub Repo)
Airgorah is a new, Rust-based WiFi auditing tool that simplifies traffic capture and password cracking via a GTK4 interface on Linux.
Summary
Deep Dive
- Requires a wireless network adapter capable of monitor mode and packet injection.
- Separates UI (unprivileged) from hardware interaction (privileged agent).
- Targeted at network security testing and auditing.
- Uses Rust for memory safety in low-level packet processing.
Decoder
- Deauthentication Attack: A type of denial-of-service attack where a client is disconnected from an access point by forcing the handshake to reset.
- Monitor Mode: A mode for wireless cards that allows them to capture all traffic on a channel, not just traffic intended for the device.
- PMKID: A identifier used in WPA3/WPA2-PSK handshake capture for offline cracking.
Original Article
Airgorah
A WiFi security auditing software
Airgorah is a WiFi security auditing software that can capture nearby WiFi traffic, discover clients connected to access points, perform deauthentication attacks, capture handshakes and PMKIDs, and crack the password of access points.
It is written in Rust and uses GTK4 for the graphical part.
Legal
Airgorah is designed to be used in testing and discovering flaws in networks you are owner of. Performing attacks on WiFi networks you are not owner of is illegal in almost all countries. I am not responsible for whatever damage you may cause by using this software.
Requirements
This software only works on linux. The graphical interface runs as a normal user (so it works under both X11 and Wayland). When a privileged operation is needed, it launches a small privileged agent (airgorah-agent) via polkit, which will prompt for authentication once.
You will also need a wireless network card that supports monitor mode and packet injection.
Installation
The installation instructions are available here.
Usage
The documentation about the usage of the application is available here.
License
This project is released under MIT license.
Contributing
If you have any question about the usage of the application, do not hesitate to open a discussion.
If you want to report a bug or provide a feature, do not hesitate to open an issue or submit a pull request.
Uptime Kuma (GitHub Repo)
Uptime Kuma is a widely-used, self-hosted monitoring tool that provides a reactive dashboard for tracking HTTP, TCP, and Docker container health.
Summary
Deep Dive
- Supports multiple monitoring types including HTTP(s), TCP, DNS, and Docker containers.
- Uses Vue 3 and WebSockets for a high-performance, real-time UI.
- Allows for custom status pages mapped to specific domains.
- Features a reactive design that simplifies adding new services and managing incident alerts.
Original Article
Uptime Kuma
Uptime Kuma is an easy-to-use self-hosted monitoring tool.
Live Demo
Try it!
Demo Server (Location: Frankfurt - Germany): https://demo.kuma.pet/start-demo
It is a temporary live demo, all data will be deleted after 10 minutes.
Warning: Be careful - the top result in Google, uptimekuma.co, is not our official website, we don't control the website. I recommend following the installation guide in this GitHub repo.
Features
- Monitoring uptime for HTTP(s) / TCP / HTTP(s) Keyword / HTTP(s) Json Query / Websocket / Ping / DNS Record / Push / Steam Game Server / Docker Containers
- Fancy, Reactive, Fast UI/UX
- Notifications via Telegram, Discord, Gotify, Slack, Pushover, Email (SMTP), and 90+ notification services.
- 20-second intervals
- Multi Languages
- Multiple status pages
- Map status pages to specific domains
- Ping chart
- Certificate info
- Proxy support
- 2FA support
How to Install
Docker Compose
mkdir uptime-kuma
cd uptime-kuma
curl -o compose.yaml https://raw.githubusercontent.com/louislam/uptime-kuma/master/compose.yaml
docker compose up -d
Uptime Kuma is now running on all network interfaces (e.g. http://localhost:3001 or http://your-ip:3001).
Warning: File Systems like NFS (Network File System) are NOT supported. Please map to a local directory or volume.
Docker Command
docker run -d --restart=always -p 3001:3001 -v uptime-kuma:/app/data --name uptime-kuma louislam/uptime-kuma:2
Uptime Kuma is now running on all network interfaces (e.g. http://localhost:3001 or http://your-ip:3001).
Non-Docker
Requirements:
- Platform: Major Linux distros, Windows 10 (x64) or higher.
- Node.js >= 20.4
- Git
- pm2 - For running Uptime Kuma in the background
git clone https://github.com/louislam/uptime-kuma.git
cd uptime-kuma
npm run setup
# Recommended: Run in the background using PM2
npm install pm2 -g && pm2 install pm2-logrotate
pm2 start server/server.js --name uptime-kuma
Motivation
- I was looking for a self-hosted monitoring tool like "Uptime Robot", but it is hard to find a suitable one.
- Wanted to build a fancy UI.
- Learn Vue 3 and vite.js.
- Show the power of Bootstrap 5.
- Try to use WebSocket with SPA instead of a REST API.
- Deploy my first Docker image to Docker Hub.
Discussion / Ask for Help
Warning: For any general or technical questions, please don't send me an email, as I am unable to provide support in that manner.
I recommend using Google, GitHub Issues, or Uptime Kuma's subreddit for finding answers to your question.
Powering agentic cloud governance: What's new in Organization Policy
Google Cloud now lets developers use Gemini to write custom organization policies and interact with governance tools using natural language.
Summary
Deep Dive
- MCP server enables agents to perform posture audits, detect drift, and manage exemptions.
- Gemini can now generate policy constraints based on plain-language requirements.
- Includes functionality to generate compliant/non-compliant test configurations for dry-run validation.
- Supports services like GKE, Cloud Run, and Google Cloud Agent Runtime.
Decoder
- MCP: Model Context Protocol, an open standard that allows AI agents to interact with external tools and data sources safely.
- CEL: Common Expression Language, the syntax typically used for Google Cloud policy logic.
Original Article
Powering agentic cloud governance: What's new in Organization Policy
Even as sophisticated, new zero-day exploits continue to emerge, most cybersecurity incidents still start from the same common sources: an attacker locating and exploiting a fundamental, easily preventable cloud misconfiguration.
Organization Policy Service can help you secure resource configurations and set up security guardrails across your Google Cloud resource hierarchy. Using a catalog of over 100 predefined policies for many common cloud governance needs, as well as Custom Organization Policy, you can more easily set up boundaries with granular control over resources.
To help platform and security teams author, audit, and scale guardrails with greater speed and simplicity, we are introducing the general availability of our Organization Policy Model Context Protocol (MCP) server and natural-language constraint authoring powered by Gemini.
Managing Organization Policy with agents
Rather than forcing administrators to manually parse complex hierarchy chains, the Organization Policy MCP server offers tools that allow autonomous agents and pipelines to interact directly with Organization Policy.
Using the Organization Policy MCP server, agents can assist administrators with posture audits, drift detection, and exemption management. Key capabilities include:
- Constraint discovery: Tools like 'ListConstraints' and 'GetCustomConstraint' allow agents to understand the specific guardrails that apply to a project, folder, or organization.
- Policy evaluation: The 'GetEffectivePolicy' tool calculates the effective policy after considering hierarchy inheritance and tag-based conditions. You can use this to verify if a configuration change will be compliant.
- Manage policy: 'CreatePolicy' and 'CreateCustomConstraint' allow platform teams to manage guardrails via conversational workflows.
For example, you can combine your preferred coding agent with the Organization Policy MCP tools to create a custom dashboard to visualize the constraints applied to an organization and identify the folders and projects in the organization that have been exempted from the policy. You can also build custom agents and skills to streamline managing Organization Policy.
Simplifying custom constraint authoring with Gemini
While the Organization Policy constraint catalog provides built-in managed constraints for common use cases, each organization has its own unique architectural standards. Custom Organization Policy enables customers to express these standards as a custom constraint and enforce them across supported services.
Traditionally, administrators would write these rules in common expression language (CEL), which required a sharp understanding of Google Cloud resource schemas. Today, administrators can author custom Organization Policy in natural language through a Gemini prompt.
For example, you can prompt Gemini to require the use of agent-identity in Agent Runtime, to require that all agents deployed to Agent Runtime use an approved Agent Gateway, and to prevent users from disabling auto-upgrades on GKE, and refine custom constraints conversationally.
To help ensure safe deployments, Gemini can also generate compliant and non-compliant test resource configurations that can help you test policy and validate in dry-run mode before enforcing the constraint.
For targeted examples, the sample custom constraints library from Google Cloud Professional Services features pre-authored custom constraint samples based on CIS and PCI standards. Custom constraints are supported on a wide range of Google Cloud services including core AI and compute services including Agent Runtime, Agent Search, and Agent Studio on Gemini platform, Cloud Run, and Google Kubernetes Engine.
Build faster, with confidence
Organization Policy provides proactive cloud governance of resource configurations on Google Cloud. It lets administrators enforce critical security guardrails so engineering teams can build and deploy quickly and securely.
With a growing catalog of built-in guardrails, autonomous tools for administrators, and the ability to tailor custom constraints across Google Cloud services, Organization Policy gives enterprises the control they need to scale safely.
Explore the Organization Policy documentation to learn more about resource governance in Google Cloud and set up your foundational guardrails.
The Future of Data Science and Analytics Engineering at Klaviyo
Klaviyo is shifting data science roles toward human oversight of AI, as the speed of AI-generated code now exceeds human review capacity.
Summary
Original Article
AI can produce code and analysis faster than people can review it at Klaviyo, making human attention and clear ownership the main bottlenecks. Data science and analytics engineering are merging into a broader role focused on guiding AI, checking its work, and turning useful findings into action.
What thousands of conversations look like inside a language model
A visual mapping experiment suggests that LLM token states for different languages often form overlapping geometric regions in vector space.
Summary
Deep Dive
- Experimental Setup: Used 900 QA examples per language; processed via frozen base model Qwen3-1.7B.
- Findings: Global PCA axes showed Chinese more separated from English than European languages were.
- Controls: Transliterating Chinese to Pinyin showed that written form influences vector location, but not entirely.
- Methodology: PCA projections only capture a fraction (~17.6%) of the variance in a 3D view, highlighting the difficulty of visualizing high-dimensional neural states.
Decoder
- Residual Stream: The central shared workspace in a transformer architecture where vectors are refined by successive layers.
- PCA (Principal Component Analysis): A technique used to reduce the dimensionality of large datasets while retaining as much variance as possible.
- Embedding Table: The initial lookup table that maps tokens to numerical vectors.
Original Article
Full article content is not available for inline reading.
Five New Google Flow Tools Designed by Visual Artists
Google updated its Flow AI creative studio with five new tools built by artists to handle character consistency, retro styles, and cinematic camera angles.
Summary
Original Article
5 new Google Flow Tools designed by visual artists
Today, we're releasing five new tools that show how you can keep pushing the boundaries of visual storytelling in Google Flow, our AI creative studio. These five tools — all built by creatives — prove there’s increasingly no limit to what you can build in Google Flow. See how they can help you shape your next story, from concept to final cut.
1. Character Poster, by artist and filmmaker James Gerde
James Gerde treats his social feeds like curated galleries where visual consistency is everything. He built Character Poster to turn simple story concepts into polished, beautifully designed visuals. It helps creators maintain a signature style across their social stories — and can save hours of production work.
2. Character Arc, by filmmaker and visual storyteller Anurag Tiwari
Anurag Tiwari built Character Arc to show how a character changes over time. It lets storytellers visualize the same character across different ages and life stages without losing their core look, speeding up character development before full production begins.
For his documentary “Children of the Sun,” which features present-day interviews with subjects reflecting on events from 1974, Anurag is using Character Arc to trace the protagonist’s life from childhood through adulthood.
It helps me build his background while keeping him recognizable as the same person across different periods of the story.
3. Pixel Machine, also by Anurag Tiwari
Inspired by his love for retro digital aesthetics, Anurag also created Pixel Machine to give filmmakers and artists a playground that goes beyond polished AI visuals. It converts images into bitmap, ASCII, CRT, pixel art, and glitch styles, making it easier to design stylized title treatments, transitions, and music video visuals.
4. Lucid Glass, by artist and creative director Ezra Paulekas
Ezra Paulekas designed Lucid Glass to recreate the way light passes through glass, generating prisms, halos, refractions, and optical blooms. It works like an "impossible lens," giving photographers and filmmakers effects they could never capture with glass alone.
5. Pocket Studio, by visual development artist and filmmaker Ryan McCoy
Drawing on more than two decades in previsualization and visual effects for major motion pictures, Ryan McCoy built Pocket Studio to turn everyday phone footage into more polished shots. From a single take, the tool generates multiple camera angles and upgrades the visual fidelity, all while keeping the actor’s original performance intact.
Try these creative tools and build your own in Google Flow today.
Adobe's AI Experiments Let You Walk Around in Photos and Interview Your Characters
Adobe’s latest 'sneaks' feature AI that turns flat photos into 3D environments and allows users to interview AI characters to build their personalities.
Summary
Original Article
Adobe’s AI experiments let you walk around in photos and interview your characters
Here are some of the experimental ‘sneaks’ being showcased at Adobe Max.
Some of the most impressive tech that Adobe showcases at its annual Max conference each year is the experimental features it’s developing, and the latest of those tools are being announced today: they can transform images into explorable 3D scenes, design AI characters by talking to them, and turn photo editing into a copy/paste process.
The first (and weirdest, in my opinion) is Project Screen Test, which is designed to make AI-generated characters easier to develop and use for video content. Instead of just describing how the character should sound and behave in text prompts, the tool lets you talk to the character and direct them in an audition-like environment. You start by describing some base personality traits, such as “anxious, shy, awkward,” and giving the character a voice, either by selecting a preset, describing how it sounds in a prompt, or uploading sample audio for it to reference.
You can then expand its persona in further detail by conversing with the character as if you’re talking on FaceTime, outlining its backstory, temperament, goals, flaws, fears, movement quirks and more. Once you’re happy, the tool lets you generate animated screen tests with the character by describing an environment or scenario you want to put them in.
This is currently a standalone creative exercise, and it’s not clear how (or if) it could feature in any of Adobe’s existing AI video tools. The results provided a more identifiable and consistent personality compared to many soulless animated AI characters I’ve seen, but that’s the entire reason this exists — AI video generation provides far less creative control compared to traditional animation, where every aspect of a character is included with absolute intention.
There’s also Project Sight Span, which reframes photographs and images by turning them into a 3D environment that you can navigate through and reposition. The 3D space generates objects and backgrounds that are outside of the original frame, but feel like a natural part of the image, even if you’re turning to see what’s behind you. While the visuals in the 3D environment are pixelated and ill–defined, the resolution quality is restored when you select your chosen angle.
Sight Span works with any flat image, including photos, paintings, and illustrations, and can preserve and replicate artistic styles. Even repositioned tattoos and facial features in photographs remain surprisingly recognizable and true to the original reference. It’s a bit like a version of Apple’s Spatial Reframing feature on steroids: you can reposition more aggressively and change the shooting point entirely if you want, but it also feels far less deserving of being called a photo afterward.
Project Sound Bender is an AI audio mixing feature that lets you customize sound effects as if you’re clicking your cursor around a color wheel. It works by uploading a video clip and then describing your desired audio in a text prompt, such as “bubbling sci-fi cauldron noises.” Sound Bender then generates sound effects and automatically syncs it to the video, and provides a square sound canvas that lets you adjust the noises in real time. Each corner of the canvas will automatically assign a relevant AI-generated sound, such as dense gurgling with popping bubbles, rumbling boiling rocks, pitched bloops, and crackling electric fire. You can drag your cursor around the canvas to select which noises stand out the most, or customize each corner with different sound effects.
Adobe also showcased Project Rinse Repeat, which lets you copy edits and then automatically apply them to other objects or people in the same photo. For example, if you painstakingly airbrush out acne or blemishes on one person, Rinse Repeat will adapt your specific changes to anyone else in a group of people who need similar touchups. It can also be used to add shadows to products in marketing graphics, even if the objects are different shapes and sizes. Rinse Repeat is built around Photoshop, and the edits you apply to other subjects remain fully editable because the tool recreates them with native Photoshop tools.
These “sneaks” are tech previews of capabilities that might eventually be developed into tools for Adobe’s Creative Cloud apps or Firefly platform, so there’s no guarantee that they’ll graduate into being available features. Almost all of these projects are built around Adobe’s generative AI developments, however, and given AI tools now play a central role in the company’s growth strategy, it’s likely that at least some of the experiments will see an official launch. As wacky or fun as these AI experiments are, they’re also an omen. If they’re useful enough, that means AI may eventually replace the tools and stock libraries that creative professionals currently use, regardless of how complex their feelings are about the technology.
A notable theme across these sneaks is making complicated photo editing, filmmaking, and audio mixing workflows feel more accessible and intuitive, adapting them into something that’s more interactive and easy to use. In some cases the features almost seem like toys, but with the benefit of allowing you to make practical changes to creative projects.
“Making these interactions more intuitive is also about giving experienced creatives more precise control,” senior principle Adobe research scientist Gautham Mysore, told The Verge. “Sound Bender’s sonic exploration capability and Sight Span’s camera positioning are examples of how professionals could apply their judgment and refine a result more directly. That connection between creative intent and control is a big part of what we’re exploring.”
Every click has a price now. Who designs how that “feels”?
The rise of credit-based AI pricing is forcing UX designers to build cost-transparency into interfaces to prevent user anxiety and financial surprises.
Summary
Original Article
As AI products shift toward credit- and usage-based pricing, everyday interactions increasingly carry a visible financial cost, creating new challenges for user experience design. Unpredictable charges, unclear token consumption, and sudden usage limits can undermine trust, making transparent cost estimates, spending controls, and understandable feedback essential interface features. Designers must also decide when expensive AI processing is justified and when simpler alternatives can deliver the same result without introducing unnecessary cost or anxiety.
Protect Human Writing by Poisoning Unauthorized AI Training (Website)
ShieldFont is an anti-scraping web font that visually swaps characters in HTML to make site content legible for humans but scrambled for automated crawlers.
Summary
Decoder
- Scraper: Automated software that crawls websites to harvest data, often used to train large AI models without authorization.
Original Article
Publish for humans, not for crawlers. A web font that swaps the words in your HTML, so readers see your writing and scrapers get a decoy copy.
On the Brutality of Amdahl's Law
AI progress faces significant real-world economic friction due to physical and institutional bottlenecks that intelligence alone cannot bypass.
Summary
Decoder
- Amdahl's Law: A formula used to find the maximum expected improvement to an overall system when only part of the system is improved.
Original Article
The world is far too slow to see the high double-digit or more economic growth and seismic transformations in the economy that AI is predicted to bring. Intelligence isn't the only technology curve to go quasi-infinite in the last century and a half. Electricity, communication, computers, transportation, and agriculture have all experienced extreme growth and transformation. Despite this, we've only seen 2% GDP per capita growth for nearly 150 years.
Become a founding member of TLDR's AI community (2 minute form)
TLDR is launching a curated community for AI builders to discuss industry news and technical launches.
Summary
Original Article
TLDR is launching a curated community for AI builders to discuss major news and launches. The best links and comments will be featured in TLDR AI. Apply here to join.
Apple Bets on Siri AI Apps as the Main Draw for Its Smart Glasses
Apple is prioritizing Siri-integrated third-party apps as the primary selling point for its upcoming smart glasses due in 2027.
Summary
Original Article
Apple rolled out its new Siri AI earlier as it needs software makers to begin preparing for its smart glasses, which it plans to ship in 2027. The company is counting on integration with apps people already use on their iPhones as a key differentiator for the glasses. Its large developer ecosystem could give it an advantage over Meta and Google's offerings as it potentially provides countless third-party integrations on launch. The hardware for the glasses won't be dramatically different from rival offerings, which makes it more crucial for Apple to nail the developer ecosystem.
Making a programming language
Compilers are effectively pipelines of small programs that transform text into machine-executable binary code through sequential stages of abstraction.
Summary
Original Article
A compiler is a line of small programs. Each turns one shape into the next. The programs transform text into tokens and tokens into a tree, then the tree gets checked, and either an interpreter runs it or it becomes a native binary. No single stage is hard - the work is in making errors useful and keeping two backends in agreement.
Neumorphism CSS Generator (Website)
This web-based tool automates the creation of neumorphic CSS, but users should be wary of accessibility pitfalls like low contrast and poor affordance.
Summary
Decoder
- Neumorphism: A design style that uses specific shadow pairings (a light and a dark shadow on the same color) to make elements look physically extruded from or pressed into a background.
- Skeuomorphism: A design philosophy where digital interfaces mimic real-world physical objects.
- Affordance: The quality of an object that suggests how it should be used, such as a button looking 'pressable'.
- Inset: A CSS property that draws a shadow inside the border of an element, creating a 'pushed in' look.
Original Article
Easily generate soft UI and Neumorphism CSS styles. Customize colors and shadows, then copy the code.
Everything You've Saved, in One Visual Library (Website)
Magpie acts as a universal visual bookmarking library that uses AI to tag and search social media saves across multiple platforms.
Summary
Original Article
One visual library for everything you save on Instagram, X, TikTok, Pinterest, and more. Tagged by AI, searchable by meaning and color, and kept even when the post is gone.
Circuit: Civic Shuttle Brand Identity by The Collected Works
The Collected Works redesigned Circuit’s electric shuttle service branding, shifting from fragmented local identities to a cohesive, infrastructure-focused urban design language.
Summary
Decoder
- Fleet Wrap: Custom graphics or branding applied directly to the exterior of vehicles.
- Generative tool: Software used to create design assets algorithmically based on predefined parameters.
Original Article
The Collected Works has redesigned Circuit, an electric shuttle service operating across 50 cities, to replace its fragmented local identities with a cohesive brand inspired by dependable public infrastructure. The refreshed system retains Circuit's original C symbol, introduces Stadt and Haffer typography, and uses directional arrows, road-inspired graphics, and a restrained green palette to evoke urban transportation. A custom generative tool creates continuous route-like graphics, while standardized fleet wraps and flexible local messaging help the identity remain consistent across different cities.
Japanese Logos: 43 Classic Logo Designs and the Stories Behind Them
Japanese graphic design from 1960 to 1990 established a global standard for corporate identity by merging modernist geometry with strategic business goals.
Summary
Deep Dive
- Modernist influence: Japanese design leveraged geometric reductionism popularized post-WWII.
- Strategic integration: PAOS consultancy proved that logos were not just aesthetic, but instruments for corporate-wide change.
- Olympic impact: Events like Tokyo 1964 and Expo '70 provided a national stage for experimentation.
- Identity programs: Shifted companies from fragmented logos to unified house styles.
- Symbolism: Designers balanced 'mon' (traditional Japanese crests) with international modernist trends.
- Case studies: Covers NTT privatization, JAL's crane mark, and Bandai's transition to a multinational toy giant.
Decoder
- Corporate Identity (CI): A comprehensive design system that includes logos, typography, and visual language to express a company's business strategy.
- PAOS: An influential Japanese design consultancy that pioneered long-term corporate identity strategy in the 1970s and 80s.
- Mon: Traditional Japanese family crests, often highly geometric and symbolic.
Original Article
Japanese graphic design produced some of the world's most influential corporate identities between the 1960s and early 1990s, with designers including Yusaku Kamekura, Ikko Tanaka, Kazumasa Nagai, and Takenobu Igarashi. Forty-three classic logos spanning Sony, NTT, Japan Airlines, Mazda, and the Tokyo 1964 Olympics demonstrate how modernist geometry, cultural symbolism, and distinctive visual ideas helped establish recognizable identities. Many also emerged from strategic corporate identity programs, particularly those developed by PAOS, that connected visual design with broader organizational transformation.
Bruno's HelloAsso identity follows France's love of brands with values
Bruno rebranded the French nonprofit platform HelloAsso using a mosaic-based visual system that emphasizes its community-driven mission.
Summary
Decoder
- Nonprofit platform: Digital infrastructure (like HelloAsso) designed to assist community organizations with fundraising and management.
Original Article
Bruno has rebranded HelloAsso, a free platform serving 480,000 French nonprofits, with a new identity built around the idea of individuals coming together to form a collective. Its central graphic device is a mosaic of real member portraits, where individual squares can split to accommodate text, supported by bold Franklin Gothic typography, candid photography, and an expressive color palette. The system extends across HelloAsso Sport, Generosity, Culture, and its Fund for Engagement, giving each offering a distinctive visual personality while maintaining a recognizable shared identity.
The toy company Tomy has a new logo, and people are baffled
Toymaker Tomy is unifying its global identity as Takara Tomy with a controversial new abstract logo representing 'play' that launches April 1, 2027.
Summary
Decoder
- Katakana: One of the four Japanese writing systems, primarily used for foreign words or emphasis.
- Wordmark: A distinct text-based logo design that identifies a brand by its name rather than an abstract symbol.
Original Article
Japanese toymaker Tomy is adopting the unified global name Takara Tomy and replacing its familiar blue-and-white logo with a new identity featuring a teal-to-magenta gradient symbol inspired by the Japanese characters for "asobi" (play).
Brand Insistence: Why the Strongest Brands Have No Substitutes
True market dominance happens at 'brand insistence,' where customers refuse substitutes even at higher costs or lower availability.
Summary
Original Article
The strongest brands push customers past preference and loyalty to insistence, where they'll wait, search, or pay more because there's no substitute.