Investigating unintended model actions in our evaluations and internal use

This report describes examples of unintended model actions we’ve observed during evaluations and internal use of Claude. It is part of our effort to publish more frequent standalone reports on model behavior and alignment beyond our system cards, which we publish with each model release, and our risk reports, which we publish every three to six months as part of our Responsible Scaling Policy. We believe it’s important to be transparent about what we see our models do during testing and use.

The behaviors can be grouped into four categories:

• Claude exploiting a basic flaw in software to run commands on a server;
• Claude submitting a sensitive form on a real website when it should not have;
• Claude working around a restriction to reach data that was gated by a token or a fee; and
• Claude using URL shortening services to get around limits in its fetch tool.

We have chosen not to name the organizations involved in the examples below to avoid exposing vulnerabilities in their systems, and at their request. For this reason, we also provide less detail about each case than we otherwise would. Some of the cases described below involved websites run by U.S. government agencies at the federal, state, and local levels. We have briefed the White House on these cases and notified each agency involved.

The cases we’ve identified to date in these categories had minimal real-world impact. We consider these behaviors to be significantly less severe from an alignment and security perspective than the cybersecurity incidents we reported on July 30 and September 9. They resemble behaviors that we’ve described in our system cards since Claude Mythos Preview. Most are forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping.

Although the impact of these behaviors was minimal and we had already turned off live internet access for some high-risk and cybersecurity evaluations, we have now decided to expand that to include all our internal evaluations until we have confirmed that our security and monitoring measures (described in the remediation section of this post) reliably catch behaviors like these. Below, we also discuss alignment considerations and give more detail on how we’re modifying training to reduce the likelihood of further misbehavior.

We identified most of these cases through a review of transcripts that we began in July. Our review first focused on our cybersecurity evaluations—tests where a model is deliberately asked to probe or attack a test system, and where internet access is meant to be disabled. We have since extended our scanning to encompass a much wider range of instances where Claude could have reached the internet, including tests where internet access is deliberately enabled so Claude can be evaluated on real-world tasks. We began by looking for incidents of similar severity to the cybersecurity incidents we reported this summer; we have not found any to date. We then broadened the search to lower-severity cases, where a model interacted with real websites or systems in ways we didn’t intend.

We are also now scanning a much larger pool of lower-risk transcripts, as well as our use of Claude within Anthropic and in reinforcement learning (RL) environments where Claude has access to the internet. As this work continues, we plan to report new instances of unintended behaviors. All cases reported here involved Claude interacting with the outside world; to our knowledge, none of them involved customer data or Anthropic’s own internal systems.

In this post, we explain why we run evaluations, which are where most of these cases occurred; describe each behavior in more detail; and offer a preliminary view of what the behaviors suggest about Claude’s alignment and how we’re mitigating them.

Why and how we run evaluations

The behaviors described in this post are not specific to evaluations, but many of the cases we’ve identified to date occurred during evaluation runs. We continually test models on a wide range of tasks prior to releasing them. We use many different evaluations (standardized sets of tasks, scored the same way each time), each of which helps to paint a picture of Claude’s skills in a particular area. Many of the evaluations we use are public; they are written by outside researchers and can be run by any developer, allowing us to compare capabilities across models. Other evaluations we build in-house.

Because language models are non-deterministic—that is, their responses always involve some element of randomness, and they may carry out the same task slightly differently each time—we have Claude complete each evaluation task hundreds or thousands of times (each attempt is called a run). Testing so many times allows us to understand how a model typically performs, and also to catch rare cases where a model does something we don’t expect. What we learn from evaluations shapes decisions across Anthropic, including how we train models, what safeguards we build around them, and when we release them.

Evaluations are also where a model’s behavioral issues are sometimes first revealed. Models learn much of what they can do through RL, in which they attempt a task within a specific environment many times over and are rewarded when they succeed. However, training environments are imperfect. If training rewards something we didn’t intend—such as finding loopholes or working around a restriction—the model learns that the workaround pays off and may then apply it elsewhere. This is known as reward hacking. While we have processes in place to identify and filter out reward hacking during training, evaluations are a critical mechanism for understanding a model’s behavioral propensities in different contexts and on different sets of tasks than those we have in training.

Some tasks—such as searching the web for hard-to-find information—are difficult to realistically simulate in an environment without internet access. Running them with access has been standard practice within the industry: public web search benchmarks are run on the live internet by default, and running them the same way allows us to compare our models with other models. Most of the cases in this report occurred during such evaluations.

Overview of behaviors

Below, we describe four categories of behavior in which Claude acted on real websites and systems in ways we did not intend, each of which involved an organization or person outside Anthropic.

Exploiting a basic flaw in software to run commands on a server

When Claude could not complete an evaluation task directly, either because its own tools were deliberately limited or because an outside service it needed was unavailable, it used other tools hosted on a third party’s site to achieve its goal. In some circumstances, it did this by exploiting basic flaws in that site’s software, using SQL or command injection (a way of sending input that the site mistakenly runs as instructions).

For example, one evaluation asked Claude Mythos Preview to run a scientific analysis. The public tool it needed to perform that analysis was hosted by a university; when the model sought to access it, the tool returned an error. Looking for another way to complete the task, Claude explored the tool’s website, found a script on the university’s server that would return any file it was asked for, and used it to copy files from the server, including the script’s own code. In that code Claude found the injection flaw that let it run commands on the server, and used it to run the calculation.

In another instance, the data Claude Mythos 5 needed was free, but Claude could not access it without accepting a data use agreement, which it did not have the tools to do. Claude used applications hosted on the website to load the page and accept the agreement on its behalf.

We saw this behavior on DeepSearchQA and BrowseComp (both public web research evaluations), on LABBench2 (a public biology research benchmark), and during internal evaluations.

Submitting a form it should not have

We saw instances where Claude submitted an online form when it shouldn’t have. Generally, this occurred when an evaluation’s instructions were ambiguous, or when a misconfiguration within the environment prevented Claude from working with dummy forms.

For example, in a case that happened multiple times on the same evaluation, an unreleased, non-frontier research model was meant to fill out a practice copy of a government form. When the copy failed to load or the model closed it by mistake, the model instead navigated to the website where the real form is normally hosted and submitted the form there.

In another evaluation, Claude Haiku 4.5 was instructed to complete a form up until the final submission stage, but to stop before actually submitting. Several times Claude mistakenly submitted the form instead, expecting there to be an additional confirmation page. In these instances, the model’s chain-of-thought stated that it was demonstrating the process, not submitting a real request.

In a third example of this behavior, Claude Haiku 4.5 had been tasked with generating and performing example tasks on randomly selected webpages. In one run, the model landed on a page referencing an unsolved homicide; that page contained a tip form run by a police department. Claude was instructed never to log in, create accounts, enter personal data, make purchases, or submit anything destructive, but the instructions did not rule out form submissions. Claude filled out the form with the following: “I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant.” (The website did not include a description of the perpetrator.) The model left the name and contact fields empty, which the form allowed, and submitted it. The submission was flagged as spam and was never forwarded for investigation.

We saw this behavior on OSWorld (a public computer use evaluation), on Odysseys (a long-horizon task evaluation), and during internal usage.

Working around restrictions to reach gated data

Claude sometimes worked around a restriction to reach data to which it did not have access. Generally, this occurred when a server refused Claude’s request, or when the data was offered only for a fee.

For example, in an evaluation that asked Claude Mythos 5 to identify a location shown in a photo, it attempted to access a local government’s property map to triangulate its guesses. Claude could read webpage text, but the evaluation limited its ability to click through it as a person would. Instead, it read the settings file the site sends to browsers, found working access tokens, and used them to send requests directly to the server behind the map to access the data.

In another example of this behavior, an Anthropic researcher used Claude Mythos 5 on a statistics project that required it to pull public data that was available from a state agency for a fee. Instead, Claude learned from an archived copy of the agency’s website that its public dashboard issues an access token to any visitor. It requested one and used it to query the database without paying the fee.

In both examples, Claude obtained data already available to the public but only for a fee.

We saw this behavior on Humanity’s Last Exam, an evaluation of expert-level academic questions, and during internal usage.

Using URL shortening services

Some of our fetch tools, which let Claude read webpages, limit the length of the URLs Claude can request. This limit is in place to reduce the risk from Claude using long URLs to take certain unwanted actions, such as exfiltrating larger amounts of data. We saw several models, including Claude Opus 5 and Claude Mythos 5, get around this limitation by using free URL shortening services. We first identified this behavior internally, but as we were writing this post, an operator of the da.gd URL shortening service let us know that they had also found Claude using their website for this purpose.

Remediation

We’ve taken several preventive measures. Some of the public evaluations we no longer run; others we have moved to their offline versions, or rebuilt them so that their tasks do not reach live websites. We’ve also made broader changes. We have updated the guardrails on some of our internet access tools, such as the web fetch tool, to heavily restrict what the model can do with them. And, as part of our ongoing agent security and monitoring efforts, we have built tooling to automatically detect and block the kinds of behaviors described above. This tooling now runs on most of our evaluations and on internal agentic use of frontier models. When we tested it against the cases described in this post, it blocked all of them.

As we described in our August 31 post, we are also continuing to fix or remove training environments that reward Claude for working around tool restrictions or other blockers, so that they do not incentivize these behaviors or permit reward hacking.

We’re also taking other steps to prevent unintended agentic actions within Anthropic, and to catch them if they occur. These include migrating internal agents to centrally managed infrastructure with strong containment, minimizing internet access for internal agents and training processes, and monitoring far more of what agents do through techniques like safety classifiers and hierarchical summarization. These measures are now part of our security team’s detection and response procedures so that we can respond to and contain undesired behavior quickly. We expect to publish more on these approaches, and to build them directly into our products.

Discussion and alignment takeaways

While we have not completed a full alignment assessment of these cases, we consider them to be less severe than the cybersecurity incidents from this summer. We explain why below, and share the general lessons we’ve drawn from them.

Alignment context

As we establish a regular public reporting process for behaviors like these, we also want to situate them in terms of their severity. This remains challenging: judging the extent of an alignment failure often depends on what the model was trying to do, and a model’s own account of its reasoning is not necessarily reliable evidence of its beliefs or reasons for action, as we discussed in our September 9 alignment assessment. As models grow more capable, they can interact with the world in more sophisticated ways, which makes more complex failures possible. For example, the biased reasoning we described in that assessment was not new, but it led to more serious consequences in a setting where models could capably act on real systems for hours at a time. An alignment severity framework built around today’s failures could become outdated quickly, given how fast AI capabilities are advancing.

Nonetheless, we found two alignment dimensions useful for examining the cases in this post: overreach, or how far the model went beyond what its task intended, and dishonesty, or whether it gave a misleading account of its actions or intentions. On overreach, we consider these cases to be substantially less concerning than the incidents we reported this summer, in which Claude gained access to real third-party systems for hours during cybersecurity evaluations. In the cases we report here, in contrast, Claude bypassed access controls to reach gated but publicly available data, or exploited software flaws and ran commands on servers (mainly to reach non-sensitive data).

On dishonesty, the comparison is more mixed. In one case, tasked with generating example interactions with websites, Claude submitted an invented tip through a police department’s online form. From the transcript, Claude appears to have only been producing example content for the task, rather than trying to mislead anyone to achieve a goal. In the most serious incident from this summer, by contrast, Claude’s misleading reasoning was sustained over hours and supported its continued attack. Still, judging dishonesty with confidence typically requires a deeper assessment than we have done here, such as replaying transcripts with modifications to test what drove the model’s behavior, so our view of these cases may change with further analysis.

Lessons and next steps

None of the behaviors we’ve described here are new, and they do not change our overall view of Claude’s alignment. In many of the cases where we observed these behaviors, Claude had been given tasks that were ambiguous or impossible to complete. We and others have observed that when models are given impossible-to-complete tasks, they will pursue unintended and sometimes misaligned strategies to achieve their goals.

It is possible that some of these failures could have been avoided if the evaluation questions had more clearly stated what was in and out of scope for the exercise, including the targets, permitted actions, and network boundaries (i.e., what the model should and shouldn’t access). However, Claude encounters ambiguous and impossible tasks every day in real use, and, indeed, several of the cases we observed occurred during regular agentic use of Claude.

Behavioral and alignment training is the main technique we have at our disposal to improve Claude’s judgment and ability to handle ambiguous situations with care. Historically, we have been more focused on teaching models to respect boundaries and be appropriately cautious in coding environments. We are now expanding these environments to applications like search and computer use, which were involved in the cases described here. However, alignment training is not yet sufficient or fully robust on its own, at least in the short term, so we also rely on defense-in-depth approaches, including the classifiers and safeguards described above.

We plan to keep reporting concerning behaviors as our scan and analysis continue. We hope these reports help other developers check for similar behaviors in their own models, since many of the evaluations involved are public and widely used. While these cases had minimal impact, we do not want to diminish the findings, because the same behaviors could do far more harm as models become more powerful. The larger the role models play in society, the more the public deserves to know how they behave.

Full article content is not available for inline reading.

Read the original article →

The Deno team is joining Cloudflare to radically simplify self-hosting Workers and Durable Objects so developers can use the same primitives in more places.

For what this means and how it came about, here’s the story from Ryan Dahl and Kenton Varda.

The Deno team is joining Cloudflare, and we're merging workerd and celld! For more on what's happening to the Deno runtime and our various efforts, see my post on the Deno blog.

Long ago, I stood in a Berlin warehouse and introduced Node.js with a 500-line JavaScript IRC server and live audience participation. Node.js was an exploration of what a purely asynchronous programming model could enable. Async I/O was already known to be important for fast servers, but it was difficult to wield. Removing sync networking made it easier to build systems that previously required much more machinery. What made the IRC demo exciting was how little code it took.

I started Deno to find more powerful abstractions like this — looking for places where we could simplify and remove capabilities while giving engineers ever more power. With Deno, we improved the experience of writing JavaScript, but we didn’t fundamentally change what developers had to assemble around the runtime. The bigger problems are around networked applications: distributing computation, coordinating state, data storage in general, and autoscaling to meet demand.

Something about that old IRC demo has always bothered me — it was a single server, single thread even. It could handle many user connections, but it would start getting slow if you had too many. Scaling IRC eventually means multiple machines with the channels shared across them. A modern chat app also needs to store message history, user accounts, and other data.

Could the programming model make partitioning across machines part of the design from the start?

The solution clicked into focus when I read about Cloudflare's Durable Objects. It provided a very powerful abstraction: the distributed singleton with a SQLite database. Each Durable Object is like a small, individually addressable server with its own relational database. Its JavaScript execution is single-threaded (which is easy to reason about), it handles WebSockets, and its local SQLite database can be accessed synchronously. By building a chat application using one DO per channel, you have sharded your data and WebSocket connections, and thus made your application scalable.

Much more than chat can be built with DOs. On top of this abstraction other services can be built: Queues, KV, Durable Execution (CF's Workflow API), even Git storage. The Durable Object abstraction is simple yet incredibly powerful.

But running this model outside of Cloudflare has remained difficult. Cloudflare’s network is great — but there are a million reasons one might want to run it themselves. Workerd is already open source, but its Durable Objects support is limited to a single instance. What I wanted was the distributed version: IRC channels spread across machines, with the platform handling placement, routing, and durable storage. The power of this abstraction is how it scales. That's why I started celld.

Building and operating Deno Deploy taught me how complicated cloud infrastructure could become: multiple public clouds, multiple databases, and deeply entangled services. With celld, I wanted the opposite: one binary, written in Rust, with object storage as its only external service dependency.

What excites me is the leverage this model gives developers: compute, relational databases, queues, and real-time communication primitives designed for applications distributed across machines. To run it, you manage many instances of celld and a single object storage bucket. Your application can contain many services without each requiring a separate infrastructure project. This is much closer to the simplicity I've been chasing since that first Node.js demo.

We’re joining Cloudflare to make the Workers programming model a mainstream way to build servers. By bringing celld and workerd together, we want it to be radically easy to build and operate distributed applications on your own infrastructure.

This may come as a surprise to some, but when the Deno team released celld in August, we were delighted.

Why a surprise? Well, as it happens, there is a popular theory that suggests we shouldn't be happy.

You see, it is commonly claimed around the Internet that we made Cloudflare Workers work differently from other cloud platforms in order to create "lock-in." The claim goes, software built on Workers is hard to move to other hosting platforms, therefore if you write your app on Workers you are "trapped" on Cloudflare forever. This is especially true of apps that use Durable Objects: they are not just written differently but architected differently. All this is, supposedly, a clever trap: once you are too deep to be able to switch, the rug will somehow be pulled out from under you.

Under this theory, celld is a threat: as an open-source implementation of Workers and Durable Objects, it allows applications built on Workers to seamlessly migrate to other providers. No more trap!

This theory is wrong

Workers is different because it is better

We built Workers differently because we believe it is a better way to build web applications.

• The Workers design makes it dead simple and dirt cheap to manage an application that runs in hundreds of locations around the world, a feat which no other popular hosting platform has achieved. The pricing isn't a trick: we've genuinely created a novel architecture that is more efficient than competitors, and we've passed on the savings to you.
• Workers' live environment design (aka "bindings") simultaneously makes configuring access to external resources easier and more secure. Usually, those two things don't come together!
• Durable Objects make it easy to build real-time collaboration and distributed systems that are difficult or impossible in a classical three-tier web architecture.

In short, we believe we've simply created a better programming model for cloud software. But being better requires being different: we couldn't achieve these benefits while remaining perfectly compatible with existing platforms.

If we could offer these benefits and be perfectly compatible with traditional cloud platforms, we absolutely would! It would be a great benefit to our business if every kind of compute workload could simply be lifted and shifted onto Workers. But the reality is, innovation sometimes requires doing things differently. We feel that, in the long run, it's better to be different and better, than to be the same as everyone else.

"Lock-in" actually hurts us — that's why we went open source

If there were truly no escape hatch from Workers, then some of our largest customers would never have signed on with us in the first place.

When we originally pitched Workers for Platforms to customers like Shopify in 2022, the feedback was clear: They couldn't possibly build on us unless the Workers Runtime was open source.

And so, we open-sourced it. That's right: not a lot of people seem to know this, but the Cloudflare Workers Runtime, workerd, is open source. This is not a parallel implementation, it's the same code we run in production.

As a matter of fact, we have former customers that migrated away from us by using workerd. And that's OK! Those customers would never have signed with us in the first place if we hadn't been open source, and for every one that migrated away, there are many others that have stayed.

It turns out, being open source and giving people an escape hatch is good business.

But we could do better – and with Deno's help, we will

We released workerd genuinely intending for people to use it in production. A few have, but so far, not that many. Most people don't seem to have any idea that this is even possible.

Why not?

Frankly, we haven't done a great job of building out the ecosystem of adjacent services and tooling to make it really work. We'd released the runtime, and then we sort of just hoped the community would step in and start adapting it to every possible environment. It hasn't happened that way. Perhaps we were naive.

In the interest of transparency, I do have to admit that workerd has always had one big gap in its production-readiness: Durable Objects. Workerd implements them, but only in a single-instance way, good enough for local testing, but unable to scale. We have always wanted to fix this. There are TODO comments in the code going back to the original release. The problem is, our production implementation of Durable Object routing is simply not the implementation any self-hoster would want. It's a beast, designed to handle hundreds of locations, with numerous dependencies on external services operated by a team of site reliability engineers. We knew we needed something different for the self-hosting use case, but we never quite found the time to build it. (I actually made an attempt last spring, but, embarrassingly, it didn't work.)

So now, hopefully, it's clear why we were happy about celld: Here was an implementation of Workers and Durable Objects designed to be fully compatible with our own, while being focused on being self-hostable and scalable. That's what we'd wanted to build for a while, and Deno Land was building it for us!

And who better to do it than Ryan Dahl and co? Ryan is the creator of Node.js. He invented the idea of using JavaScript on the server — without his work, Workers may never have existed. And then, he, Bert Belder, and the rest of the Deno Land crew did it again, creating Deno, another JavaScript (and TypeScript) runtime that further advanced the state of the art. The Deno team understands how to create a good developer experience around a self-hosted runtime — something we never quite cracked with workerd.

Imagine, then, our excitement when the Deno team told us they'd like to join forces on this project.

The Plan

Ryan and Bert will be leading a new effort to make workerd self-hosting a first-class supported way to build and run apps using the Workers programming model. This will involve merging code and ideas from celld back into workerd. I'm incredibly excited for this work — I will personally be using it to host an instance of Cloudflare OS in my home.

We'll have more announcements about this work in the coming months. But if you can't wait, you can start self-hosting celld or workerd today.

I remember being in the room at DEF CON 23 in Las Vegas, in 2015, listening to "Let's Encrypt – Minting Free Certificates to Encrypt the Entire Web". It was still months before Let's Encrypt would launch their public beta and begin cementing 90-day certificates as the new norm. More than a decade later, Let's Encrypt have just announced their first ever reduction to their default certificate lifetime, and it's coming soon.

The Dates

Here's what's happening and when:

Any certificate issued on or after 10 Feb 2027 gets a 64-day validity period, unless you've already opted in to something shorter than that. Those options have been around for a while now, with the 6-day certificates available via the shortlived profile and 45-day certificates via the tlsserver profile. What changes in February is the validity period for certificates using the default profile, so this will impact everybody, including those that may not have been paying attention.

Why Now?

Let's Encrypt have been ahead of the curve for their entire existence, launching with 90-day certificates when 3-year certificates were the norm, and this change is a continuation of that trend.

Last year, Ballot SC-081 passed at the CA/Browser Forum, setting a schedule that takes the maximum certificate lifetime from 398 days down to 200 days (March 2026), then 100 days (March 2027), and finally 47 days in March 2029. The maximum period that domain validation data can be reused shrinks alongside it, down to just 10 days by 2029.

Let's Encrypt's 90-day certificates were already comfortably inside the 2026 and 2027 changes, so they didn't have to do anything until 2029. Instead, they're stepping down gradually, with a 64-day stop on the way to 45 days, and giving everyone the ability to test this out gradually.

As for why shorter is better, I've been banging this drum for a long time... Revocation is broken, and has been for as long as I've been writing about it, so the only reliable way to limit the damage of a compromised key or a mis-issued certificate is to make sure the certificate doesn't live very long. I wrote Why we need to do more to reduce certificate lifetimes back in 2018, watched Ballot SC22 fail in 2019, saw Apple force the issue with the 1-year cap in 2020, and then asked Are shorter certificates finally coming?! in 2024. They are, and this is what it looks like when they arrive.

What Might Break

If your ACME client supports ARI (ACME Renewal Information), Let's Encrypt will tell it when to renew, and you don't need to think about the lifetime of your certificates at all.

ARI is relatively new though, and I'd bet, given how long Let's Encrypt have been around, that many automation processes are run on a fixed schedule. For 90-day certificates, renewing at 60 days old would be quite typical. This means that for those who have a fixed 60-day renewal process, this first reduction to 64-day certificates will just catch the new certificates before they expire, an intentional choice from Let's Encrypt I'm sure. If that is the case, organisations won't see any negative impact from this first reduction, but the 47-day certificates coming in March 2029 will cause problems and Let's Encrypt are reducing their certificates to 45-day certificates in February 2028 ahead of the deadline.

What To Do Now

The best thing is to check if your ACME client supports ARI and use that wherever possible. If not, you need to check how your certificates renew and if it's a fixed schedule. If you're renewing on a fixed schedule, it ideally needs to change to be relative to the validity period of the certificate, and you should aim to renew your certificates when they're 2/3 of the way through their validity period. For your existing 90-day certificates that's at 60 days, but for these new 64-day certificates, I'd be renewing them at 42 days.

After more than a decade at 90 days, the default is finally moving, and in a little less than 18 months from now, every Let's Encrypt certificate will be valid for half as long as it is today! It's pretty crazy to look back over how the ecosystem has changed during that time, and it's fair to say that Let's Encrypt has played, and continues to play, an enormous part in that.

DigitalOcean MicroVMs: Fast, isolated compute for your AI agent infrastructure

Coding-agent platforms, sandbox products, and code-execution services all need the same thing: isolated machines that start up fast, retain their state between bursts of work, and don’t consume compute when idle. If you build it yourself, you’ll have to lease compute capacity and turn raw servers into isolated, on-demand compute. This often includes fleet management, snapshot, pause and resume, per-tenant isolation, placement, and recovery. This takes a lot of time that could be better spent on building your product.

Today, we’re announcing DigitalOcean MicroVMs in public preview so you don’t have to build and operate your own virtualization layer to run your agent infrastructure. A MicroVM is an isolated virtual machine built on open-source Firecracker that pauses when idle and resumes exactly where it left off when active.

In our previous blog post where we announced DigitalOcean MicroVMs private preview, we detailed multiple MicroVM use cases for agent infrastructure. Here, we’ll use a coding-agent platform as an example to see what building one looks like on MicroVMs. Each session consists of an agent reading a repository, writing code, and running the code and subsequent tests. This requires an isolated environment per session, which is created when the session starts and destroyed when it ends.

One MicroVM per session

When a session starts, you create a MicroVM from your container image. This packages your agent, its toolchain, and its dependencies. The MicroVM boots on Firecracker with its own kernel and a real hardware-virtualized boundary rather than a shared one. The agent inside can write and run whatever code it needs, and is designed to contain any potential bad steps to that single MicroVM, isolated from the host and other customers. The session is accessed over an authenticated HTTPS endpoint, so only requests with a valid DigitalOcean API token can reach it.

Pay only while it’s running, not while it’s paused

A coding agent spends most of its time during a session waiting, on the model to respond, on its code to run, or on a person to review its work. A MicroVM pauses when it has been idle for a time duration you set, and you don’t pay for compute while the MicroVM is paused. When the next request arrives, the MicroVM resumes in milliseconds and picks up where it left off with its memory, files, and running processes exactly where they were.

Fast startup and resume

A MicroVM starts in milliseconds, so it’s ready the moment a session begins. Resuming a paused MicroVM is just as fast, which makes it practical to pause between steps rather than holding a long-lived machine open for each session or user. It also means a burst of new sessions just spins up more MicroVMs on demand, up to your team’s MicroVM limit, instead of you having to provision capacity ahead of time.

Start every session warm, not from scratch

Creating an agent’s environment requires a lot of setup, and rebuilding everything for each new session is slow and inefficient. MicroVMs let you deal with the setup of the environment once, checkpoint it, and start new sessions from the checkpoint. Each session begins pre-warmed in the state you capture, ready for a task the moment it starts without any of the setup.

Built-in recovery and private networking

Moving coding agents beyond a proof of concept requires capabilities such as automatic recovery and private networking. If a workload crashes after it has started, the MicroVM restarts it automatically. However, with a MicroVM, a crashed workload stuck in a failure-loop will stop retrying to prevent a broken image from restarting endlessly. Additionally, MicroVMs can be run inside your VPC so they can reach your databases and internal services without exposing them to the public. If you don’t have a VPC, DigitalOcean can create one for you.

Already powering DigitalOcean Managed Agents

MicroVMs power DigitalOcean Managed Agents (available in public preview) so you have the same pause, resume, and isolation capabilities.

MicroVMs give you the virtual machine and its lifecycle: create one from an image, pause and resume it, checkpoint it, run commands inside it, and access it over its endpoint. Managed Agents is the upper layer, and its DigitalOcean Harness Runtime adds an agent control plane over the MicroVM. You don’t have to build anything to handle reading and writing files, uploading and downloading them, running and upgrading the agent harness, preserving session state across runs, and brokering governed access to tools.

Pricing

You only pay for compute usage while a MicroVM is running, at $0.022 per vCPU-hour and $0.0085 per GiB-hour, billed per second. Storage, billed at $0.05 per GiB-month on what you actually use, keeps each MicroVM’s disk and saved state available, so a paused MicroVM costs you only for its storage. Outbound data transfer on public networking is $0.01 per GiB.

A MicroVM allocated 2 vCPU and 4 GiB memory, if run for a full hour, will cost $0.078 in compute usage. A coding-agent session with this vCPU and memory configuration, that runs for 15 minutes in that hour and is paused the rest of the time, will only cost $0.0195 in compute, plus the storage it uses. So, the longer a session is idle, the less you spend on compute.

Getting started

A coding-agent platform is just one example of what you can build with MicroVMs. You can use it for other use cases such as sandboxes for untrusted code, eval and reinforcement learning (RL) environments, CI/CD runners, and pull request (PR) preview environments.

Create your first MicroVM from the Control Panel, doctl, or the API (doctl example shown below). The default HTTP port is 8080. Use --http-port to match your container.

doctl compute microvm create my-agent-session \
  --oci-ref registry.digitalocean.com/your-team/agent:latest \
  --region <region> \
  --cpu 2 --memory 4096

From there you can pause and resume it, checkpoint it, and run commands inside it. Visit the MicroVMs Control Panel to get started or see the docs to learn more.

A database migration that completes instantly on a developer laptop can stop a production system.

The SQL may be correct. The risk comes from everything around it: a large table, long-running transactions, lock queues, replicas, multiple application versions, connection pools, background jobs, and customers actively writing data while the schema changes.

Zero-downtime database migration is therefore not a special command or a feature of one framework. It is a release discipline that keeps the old and new application versions compatible while schema and data move through controlled intermediate states.

At BoundLayer, we treat PostgreSQL changes as production engineering work. Every migration has an expected lock, runtime, I/O profile, compatibility window, observability plan, abort condition, and cleanup phase.

This guide explains the practical patterns we use to evolve PostgreSQL safely in SaaS, fintech, data, and backend-heavy systems.

What Zero Downtime Actually Means

"The database stayed online" is not a sufficient definition.

A migration can leave PostgreSQL technically available while API requests wait behind a lock, connection pools fill, background workers time out, replicas fall behind, or customers receive errors.

Define migration success through service-level outcomes:

• no material increase in request error rate;
• p95 and p99 latency remain within agreed limits;
• writes continue for critical workflows;
• replication lag remains bounded;
• no lost, duplicated, or incorrectly transformed data;
• both old and new application versions remain valid during deployment;
• the change can be paused or reversed without improvisation.

Some schema operations need a short lock. The engineering goal is usually not literally zero locking; it is to make lock acquisition fast, bound its impact, and avoid long work while a restrictive lock is held.

Why Safe SQL Can Still Cause an Incident

PostgreSQL uses locks to preserve correctness while objects change. Many ALTER TABLE forms acquire an ACCESS EXCLUSIVE lock, which conflicts with normal reads and writes.

The dangerous part is often the lock queue.

Imagine a migration waiting for a long-running transaction to release its lock. New application queries can then queue behind the waiting migration. A schema change that has not even started may cause an expanding production outage.

The impact depends on:

• exact PostgreSQL version;
• operation and data type;
• table and index size;
• existing transaction duration;
• query traffic and connection limits;
• foreign keys and dependent objects;
• replicas and logical decoding slots;
• whether the operation scans or rewrites the table;
• disk, WAL, I/O, and CPU headroom.

Do not classify migrations only as "DDL" or "data migration." Inspect the specific execution and locking behavior on the production version.

The Expand-Contract Pattern

Expand-contract, also called parallel change, is the foundation of zero-downtime schema evolution.

Instead of replacing the old contract in one release, the team introduces a compatible new contract, moves traffic and data gradually, verifies the result, and removes the old contract later.

1. Expand
   Add new schema without breaking old code

2. Migrate
   Backfill data and support both representations

3. Switch
   Move writes and reads to the new representation

4. Verify
   Prove correctness and observe a full production window

5. Contract
   Remove old code, columns, indexes, and compatibility paths

Each phase is a separate deployable and observable step. Old and new application instances can run together during a rolling deployment.

The contract phase must have an owner and a scheduled date. Otherwise temporary columns, dual writes, and unused indexes accumulate into permanent complexity.

Preflight Every Production Migration

Before executing a migration, answer these questions.

What lock will it request?

Read the documentation for the deployed PostgreSQL version and test the operation. A short metadata change can still wait indefinitely if another transaction holds a conflicting lock.

Will it scan or rewrite the table?

A scan consumes I/O and can run for hours on a large relation. A rewrite can require substantial temporary disk and generate heavy WAL, replica traffic, and backup load.

How large is the object in production?

Staging usually does not reproduce row count, index size, write concurrency, data skew, or long-lived transactions. Use production metadata and a representative restored copy for rehearsal.

Which application versions will coexist?

During rolling deployment, version N and N+1 may serve traffic simultaneously. The schema must work for both.

What else depends on the object?

Check views, functions, triggers, foreign keys, analytics, ETL, CDC connectors, exports, BI tools, and manually maintained integrations.

What is the abort condition?

Specify thresholds for lock wait, latency, errors, replica lag, CPU, disk, WAL, and queue depth. Decide who can stop the migration.

Is rollback technically valid?

Reversing DDL does not reverse already transformed data or external side effects. Forward repair is often safer than a blind "down" migration.

Bound Lock Acquisition With Timeouts

A migration should fail quickly rather than wait in a production lock queue.

Set a short lock_timeout for operations that are expected to acquire their lock almost immediately:

SET lock_timeout = '2s';
SET statement_timeout = '15s';

ALTER TABLE accounts
  ADD COLUMN risk_tier text;

The values are workload-specific. The principle is to distinguish "could not safely acquire the lock" from "started valid work that needs more time."

Retry lock acquisition with bounded delay during a controlled window. Do not increase the timeout until the command eventually blocks production.

Use a separate session and explicit transaction scope so settings cannot leak into unrelated work. Log the attempt, database, migration version, application release, and failure reason.

Before retrying, inspect blockers and long-running transactions. An old idle-in-transaction session may be the actual defect.

Adding a Column Safely

Adding a nullable column without a volatile default is generally a fast metadata operation on modern PostgreSQL, but it still requires a lock.

Use an expand sequence when the new field will eventually be mandatory:

1. Add the nullable column.
2. Deploy code that can read old rows and write the new field.
3. Backfill existing rows in small batches.
4. Verify that no nulls remain.
5. Add and validate an equivalent check constraint.
6. Apply NOT NULL in the safest form supported by the deployed version.
7. Remove temporary compatibility logic later.

Do not combine schema expansion, full-table backfill, and constraint enforcement in one migration transaction.

Defaults also need semantic review. A database default applies when a column is omitted, not necessarily when application code explicitly writes NULL. Confirm how all writers behave.

Adding Constraints Without a Long Blocking Scan

Validating a foreign key or check constraint against all existing rows can be expensive.

PostgreSQL supports adding some constraints as NOT VALID, which begins enforcing them for new or changed rows without immediately scanning all historical data:

ALTER TABLE payments
  ADD CONSTRAINT payments_amount_positive
  CHECK (amount_minor >= 0) NOT VALID;

After fixing historical violations and selecting a safe period, validate separately:

ALTER TABLE payments
  VALIDATE CONSTRAINT payments_amount_positive;

The current PostgreSQL ALTER TABLE documentation explains that validation uses a less restrictive lock than adding and validating the constraint in one blocking operation because new writes are already being checked.

Keep the commands in separate transactions. If both are wrapped in one transaction, the initial lock remains held until commit and defeats much of the operational benefit.

For a future NOT NULL, a validated check such as CHECK (column IS NOT NULL) NOT VALID can establish evidence about existing rows before the final metadata change. Confirm the exact optimization behavior for your PostgreSQL release.

Creating Indexes Concurrently

A regular CREATE INDEX blocks writes to the table while the index is built. On a busy, large table, use CREATE INDEX CONCURRENTLY when its trade-offs fit the workload:

CREATE INDEX CONCURRENTLY idx_orders_customer_created
  ON orders (customer_id, created_at DESC);

Concurrent creation permits normal writes but takes longer, performs additional work, and cannot run inside a normal transaction block. It can also increase I/O, CPU, WAL, and replica lag.

If the operation fails, PostgreSQL may leave an invalid index. Detect it, investigate the cause, and remove or rebuild it deliberately before retrying.

Validate the index after creation:

• it is marked valid and ready;
• its definition matches the intended query;
• query plans use it under production-like parameters;
• write latency and storage remain acceptable;
• no duplicate or redundant index should be removed later.

For a unique or primary-key constraint, PostgreSQL can attach a suitable unique index created concurrently using ADD CONSTRAINT ... USING INDEX. The official documentation describes this as a way to avoid blocking table updates while building the index.

Remember that an index is a production cost, not a free optimization. Every write must maintain it.

Rename a Column Across Multiple Releases

Renaming a column directly breaks application code, jobs, and integrations that still use the old name.

Use a shadow-column migration.

Suppose users.name must become display_name.

Release 1: Expand

Add display_name as nullable. Keep name unchanged.

Release 2: Dual write

Write both fields for every new change. Reads continue from name, or use a compatibility fallback.

display_name = row.display_name ?? row.name

Decide where dual-write correctness lives. Application writes are easier to understand but every writer must be updated. A temporary database trigger covers unknown writers but adds hidden behavior and needs careful testing.

Background backfill

Copy old values in bounded batches and record progress.

Release 3: Switch reads

Read display_name, retain fallback and monitoring temporarily, and compare representations.

Release 4: Stop old writes

Make the new column authoritative. Observe for an agreed period.

Release 5: Contract

Remove old-column dependencies, then drop name in a dedicated migration.

This takes longer than a rename command, but it supports rolling deploys, rollback, and verification.

Changing a Column Type

Some type changes are metadata-only; others rewrite the table or invalidate application assumptions.

For a high-risk conversion, create a new column with the target type and migrate through expand-contract:

ALTER TABLE events
  ADD COLUMN occurred_at_v2 timestamptz;

Then:

1. Deploy dual-write logic.
2. Backfill in batches using an explicit, tested conversion.
3. Quarantine values that cannot convert safely.
4. Compare old and new representations.
5. Switch reads behind a feature flag.
6. Stop writing the old field.
7. Remove the old field in a later release.

This is especially important for currency, identifiers, time zones, JSON structures, enums, and precision changes. A syntactically successful cast may still be semantically wrong.

For financial data, define rounding, currency unit, historical interpretation, and reconciliation before migration.

Backfill Data Without Overloading Production

A backfill is a production workload. It competes with customer traffic for CPU, buffers, I/O, WAL, locks, vacuum capacity, and replica bandwidth.

Do not run one unbounded update across a large table.

Use small, committed batches selected by a stable key:

WITH batch AS (
  SELECT id
  FROM accounts
  WHERE risk_tier IS NULL
  ORDER BY id
  LIMIT 1000
  FOR UPDATE SKIP LOCKED
)
UPDATE accounts AS a
SET risk_tier = derive_risk_tier(a)
FROM batch
WHERE a.id = batch.id;

The exact query depends on concurrency and derivation logic. The operational pattern is more important:

• commit each batch;
• make the update idempotent;
• persist a checkpoint or derive remaining work safely;
• throttle between batches;
• pause automatically when service metrics degrade;
• monitor WAL, replica lag, dead tuples, and vacuum;
• expose rows completed, remaining, failed, and retried;
• allow multiple workers only with proven partitioning.

Avoid offset pagination because concurrent changes and large offsets make progress unreliable. Use a key range, cursor, or work-claiming strategy.

If derived values depend on mutable source fields, define what happens when an application write races with the backfill. Dual-write code, row versions, or conditional updates may be required.

Keep Dual Writes Consistent

Dual writes are temporary compatibility infrastructure, not a guarantee of consistency by themselves.

Writing two columns in one database transaction is manageable. Writing to PostgreSQL and a separate service, broker, or database creates a distributed consistency problem.

For external propagation, consider a transactional outbox or log-based CDC so the business state and event publication do not depend on two independent commits.

During a migration, instrument divergence:

• old value does not match transformed new value;
• only one representation is populated;
• writes still arrive from an old application version;
• backfill overwrote a newer value;
• downstream consumers still query the old schema.

Do not contract until divergence is zero or every exception is understood.

Removing a Column or Table

Destructive migrations should come last, never in the same release that stops using the object.

Before removal:

1. Stop application reads.
2. Stop all writers and background jobs.
3. Remove ORM mappings and wildcard SELECT * assumptions.
4. Check views, functions, triggers, reports, exports, and integrations.
5. Monitor database access for an observation period.
6. Preserve data according to backup and retention policy.
7. Remove dependent constraints or indexes deliberately.
8. Execute the drop with a short lock timeout.

PostgreSQL can make dropping a column logically fast because it does not immediately rewrite every row, but lock acquisition and dependent-object behavior still matter. Disk space may not be reclaimed until later row rewrites or maintenance.

Migration Observability

Every significant migration should have a dedicated dashboard or run view.

Database signals

• lock wait and blocking sessions;
• active and long-running transactions;
• transaction age;
• CPU, I/O latency, buffer activity, and disk headroom;
• WAL generation and archive throughput;
• replica lag and replication-slot retention;
• dead tuples and vacuum progress;
• connection-pool saturation;
• index build or table operation progress where exposed.

Application signals

• request rate, error rate, and latency percentiles;
• database timeout and serialization errors;
• queue and job lag;
• failed writes and retries;
• old-code and new-code instance counts;
• feature-flag state.

Data signals

• rows remaining in the backfill;
• old/new representation mismatch;
• null or invalid value count;
• business-level reconciliation totals;
• consumer progress and stale reads.

Correlate every metric with migration phase and release version. A migration is not complete when the SQL exits; it is complete when application behavior and data correctness are verified.

Test With Production-Like Conditions

Schema migration tests need more than a clean empty database.

Use a recent, protected, and appropriately anonymized production snapshot when policy permits. Reproduce:

• table and index sizes;
• realistic data distributions;
• concurrent read and write traffic;
• long transactions;
• replicas or representative WAL consumption;
• limited disk and I/O headroom;
• application versions N and N+1 running together;
• retries, cancellation, and partial completion.

Measure lock acquisition separately from operation duration. A fast migration in an idle staging environment says little about a busy production table.

In CI, run compatibility tests in this order:

1. Old application against old schema.
2. Old application against expanded schema.
3. New application against expanded schema.
4. Mixed old and new applications during dual-write.
5. New application after backfill.
6. New application after contract.

This catches the common error where a migration and application work only when deployed simultaneously.

Rollback and Forward Recovery

Rollback planning starts before deployment.

For the expand phase, application rollback is usually straightforward because the old schema remains available. During dual-write, the old representation must continue receiving valid values if rollback is a requirement.

After destructive contraction, rollback becomes much harder. Recreating a column does not restore its data. Restoring a full backup can discard new production writes.

Prefer these recovery options:

• disable the new read path with a feature flag;
• stop or throttle the backfill;
• deploy a compatibility fix forward;
• replay an idempotent transformation;
• restore selected data from a retained source or archive;
• delay destructive cleanup until rollback is no longer operationally necessary.

Document the point of no easy return. Require explicit approval before crossing it.

Migration Ownership in Continuous Delivery

Unsafe database changes often come from process ambiguity rather than lack of SQL knowledge.

A mature migration workflow assigns:

• an author responsible for technical behavior;
• an application owner responsible for compatibility;
• a database or platform reviewer for high-risk operations;
• an operator with authority to abort;
• a cleanup owner and deadline;
• evidence required before each next phase.

Migration tooling should support:

• immutable version history;
• one execution owner or advisory locking;
• dry-run or SQL inspection;
• prohibited-operation checks;
• per-migration transaction control;
• lock and statement timeouts;
• structured logs and metrics;
• separate schema and long-running data jobs.

Do not let every application replica race to apply production migrations on startup. Run migrations as an explicit release step with controlled concurrency and visible status.

Common Failure Modes

Running DDL automatically on application startup

Multiple replicas can race, and a risky migration becomes coupled to service recovery. Use a controlled release job.

One transaction for every step

Long backfills and constraint validation hold locks, retain row versions, and make progress all-or-nothing. Separate compatible phases.

No lock timeout

The migration waits behind production traffic and creates a lock queue. Fail fast and retry deliberately.

Regular index creation on a hot table

Writes stall for the build duration. Evaluate concurrent creation and its resource cost.

Destructive rename in the same deploy

Old instances and background jobs fail during rolling deployment. Use expand-contract.

Unthrottled backfill

Customer traffic, replicas, and vacuum lose resources. Batch, measure, throttle, and pause.

Declaring success when row count reaches zero

The migrated values may still be wrong. Reconcile business invariants and compare representations.

Never performing the contract phase

Temporary compatibility code becomes permanent architecture. Assign cleanup ownership before expansion begins.

A Practical Migration Runbook

1. Classify the change

Identify lock mode, scan or rewrite behavior, object size, dependencies, compatibility requirements, and business criticality.

2. Rehearse

Run the exact operation against production-scale data with concurrent traffic. Record duration, resources, WAL, locks, and failure behavior.

3. Expand

Apply the smallest backward-compatible schema addition using bounded lock and statement timeouts.

4. Deploy compatibility code

Support both schema versions, add feature flags, instrument old and new paths, and verify mixed-version behavior.

5. Backfill

Use an idempotent, resumable, throttled job with progress and correctness metrics.

6. Switch

Move reads or writes gradually. Compare results and keep a rapid application-level rollback.

7. Observe

Wait through a representative traffic and operating period. Confirm no unknown consumers use the old contract.

8. Contract

Remove compatibility code and old schema in separate, controlled changes. Verify backups and retention first.

Final Takeaway

Safe PostgreSQL migrations are built from compatibility and evidence.

Expand the schema first. Keep old and new code working together. Backfill in bounded, observable batches. Switch traffic gradually. Validate data and service behavior. Remove the old contract only after production proves it is no longer needed.

The individual SQL commands matter, but the release sequence matters more. A disciplined expand-contract process turns database evolution from a maintenance-window event into routine continuous delivery.

Understanding why an application is using more resources than expected, whether that is CPU or memory, can be challenging. Logs and aggregate metrics can only get you so far. Thankfully there is a better way: CPU or memory profiling can show you the exact function where your application is using CPU or allocating memory.

Today we are happy to announce support for CPU and memory profiling of Workers and Durable Objects. From the Workers Observability page, you can now request an on-demand CPU or memory profile of an active Worker, inspect it as an interactive flamegraph, and download the profile file for further analysis.

This method of profiling gives you a useful perspective into what your code is doing and how you can improve it in a real-world setting. That’s because the best way to understand your application is to profile it in production.

To try this on one of your Workers, you have a choice of using the CLI or the Cloudflare Dashboard.

To use the CLI, make sure you have the cf package installed, then simply run:

cf workers versions profile latest \
     --worker-id "$WORKER_ID_OR_NAME" \
     --duration-ms 5000 \
     --profile-type cpu > worker-cpu.pprof

To use the dashboard, head to the Cloudflare dashboard and then access the list of Workers on your account by navigating through Build → Compute → Workers & Pages.

Select your Worker and navigate to its Observability tab. You can then use the drop down to select “Flamegraph”:

You can then request both CPU and memory profiles for your Worker on this page. The duration determines how long the profiler should run for on your Worker. You can also select different versions of your Worker to be profiled. Your Worker needs plenty of traffic to be successfully profiled, so make sure you choose a version that has enough traffic.

Once you click the Capture Profile button, the profiler will run for the duration that you have selected, and you will then see a flamegraph representing the results of the profile being rendered. Each rectangle in the flamegraph represents a function call, with the width representing the amount of CPU time or memory being used by that function. You can click on different functions to focus on and hover to see more details about them:

Don’t be afraid to capture a couple of profiles, then look for the widest functions to determine what is using the most resources in your Worker. You might also find that the table view is useful to get a quick sense of which functions are most commonly seen in the profile.

If your Worker is implemented in TypeScript, then you should make sure that you have source maps enabled for your project, otherwise your profile may show obfuscated function names that aren’t easy to understand.

We have already seen multiple teams at Cloudflare using CPU and memory profiling to identify opportunities for optimizing memory and CPU usage, fixing out-of-memory (OOM) errors, and improving performance. Below are some examples of this.

Using CPU profiling to find optimizations

Let’s look at a real example of how you can use CPU profiling to optimize your Workers. We will look at a CPU profile of the Worker that implements the R2 binding. This Worker receives a lot of traffic and so eliminating even the smallest amount of wasted CPU cycles can have a massive impact on its performance.

We set up the profiler to capture CPU traces, with a duration of 50 seconds. Once we received the rendered profile, we saw this:

As mentioned previously, the widest boxes are those which are using up most of the CPU time. Some of them aren’t surprising, for example decryptBlock is R2 decrypting object data on the way out, and fillResponse is just moving bytes. There weren’t any clear ways to optimize these functions.

As some of the boxes are less visible, it can be a good idea to take a look at the table view. Sorting by Samples points us at genericR2JsonReplacer:

This function represented over 5% of the CPU time, but what caught our attention was the function calling itself recursively. This looked like wasted work, and it was.

Even though the replacer was being called by JSON.stringify on every node in the JSON tree, the replacer itself was also walking the tree. This meant that a value nested five levels deep was processed five times. Fixing this enabled us to eliminate the duplicate work and make the genericR2JsonReplacer function 2.7x faster.

The second candidate in the profile that we looked at was a duplicate call to metrics.

When looking at the code, we saw something close to this:

function ship() {
  if (!registry.metrics().length) {
    return;
  }

  // ... more code here ...

  return Response(registry.metrics());
}

The metrics call was heavy enough that just one call represented 1% of the CPU time in the profile. So storing the result of the first call in a variable and then not calling it again saved us a fair chunk of CPU time.

How we used Worker profiling internally to fix memory problems

Internally we had a Worker with a memory problem. P999 memory sat around 133 MB against the 128 MB Worker memory limit. This caused the Worker to be evicted frequently with an “Exceeded Memory” error.

The image above shows a screenshot of the “Errors by invocation status” graph, with the number of “Exceeded Memory” errors clearly dropping as a result of the fixes identified through profiling.

What was hard to work out was what was causing these errors. The errors and metrics pointed at memory as the source of the problem, but it didn’t explain which part of the code caused it.

The team took a heap profile of one of the Workers running in production and opened it with pprof. They saw that their Prometheus code accounted for roughly 66.7% of allocations in the profile. This code was supposed to be disabled, but enough of it was still running that it created a problem. Because it was instrumenting a lot of the code paths in the Worker, it ended up using a lot of its memory, even though the collected data never actually left the Worker.

This code wasn’t suspected as a culprit of the memory issues, because it was thought to have been disabled. The profiler showed that the code was only partially disabled, with the memory cost still being paid as if it was fully enabled.

The team removed the Prometheus code path completely. This quickly improved the memory usage of the Worker:

This gave the P999 about 10 MB of headroom below the 128 MB limit.

Following a profiling request to the edge

Workers have supported local profiling through Chrome DevTools for a while, but it has not been possible to start a profiling session on Workers running in production. While profiling a Worker locally is useful, the Worker does not receive the same kinds nor number of requests as one does in production. So how do we profile a Worker or a Durable Object running there?

One of the best features of Workers is that their requests are routed to the data center that is nearest to the requesting client. This reduces latency, but it does mean that Workers must be replicated across different data centers and metals (physical servers). In addition, when a Worker is receiving a lot of requests in a particular data center, it can be replicated in a particular metal multiple times to handle all the traffic.

Durable Objects add another layer of indirection: they can be placed dynamically and can move.

This and a few other features of Workers means that before we can start a profiling session, we need to first answer a few questions:

• Which version does the developer want to profile?
• In which data center has that version run recently?
• Is the Worker’s isolate still loaded in that data center?
• Does the isolate belong exclusively to the requesting account?
• For a Durable Object, where is the exact live primary actor?

A few of these questions are answered by the client making the profiling request. In most cases this will be the dashboard, but you can also send these requests yourself (or have your coding agent do it for you). An example request might look something like this:

curl 'https://api.cloudflare.com/client/v4/accounts/<account_id>/workers/workers/<worker_name>/versions/latest/profile' \
  --data-raw '{"duration_ms":5000,"profile_type":"cpu"}'

Notably, this specifies both the script name and version that should be profiled as part of the request URL.

Right now, a new isolate is not started to produce a profile, because the goal is to observe real production execution. This means that if your Worker is receiving very little traffic, you may find that it is hard to profile. So it’s important you pick a version of your Worker that is deployed.

Profiling an isolate without stopping the Worker

A profile is useful only if the application can continue to execute while samples are collected. The Workers Runtime therefore holds the isolate lock only for profiler lifecycle operations. For a CPU profile, the process looks like this:

1. Acquire the isolate and its lock.
2. Create a V8 CPU profiler and begin sampling at a one millisecond interval.
3. Release the lock so normal requests can run.
4. Wait for the requested duration.
5. Reacquire the lock and stop profiling.
6. Release the lock and serialize the result outside the critical section.

Holding the lock through the duration of the profile would prevent JavaScript execution from continuing, which would make the profile useless.

Durable Objects

Durable Objects are different from Workers. Regular Workers are stateless: no one Worker is special or different from the rest, and we pick a running isolate of the requested Worker at random. Durable Objects are stateful, and they have a name. That provides us with the very helpful ability to grab a profile of a specific isolate and a specific object anywhere in the world. When profiling a Durable Object, you can choose by name which object to profile. The runtime will then route the profiling request to the right metal that owns the specific actor and provide a profile of the specific isolate running that object.

What’s next?

On-demand profiling is just the start. It enables you to explore your Workers performance in ways that have not been possible before. But it does have some limitations worth noting:

• You must explicitly start the profiling session. This can mean that you miss important time periods in which your Worker is misbehaving, and where you would like to see a profile to understand what went wrong.
• The memory profiler can only show you allocations that occurred during the profiling window. So if your Worker is allocating a lot during start-up, you will not see it.

In order to resolve this we are already working on something new: continuous profiling. The idea is that we will capture profiling samples for your Worker automatically, so that you can simply explore the profiles in the dashboard. This should make it easier to capture more rare events.

To learn more about profiling in Workers, including the feature presented in this blog post, check out our documentation.

The New Big Ball of Mud: Why Agentic AI Systems Turn Fragile

AI makes building nearly free, so users grow agentic systems piecemeal into fragile messes they no longer understand.

Building systems for working with AI agents is software engineering, whether or not the builder knows it. And increasingly the builders aren’t engineers: a growing crowd of vibe architects with little technical background are crafting complex systems that rely on large context libraries to help them in their daily work. In a recent study of these users, we recognized a familiar pattern that has plagued software architects for decades.

Software engineers learned long ago that systems patched together to meet today’s needs turn fragile and unmaintainable. They learned to plan their architecture so that changing a system later wouldn’t cost more time, money, and expertise than they had. AI makes all these resources feel unlimited, and we saw the result: with nothing forcing them to prioritize, these vibe architects were building big balls of mud. In the AI era, a big ball of mud works surprisingly well — until accumulated complexity makes the system difficult to direct, diagnose, or reconstruct.

What Is a Big Ball of Mud?

A big ball of mud is a system scraped together for short-term functionality, not architected for long-term durability.

The term comes from Brian Foote and Joseph Yoder’s 1997 paper, which observed that most real software at the time looked this way. Such a ball-of-mud system may be useful, but its fragility makes it costly to maintain. Like a shantytown built hastily out of necessity, it needs constant patching and jury-rigging to keep running, rather than holding up like the infrastructure of a master-planned city.

Big balls of mud typically emerge when scarcity meets reality. There’s not enough time, money, or expertise, yet something must be created. Early software teams worked in a “code-and-fix” mode, which sufficed until problems got harder and their big balls of mud became unsustainable. The industry responded with the heavily planned “waterfall” era, modeled on mature engineering disciplines. But when it proved impossible to foresee every need up front, it moved toward an Agile framework.

Our participants were retreading this history. They coded and fixed, with little forethought about architecture or durability, planning to keep molding the clay to fit whatever problems arose.

The difference today is that AI has turned that scarcity into abundance. Building is cheap, so nothing forces a builder to prioritize, and people overbuild. One advanced participant said:

“When I talk to people who say they are advanced in AI, what I often see is what I consider overengineered. It has a lot of fancy words, a lot of flashy tools, but it doesn't get you that much more value for the time you spend in it … Because AI makes it so easy for you to build something, people actually start overbuilding because it feels productive."

Most of our participants built this way, often working alone, guided by the AI itself and by posts on X or Reddit, with subscriptions that put vast numbers of tokens at their disposal.

5 Ball-of-Mud Patterns

Foote and Yoder described the patterns by which a big ball of mud forms and persists. We saw 5 of them in our sessions. These 5 aren’t strict stages — a system keeps growing piecemeal while its owner patches and sweeps — but they tend to appear roughly in the order below.

1. Throwaway Code

It starts when a user has the AI build a proof of concept for something that seems useful. For example, they create a rules.md file to restrain their agents from annoying or problematic behaviors. The file isn’t planned or thoroughly tested; it is a first draft. But the user quickly moves on to other things while the AI diligently continues referencing the context that was created, as instructed. And before long, the user’s workflows have adapted to rely on it.

One participant described a daily automation that summarized his emails, tasks, and meeting notes into that day’s priorities:

“I didn't write the structure myself… I worked with Claude […] to just build the structure for itself […] now I'm kind of […] stuck in the way that it works. And I've never really taken the time to greenfield improve what I did."

These throwaway pieces are rarely useless, so they stay. Participants seemed to have two reasons for not having the AI rebuild them: a lack of time and fear of breaking the existing value. The system was like a pair of worn-out shoes: functional enough to keep wearing, but not necessarily comfortable. Participants kept wearing them instead of taking the time to get a new pair that might do a worse job or not fit as well. Eventually, the shoes wore out, and reconstruction became necessary, but users deferred that investment and risk as long as they could.

2. Piecemeal Growth

As new needs arise, users bolt new functionality onto the original version of their system that they’re hanging on to. One participant’s tool went through three stages:

Stage 1: The participant noticed that her team had limited visibility into work tracked in Asana, so she used AI to build an Asana widget that surfaced each project’s status without requiring stakeholders to sift through more than 50 tasks.

Stage 2: Then, a new need arose. Stakeholders using the widget assumed projects had stalled because many tasks in Asana hadn’t been checked off properly. So, she had the AI add a dashboard component alongside her widget to show more granularity.

Stage 3: She then saw a chance to improve the whole organization’s visibility into every project. She and the AI merged the original widget and newer dashboard into one tool, adding visualizations and weighted algorithms to predict completion dates and time tracking for finance.

This participant could not write code. Yet, with AI removing the brakes, a small throwaway widget grew into a crossfunctional dashboard. This is piecemeal growth.

Our participants were proud of what they had built, and rightly so. The problem is that the number of ways to extend the system never stops growing — particularly once others have input. Several participants regularly asked the AI itself to suggest improvements, but none worked within planned release cycles. Their creations had no “finished” state, even temporarily — and that is when their systems started turning into a big ball of mud.

One of the most advanced users in the study described his wakeup call:

“[Claude is] always hammering me that I'm stuck in this prerevenue infrastructure loop. And basically, what it tells me is, ‘you're constantly trying to refine the system, and you're not actually going out and using the system.’”

Other participants could not locate, in front of us, important files their systems relied on; they had to ask the AI to find them and explain what they contained. The system had slowly grown to the point where they no longer understood exactly where everything lived, how the AI was using it, or what it contained. If they had to recreate the system, they probably couldn’t.

3. Keep It Working

Eventually, a system breaks, drifts, or goes stale, and the user must patch it, abandon it, or overhaul it. Our participants mostly patched, because they relied on their systems. A startup founder described her feelings this way:

“I might even be dependent on it […] If I experience even a slight amount of challenge […] I ask it to help me instead of actively thinking about it myself […] I think I would definitely feel frustrated if the tool was taken away.”

When one participant's AI desktop application crashed and deleted his chat history, he retrieved the files from his trash, dropped them into his Downloads folder, and began running every session out of it. This is what a big ball of mud looks like.

But there are other ways the system can fail, too. For example, one participant connected Todoist (a to-do app) to the terminal where he primarily worked with the AI. The connection failed regularly. Rather than switching to-do apps or reconfiguring the system, he reconnected it by hand each time.

Luckily, AI lowers the barrier tremendously for changing the system. If the context files that an agent uses aren’t working, fixing them might just be a quick prompt away. This was how our participants maintained their systems: they described the change and let the AI fix files wherever it saw fit. Such a process is easy, but it can erode the user’s understanding of where information lives in the system. Users end up dependent on the AI to find, understand, and update it for them. They are more focused on short-term fixes than long-term clarity and durability.

4. Sweeping It Under the Rug

When users can’t or won’t clean up a mess, they contain it: everything goes into one place so the rest of the system stays tidy. In agentic systems, the rug is really a junk drawer — the user keeps needing what’s in it, but only the AI can find anything. One participant emailed himself articles and links throughout the day for the AI to collect. There were too many to process, so they went into a folder called to be synthesized, with no naming scheme. When he later asked the AI to pull relevant material from that folder for a project, it retrieved the wrong articles, and he spent 45 minutes fixing the instructions instead of doing real work.

Another participant let Claude save the files it created as Google Docs in the top-level folder of his Drive. Claude would have filed them anywhere he asked; he just never did.

In both cases, the folder was the junk drawer: everything the AI compiled or created went in, and nothing was sorted.

This pattern resembles Harry Potter visiting the Gringotts bank. Harry knows he has a vault, and he vaguely knows what’s in it, but he can’t find it or access it without a goblin guide. Our participants relied on the AI to navigate their mess, and it did so surprisingly well, which makes it tempting to keep sweeping more and more under the rug. Whether the AI can keep compensating as the pile grows is an open question; the 45 minutes lost to the wrong files suggest it has limits.

5. Reconstruction

Eventually, many systems, AI-centric or not, are rebuilt or abandoned. We saw three triggers for reconstructing an agentic AI system.

The System No Longer Fits the Need

In traditional software, this trigger came sooner because changes were costly and got deferred. A conscientious vibe architect can stave it off longer by having the AI make updates as needs change.

But the fit between user and system can also expire as the user outgrows the system. As users learn what the AI can do, their way of working changes, until rebuilding looks easier than more tweaking. One participant drew a flowchart of how he now wanted every session to run (clarify intent, gather context, orchestrate the model, check the outputs) — steps his current system didn’t support. Unlike his piecemeal first build, this one was planned. He uploaded the flowchart to the AI and asked it to implement it.

Other times, the system was built for someone else. One participant inherited an agentic workflow and a context library from his manager; over time, he found they didn’t perfectly fit how he worked, so he personalized the system. He wasn’t patching holes to preserve the original design; he was renovating it. He called this drift the hardest part of sharing an agentic workflow with another person.

Nobody Understands the System Anymore

As a system grows piecemeal, even its creator can lose track of how it fits together. One participant, a product lead, had built a context library that his whole team was required to use. By the time we met him, he had rebuilt it once and was rebuilding it again:

"At some point around December, this had grown to something that was beyond what I was still able to keep in my head. And so, I had to rework this all to make it manageable again."

This can also happen when a system is handed off to others who did not create it. Someone else’s big ball of mud is hard to understand because you have no mental model for how it's put together. The AI can help the new owner get up to speed, but they will likely want to renovate.

A Better Way of Working Emerges

Sometimes the system works fine, but the user discovers a better way to do the same work. One participant did this in two ways: he regularly picked up new skills and techniques on social media and implemented them in his system. He also regularly asked the AI to review how he worked with it and suggest improvements — and then, usually, to implement them wherever it saw fit.

How Do I Know If I Should Reconstruct?

The stages building up to a big ball of mud aren't a mistake; they're a legitimate strategy that AI has made more manageable than ever. So, when is reconstruction necessary, and how can vibe architects prepare for it?

Start with a very simple question: If you wanted to change how the system works, would you know what to change in the context library the model relies on? The context library is made up of 3 types of contexts with different roles:

• Global context: instructions or knowledge that apply across the system
• Local context: information and instructions for a particular task, project, or workflow
• Ambient context: information streams such as email, Slack, or meeting transcripts that are available to the AI but aren't deliberately maintained as persistent context

If the answer is yes, meaning that you still know which of these types of context needs to change, then total reconstruction is likely far off because you still understand the system well enough to make small renovations without overhauling it.

If the answer is no, the different types of context have blurred together. Nothing is necessarily broken, but the AI is the sole navigator, and you are probably just hoping the system doesn't break.

Keep Tabs on Each Context Type Separately

The key to maintaining the integrity of an agentic system is keeping tabs on each context type. Maintaining each one independently makes it possible to avoid forced reconstruction for a long time, much like how many buildings last for hundreds of years, with each piece slowly maintained or replaced as needed. It's when the context types start blurring together that users accelerate toward system abandonment and reconstruction. And at that point, users will struggle to know exactly what needs to change and will rely heavily on the AI's understanding of the system to salvage what’s working and rebuild from there. It will be difficult to precisely guide the reconstruction process.

Keeping tabs on contexts doesn't mean memorizing the system. It means a few small habits:

• Keep the types of context separate. When one markdown file contains both global and local context, the lines have blurred, making it difficult to parse which parts of the system play which roles.
• Name the context type when directing a change. Even if users don't know exactly where each file lives or every word inside it, knowing what's happening in each context type allows them to tell the AI, "go update XYZ in the global context" rather than "go fix this wherever you think it matters," retaining an understanding of the impact changes have.
• Audit the global context from start to finish once in a while. This context type affects everything the system does, so it should be short enough to review in one sitting. If it isn’t, instructions that don’t apply everywhere have probably crept in; delete them or move them to local context.
• Refresh local context when the outputs suffer. Local context is meant to serve a specific purpose and should be shaped accordingly when it fails to do so.

Efforts like these are renovations, not reconstruction. This is what makes reconstruction a choice rather than an emergency.

All that said, users might decide to reconstruct anyway because their ways of working have evolved. If they can articulate the roles each context type plays, they're in good shape. The goal was never to avoid reconstruction — it's to stay in a position to direct it.

Conclusion

We saw mud everywhere in our sessions — even from the most advanced participants. Foote and Yoder argued that mud is often a pragmatic response to constraints. With AI, many constraints are gone, but the mud remains, because nothing forces builders to ask whether a new piece of the system is worth its maintenance cost or to carefully plan the system’s architecture.

Unlike traditional software, an agentic system is fast and cheap to reconstruct with AI. The hard part is knowing what the new system should be. The participants who seemed best positioned were those who could articulate the limitations of their systems and what they wanted to change. The discipline that matters most is not tidier folders but periodic reflection: knowing what each piece of the system does, whether you still use it, and what you would rebuild if you started over. Mud is tolerable while the AI can still navigate it for you; the time to reconstruct is before it can’t.

References

Brian Foote and Joseph Yoder. 1999. Big Ball of Mud. In Pattern Languages of Program Design 4. Addison-Wesley, Boston, MA, 653–692.

Inside LinkedIn's data context layer: curating trustworthy context for humans and agents

Answering a data question often depends less on writing the query than on finding the right datasets and knowing how to use them correctly. We built a context layer that helps people and AI agents discover the right assets and understand their meaning, usage, and limitations. Suppose you want to know how many members engaged with a new feature last week.

The query may take a minute to write, but identifying the right datasets, applying the required filters, and understanding what each row represents can take a whole day before getting to the result you can trust.

At LinkedIn’s scale with a very large catalog of datasets and metrics, finding the right dataset is challenging, especially when similarly named datasets or different versions contain slight differences in underlying data. This often requires finding someone who knows which dataset to use and what each row represents, but their guidance may remain buried in a chat thread, leaving others to repeat the same search. Business teams rely on analysts to answer questions that inform critical decisions, often requesting additional details or clarifying requirements as they review the results. Each follow-up can require analysts to find additional data and understand its context, prolonging the analysis and delaying business decisions. AI agents can write syntactically correct SQL in seconds, yet confidently select the wrong dataset or miss a required filter. Even when the resulting query runs successfully, its results can be incorrect if the agent lacks the context needed to use and interpret the data correctly. To help people and agents find the right data and use it correctly, we built a shared data context layer that combines automatically generated dataset profiles with team-maintained knowledge about meaning, usage, and limitations.

This post explains how that context is generated, curated, and organized, and how agents retrieve the information relevant to a question without loading everything into their context window.

Generated and human-curated context

At LinkedIn’s scale, manually documenting millions of datasets is impractical, so the AI enrichment pipeline generates baseline context from schemas, source code, and query logs. Each profile describes the dataset’s purpose, fields, join keys, and representative queries, providing a starting point even when its owners have not written documentation.

Teams supplement this baseline with knowledge the pipeline cannot easily capture, including business definitions, exceptions to standard usage, known data-quality issues, and corrections to generated descriptions. For example, a metric may be appropriate for engagement reporting but unsuitable for financial close, or a join key may be valid only for data after a particular date. This guidance helps people and agents use the data correctly. For asset-specific context, generated and curated contexts are stored separately but are associated with the same asset identifier, so a single fetch returns the generated profile alongside any applicable curated guidance. Corrections in curated context can also reveal gaps in the generated descriptions and help identify improvements to the enrichment pipeline.

The generated layer: building baseline context

Teams can capture details about the data, including valid joins, required filters, known gaps, and what each row represents. Yet manually curated context rarely covers the full catalog and can quickly become outdated. The generated layer fills that gap through an AI enrichment pipeline by regularly processing source code, query logs, wiki pages, and other information to generate baseline context for analysis.

From source signals to generated context

The pipeline automatically nominates datasets with selected catalog labels for business criticality, cross-organization sharing, or powering critical dashboards. Teams can also request context through DataHub, LinkedIn’s metadata catalog platform, or supply lists of important assets. These additional paths allow teams to nominate assets that are not covered by catalog-based selection.

The pipeline moves through four stages of nomination, collection, generation, and persistence. Nomination checks for existing context and skips assets that do not need regeneration, while allowing explicit on-demand requests to refresh their profiles. Collectors gather evidence about what those assets contain and how they are used, including masked data samples, query logs, column and metric usage, source code, and wiki definitions. A large language model turns those signals into structured dataset profiles, which the final stage persists in the data context layer, where embeddings are also generated to support semantic retrieval.

The pipeline supports pluggable collectors and asset-specific prompt templates, allowing new context sources and asset types to fit into the same four-stage workflow.

Schema metadata describes an asset’s structure, while query logs and supporting documentation help explain how it is used. For example, recurring joins and filters can reveal usage patterns that are not apparent from table definitions alone.

Mining how a dataset is really used

While the schema lists the columns and their data types, users need to know how they are actually used in queries. Certain unwritten rules are hard to discover by merely looking at the schema and table definitions. For example, people use partition filters to avoid double-counting, use specific keys to join, and include a few specific columns in their queries to answer business analytical questions.

We identify these usage patterns by analyzing runtime query logs and source code, drawing on historical query logs from our shared query engine. The logs include structured input-table metadata that links each query to the datasets it uses.

With so many queries to analyze, we normalize their SQL text to group queries with the same structure and surface recurring usage patterns. Before counting executions, the collector canonicalizes each query by stripping comments, masking literals, collapsing IN lists, and normalizing whitespace. Queries that differ only in their WHERE values then share a canonical form, allowing their execution counts to be combined. We rank these canonical patterns by execution count to identify the most frequently used query structures. To capture usage outside the shared query engine, a second collector searches source code in Spark jobs, Apache Airflow pipelines, analytics scripts, dbt sources, and metric definitions. The code provides insight into each asset’s intended use and identifies assets used primarily by pipelines that might be missing in the query logs.

The collector distinguishes ad hoc queries from dashboard-generated traffic and identifies recurring analytical questions and the joins and filters used to answer them. These usage patterns provide reusable context that helps people and agents find relevant data and apply established query conventions when answering similar questions.

Learning from sample values

Schemas describe a column’s type, but sample values help show whether a string field contains a business category, a formatted date, or a structured identifier. Concrete examples help the model interpret those conventions and generate more useful descriptions when column names and types alone are ambiguous. The collector requests five rows by default to clarify field meanings and value formats, while a separate profiling step can identify common values and their frequencies for selected dimension columns. Profiling uses a recent partition or a bounded sample to complement the row examples with value-frequency information, helping the model describe the contents and patterns observed in each field. The sample-data collector applies masking rules based on field-level annotations and security classifications, along with additional safeguards, before including sampled values in model context.

Human-curated context as code

Automated enrichment provides a baseline from schemas, query patterns, and other available signals, but those signals do not capture everything teams know about their data. Business rules, exceptions, and lessons from past investigations also need to be contributed and maintained by the people who understand them.

To support this human-curated layer, we chose a familiar authoring workflow where teams capture that knowledge in a Git repository and review changes through pull requests. Asset owners approve curated context as they would code changes, while automated skills and checks catch missing or duplicate content before review.

We store user-provided context separately from generated context so that the pipeline does not overwrite the corrections, exceptions, and usage guidance contributed by teams. The pipeline refreshes only the generated layer, leaving curated content intact. The context retrieval will return both contexts into a single view.

Three levels of ownership

Context serves different audiences, from company-wide metric definitions to a team’s investigation playbooks. The repository reflects these differences in its directory layout, organizing context into global, team, and analysis scopes.

• Global: Authoritative, company-wide definitions of shared datasets and metrics, served alongside the generated context for every asset. Ownership resolves dynamically to the asset's catalog owner.
• Team: A team's operational knowledge such as playbooks, known issues, domain definitions, and notes on the datasets and metrics they own, grouped by the product area they align to.
• Analysis: A certified investigation answering a specific question, with the headline finding up front, backed by its methodology, assumptions, data sources, and the query behind it. Authored, dated, and peer-reviewed, so a past answer is reusable instead of stranded in a chat thread.

Within a team's folder, context is organized by type, so both humans and agents know what they are looking at. Every team directory carries the same eight sections.

What a context file looks like

Each context file is plain Markdown with a small block of YAML frontmatter which contains a few fields including summary, a sentence or two that becomes the file's preview wherever it appear. Authors can write the summary themselves or generate one with a provided skill and review it before submitting. If no summary is provided, the context layer generates one automatically.

From a pull request to queryable context

Teams contribute context through pull requests, with ownership and review requirements determined by the scope of the change. Team context leverages team information to identify the responsible team; global dataset and metric context uses asset ownership information in DataHub to identify the appropriate reviewers. After merge, an automated workflow synchronizes changed files into the data context layer, where their full content and summaries become available for retrieval. Derived indexes are maintained automatically.

Once published exclusively within LinkedIn, where agents access it through MCP tools, applications through a gRPC API, and employees through our DataHub deployment. Authors maintain the knowledge in one place rather than copying it into separate agent configurations or prompts.

Agents access the published context through MCP tools that support asset discovery, context retrieval, and index navigation. Relevant team context can be retrieved through embedding-based semantic search or located by browsing indexes, while known documents can be fetched directly by identifier. Tool responses bring the selected content into the agent’s working context without requiring a repository clone or a custom storage integration.

Progressive disclosure: navigating to the right context

Even though the Git repository organizes context by organization and team, an agent still needs a way to identify which files matter. Without an index, it would have to read every file to determine what is relevant, quickly filling its context window and reducing both efficiency and answer quality. The context layer organizes indexes into a tree, with each index containing a brief summary of its node and a map of its children to their summaries. This lets agents explore the relevant branches without loading the full hierarchy into their context window. When content changes, the affected indexes are regenerated automatically from the leaves upward.

Agents can traverse this hierarchy through navigate team index MCP tool, which retrieves an index at the repository, organization, team, or section level without requiring a local checkout. The agent reads the returned summaries, selects a relevant child, and makes another call for that scope. Once it identifies a document, it passes the document’s identifier to fetch team context to retrieve the full content. Agents with a local clone can instead read the corresponding index and Markdown files directly.

Progress and results

Today, people and agents at LinkedIn can retrieve an asset’s meaning, usage patterns, and known limitations through a shared context layer, reducing the time spent searching schemas, reading code, and piecing together guidance from dashboards and chat threads. Our MCP tools bring generated context and owner-reviewed guidance into an agent’s workflow, reducing the effort of gathering information and providing a more consistent basis for deciding which data to trust.

The question we opened with was never just about SQL syntax, but about knowing which dataset to trust, which filters matter, and which caveats could change the answer before filling in the FROM and the WHERE. Our context layer gives onboarded assets a generated baseline, while their owners contribute the business rules, exceptions, and lessons that automated signals alone cannot reliably capture.

Participating teams that had built their own context layers have consolidated their knowledge in our shared context layer. This simplifies management and makes that knowledge easier to discover and reuse across LinkedIn, while teams retain ownership of their domain guidance.

That shared context is already being used at scale across LinkedIn, with a large number of search and context-retrieval requests recorded consistently. Early evaluations also show improvements in agent performance. In one business-area benchmark, the permission-adjusted acceptable-answer rate rose from 25.9% with the earlier analytics-agent experience to 79.5% after the integration. A separate comparison found that the agent’s median end-to-end time to answer a question fell from 10.7 to 4.7 minutes. Together, these early findings suggest potential benefits in answer quality and speed and reinforce our confidence in giving agents access to maintained, relevant context.

What comes next

Authoring context is only part of the work, because its value depends on whether people and agents can find and apply it correctly. We are continuing to improve search, evaluation, and agent integration together so the system surfaces the right assets, identifies gaps, and retrieves relevant knowledge as a question is answered.

The goal is not to eliminate judgment, but to make the knowledge behind it easier to find, reuse, and improve so that finding trustworthy data becomes less of an investigation and more of an everyday expectation.

Agent Mode in the DuckDB CLI

TL;DR: The DuckDB v2.0 CLI has an agent mode that gets AI coding agents to a correct result faster and with fewer tokens. When the CLI detects an agent, it prints compact Markdown tables instead of padded boxes, says clearly when a result was cut, stops runaway queries early, and reports errors as JSON. Long queries announce their expected cost before they start, so the agent can decide whether to wait.

Let's Ask Claude

We asked Claude to measure the effect of agent mode. In the experiment, Claude Code answered the 22 TPC-H questions on a scale factor 100 dataset. The questions were phrased in plain English, without any SQL. Each question was answered three times with agent mode and three times with -no-agent. All 132 runs produced the correct answer. With agent mode, the DuckDB output the model read went from 123.6k to 50.8k tokens, a 59% reduction. Claude summarized the result as follows:

Quote Agent mode fixes the things about a terminal shell that quietly mislead a model. When I answered the 22 TPC-H questions through the DuckDB CLI, agent mode cut the tokens I read back by 60 percent, and every answer was still right.

– Claude Fable 5.1

The savings did not show up in the total cost or runtime. Each turn re-reads the agent’s system prompt, which is much larger than the query output, so the saved tokens amount to about 0.5% of the total input. The agent mode runs also took slightly more turns (237 vs. 224). Claude wrote up the setup, the full results and a follow-up experiment in a report.

Background

The DuckDB CLI was built for a person sitting at a terminal. Its default renderer pads columns so they line up, draws borders around the result and prints the column types under the names:

┌────────────┬───────┬────────┬───────────────┐
│    name    │   n   │ total  │  avg_amount   │
│  varchar   │ int64 │ int128 │ decimal(10,2) │
├────────────┼───────┼────────┼───────────────┤
│ customer_0 │     3 │    777 │        259.00 │
│ customer_1 │     3 │    888 │        296.00 │
└────────────┴───────┴────────┴───────────────┘

For a large result, it shows the first and last 20 rows with a row of dots in the middle:

┌───────────────┬───────┬────────┬───────────────┐
│     name      │   n   │ total  │  avg_amount   │
│    varchar    │ int64 │ int128 │ decimal(10,2) │
├───────────────┼───────┼────────┼───────────────┤
│ customer_0    │     3 │    777 │        259.00 │
│ customer_1    │     3 │    888 │        296.00 │
│       ·       │   ·   │   ·    │       ·       │
│       ·       │   ·   │   ·    │       ·       │
│ customer_4998 │     4 │   1203 │        300.75 │
│ customer_4999 │     2 │    510 │        255.00 │
├───────────────┴───────┴────────┴───────────────┤
│ 5000 rows (40 shown)                4 columns  │
└────────────────────────────────────────────────┘

A person sees the dots and the (40 shown) footer and knows that rows are missing.

Today, many CLI invocations do not come from a person at all. Coding agents such as Claude Code, Codex, Cursor, Gemini CLI and GitHub Copilot run duckdb -c "..." once per shell call, capture stdout through a pipe and pass the text to a language model. This has quickly become a common way to use DuckDB.

This reader differs from a person in a few ways:

• It has no terminal, so there is no scrolling and no pager.
• It cannot ask for more. Whatever was printed is all it sees.
• It pays for every token it reads, including padding and border characters.
• It handles errors by pattern matching on text.

For this reader, the box renderer is a poor fit. The alignment padding costs tokens and carries no information for a model. A truncated result also looks a lot like a complete one. The footer says (40 shown), but a model can skim past that line and then draw conclusions from data it never saw.

Agent mode changes the CLI’s defaults for this reader. This post describes how the CLI detects an agent, what changes in the output and how to try it.

Agent mode ships with DuckDB v2.0.

Detecting an Agent

Agents mark the commands they run with environment variables. There is no single convention for this yet, so the shell checks a list of them:

Agent mode turns on when three things are true: one of these variables is set, stdout is not a terminal, and no output format was given on the command line. If you pass -csv, -json or -markdown, you asked for a specific format, and agent mode stays out of the way.

Two flags override detection. -agent forces the mode on and -no-agent forces it off. The two combine with format flags, so duckdb -agent -csv keeps the agent behavior and only changes how the rows are printed. The DUCKDB_AGENT_MODE environment variable does the same from the environment, for when the command line cannot be changed, such as a script run from an agent’s shell. For example, to manually enable agent mode, run:

DUCKDB_AGENT_MODE=1 duckdb

The .show command reports whether the mode is active and which agent was detected:

agent: claude-code

Not every agent sets a variable. For those, a run that fails while writing to a pipe prints one hint line on stderr, as long as no agent was identified and neither flag was given:

hint: -agent renders errors as JSON and results compactly for AI coding agents (duckdb -help lists all options)

An agent that nobody told about the mode can find it this way the first time something goes wrong.

What Changes in Agent Mode

Everything below is a default. Explicit settings such as .mode, .maxrows, EXPLAIN (FORMAT ...) or anything in your .duckdbrc still take precedence.

Compact Tables

Results are printed as Markdown tables without alignment padding, with the column type in the header cell:

| name:VARCHAR | n:BIGINT | total:HUGEINT | avg_amount:DECIMAL(10,2) |
|---|---|---|---|
| customer_0 | 3 | 777 | 259.00 |
| customer_1 | 3 | 888 | 296.00 |

A model reads the | separators and does not need the columns lined up. Since this is still valid Markdown, a person reading the agent’s transcript gets a rendered table. On typical results this format is 25 to 65% smaller than the box output, with the largest savings on wide results.

Large Results

Printing every row would avoid silent truncation, but a million rows is 17 MB on the pipe. Agent mode prints a result of up to 1,000 rows and 10,000 bytes in full. Anything larger becomes a sample of the first and last 20 rows around an explicit marker row, with a footer that says what happened:

| 19 | customer_19 | 13.3 |
| … 4960 rows omitted … |
| 4980 | customer_30 | 86.0 |
...
first 20 and last 20 of 5000 rows (.maxbytes 0 for all), hash 7389c2f5ff3cefb6

The footer also tells the reader how to get the rest. The limits can be changed with .maxrows N and .maxbytes N, where -1 or 0 means no limit. Long cell values are cut at 500 characters with a visible marker such as …(+4500 chars), adjustable with .maxcellwidth N.

Stopping Early

The compact table needs no column widths, so the result can be streamed. Once the cap is reached, the remaining rows are only counted and hashed. After 100,000 more rows, the query is stopped and the count is reported as a lower bound. For example:

FROM range(1_000_000_000);

This returns in 30 milliseconds with the footer first 20 of at least 102400 rows (query stopped early; .maxrows -1 for all).

A Hash of the Result

The footer includes a hash of the entire result, covering the rows past the cap. The hash does not depend on row order and distinguishes NULL from the string 'NULL'. An agent that wants to know whether a result changed after an edit can compare two footers instead of printing two results.

The footer is printed for empty results, for results of 10 rows or more, and whenever the output was capped. It is left out when the query was stopped early, because the hash would not cover the whole result.

Errors as JSON

Errors go to stderr as JSON, using the existing errors_as_json setting. Errors raised by the shell itself, such as an unknown dot command, are wrapped in the same structure:

{"exception_type":"Binder","exception_message":"Referenced column \"foo\" not found in FROM clause!\nCandidate bindings: \"bar\"","error_subtype":"COLUMN_NOT_FOUND","location":"[7,3]","position":"7","name":"foo"}

The shell drops the candidates field, because the message already lists the candidates. Without that change, a single “no matching function” error weighs 2.7 kB.

Compact Query Plans

EXPLAIN defaults to a new compact format with one operator per line, indented by depth. Estimates, actual row counts and timings go in parentheses, followed by the operator’s properties:

UNGROUPED_AGGREGATE (est=1) Aggregates: count_star()
  HASH_JOIN (est=0) Join Type: INNER; Conditions: range = range
    FILTER (est=2) Expression: (r > 2)
      RANGE (est=10) Function: RANGE
    FILTER (est=1) Expression: (s > 2)
      RANGE (est=5) Function: RANGE

EXPLAIN ANALYZE starts with a summary line such as QUERY (time=0.0012s, read=1.2 MB). This is a regular format, so you can use EXPLAIN (FORMAT compact) outside agent mode too.

Cost Estimates and Progress

When the planner expects a statement to read a million rows or more, its estimate goes to stderr before execution starts:

estimate: ~7501215 rows read (lineitem ~6001215, orders ~1500000), ~4 rows returned

The agent can then decide whether to wait or to interrupt and rewrite the query. Smaller queries print nothing, since the line would only be noise. While a long query runs, the terminal progress bar is replaced by a plain line on stderr every 5 seconds:

progress: 42% (elapsed 12.0s, remaining ~16.5s)
progress: done (elapsed 28.7s)

Listing Tables

.tables prints one line per table, with an approximate row count and the columns:

memory.main.t1 (table, ~15 rows): a INTEGER PK, b VARCHAR

The usual box layout is hard to read through a pipe, and this gives an agent the schema in a single call.

The Startup Line

When agent mode turns on by detection, the shell prints one line on stderr before running anything:

duckdb agent mode on: CLAUDECODE is set and stdout is not a terminal; -no-agent turns it off, .help agent explains the output

The line states why the mode is on and how to turn it off.

The first version printed three lines of explanation (634 bytes) on every run. That was larger than most query results, and a model keeps its context between runs anyway, so it only needs the explanation once.

The longer text is available via .help agent. It describes the output rules and settings, and it points to a few features an agent might not find by itself:

• SET max_execution_time = <ms> to put an upper bound on a query’s runtime
• DESCRIBE <query> to get the result columns without running the query
• SUMMARIZE for a quick profile of a table or query
• .tables for the schema
• duckdb_functions() for function documentation

If you do not want the startup line, add .startup_text none to your ~/.duckdbrc.

Trying It Out

If you use a coding agent, there is nothing to configure. Agents such as Claude Code, Codex, Cursor, Gemini CLI and GitHub Copilot already set one of the environment variables from the table above, and they capture a command's output through a pipe rather than a terminal. Those are two of the three conditions for agent mode.

So as long as the duckdb binary on the agent’s path is v2.0 or newer and the agent does not request a specific output format with a flag such as -csv, -json or -markdown, the mode turns on by itself. A format flag is treated as a deliberate choice, so it switches agent mode off: the CLI assumes you want exactly that format and does not override it.

To see what the agent sees, run a query through a pipe with the flag set:

duckdb -agent -c "SUMMARIZE FROM 'my_data.parquet'" | cat

Conclusion

With agent mode, an agent spends fewer tokens per call and does not act on partial results as if they were complete. That means fewer steps to a correct result.

The current behavior is a first attempt, and the conventions around agents are a work in progress. If your agent is not detected, or the output confuses it, please open an issue or comment on the pull request. Transcripts of an agent being confused are especially useful.

Further Reading

• Shell: agent mode, render for AI coding agents when one is detected (pull request #26167)
• Shell: DUCKDB_AGENT_MODE=1/0 forces agent mode on or off (pull request #26416)
• Shell: proposal to add .mode llm (Carlo Piovesan’s proposal, the basis for the byte budget and early stop)

Google appears to be getting closer to the public release of Gemini 4 Argon. Recent Antigravity changes introduce references to the model in its selector alongside three context configurations: 256K by default, 512K consuming approximately 1.3x quota per turn, and 900K consuming around 1.8x quota per turn. These configurations resemble existing options for Gemini Flash models, although the exact relationship between context size and reasoning effort remains unclear.

Meanwhile, Gemini Web now supports low, medium, and high thinking effort settings across available models. This change could help prepare the platform for Gemini 4. Google officially announced Argon on September 30, initially granting access to selected cybersecurity partners, with paid API customers and Google AI Ultra subscribers expected to follow.

Business Insider has also revealed that Google employees are internally testing additional Gemini 4 checkpoints. According to early employee feedback, one of them, Carbon, reportedly feels comparable to Claude Opus 5.5 for coding. The report identifies Barium-B as the checkpoint selected for the public Argon release, while Carbon represents a separate and potentially more capable iteration.

Carbon is reportedly being tested through Jetski, Google’s internal name associated with Antigravity. We recently discovered another unreleased Antigravity feature called Chief of Staff, an agent previously known as Concierge. Together, these developments point to Google’s broader focus on autonomous coding agents powered by increasingly capable Gemini models.

The comparison with Opus 5.5 is particularly noteworthy considering Anthropic’s current position in agentic coding. However, it remains unclear whether Carbon will become a future Argon update or a separate Gemini 4 release.

Argon’s wider rollout could potentially begin as soon as next week, although Google has not confirmed a date. With expectations surrounding Gemini 4 already high, its real-world coding performance and availability across Google’s products will be closely watched.

Sources and related context

• Google announces Gemini 4 Argon: Supports: Google's September 30 announcement confirms initial access for trusted cyber defenders, followed by paid API customers and Google AI Ultra subscribers.
• Business Insider reports on Google's internal Carbon testing: Supports: The report supports the employee comparison with Claude Opus 5.5, Carbon testing through Jetski, and Barium-B's selection as the public Argon checkpoint.

AI Scaling vs Human Scaling

Do AI models scale as well with more tokens as humans do with more time?

That is, does an AI get the same benefit from 10x the number of available tokens as a human does from 10x the amount of time to do the task? Or might the AI require a larger scale-up of tokens (such as 20x or 100x) to match the performance improvement when scaling up the time available to a human by 10x?

People often think of giving AIs more tokens to complete a task as akin to giving a human worker more time — especially when those tokens are used to extend the AI’s sequential chain of thought or tool use. It’s a useful analogy, but it isn’t clear whether the AI systems are as efficient as the humans are at turning these extra resources into superior performance.

We often lean on this analogy when trying to understand AI’s current capabilities and how they might change. For example, we might think of scaling up AI training as making the AI more knowledgeable and skilled, while scaling up chain-of-thought gives that model more time to complete the task. But if AI systems require a 100x scale-up of tokens to match the improvement a human gets with 10x the time — and a 10,000x scale-up of tokens to match the improvement a human gets with 100x the time — then this analogy has broken down. As we shall see, this would also have consequences for whether AI systems are on track to be able to improve themselves strongly enough to trigger a runaway intelligence explosion.

It’s an important question, but I haven’t seen much good data to test it either way. The data is rare because it also requires measuring human performance on the suite of tasks, including tracking how well the humans do at a wide range of time budgets.

In this case, the humans gain about 4x as much score per doubling of the amount of resources as the AIs do. If this scaling advantage were sustained, it would mean that to get the same improvement a human gets from having 10x as much time, the AI would require about 10,000x the resources.

However, this chart has some serious limitations. A key weakness is that it used very early models, which are known to have short time horizons (only o1-preview is even a reasoning model). The authors made up for this by allowing a length of time (e.g. 16h) to represent either 1 try taking 16 hours, or the best of 2 tries taking 8 hours, or of 4 tries taking 4 hours etc — whichever setup works best for that agent. It is therefore a mixture of chain of thought scaling and best@k scaling. And something similar was also done for the human times. No human was allowed to work for more than 8 hours so the data points for human scaling which appear to show the curve flattening are really showing the effects of parallel labour (even if this had worse results than a single longer attempt would have).

So we are left with results that are suggestive of humans scaling much better with time than AI models do with more inference compute, but we’d really like to see it for more recent models which have been trained to allow use of very long chain of thought.

The best data I know of on this doesn’t directly compare humans’ performance on the same benchmark. Instead, it measures AI performance in terms of human performance. It does this using time horizons — measures of AI performance in terms of how long such work would typically take a human.

It was intended to show that across a range of agents, allowing them more tokens produced a longer 80% time horizon. This horizon is the duration of task — as measured by how long it takes a human — which the AI can successfully complete 80% of the time. For this chart, the tasks were AISI’s internal suite of capture-the-flag hacking tasks.

It is now immediately obvious that all of these models are scaling sub-linearly: 10x the number of tokens is giving less than 10x the estimated (human) time horizon. Yet each trend line is roughly straight, which on a log-log plot means the time horizon is (roughly) proportional to the token budget raised to some power. That is:

$H_{80} \propto T^\gamma$

Where $H_{80}$ is the length of the 80% time-horizon, $T$ is the number of tokens, and $\gamma$ is a parameter measuring how quickly the horizon grows as tokens are increased.

If $\gamma = 1$, that would correspond to linear scaling of horizon length, but here these lines all have $\gamma < 1$, which means horizon length is increasing sublinearly. Thus, all these AI systems would appear to be scaling less well than humans.

As expected, the values of $\gamma$ are all less than 1, indicating that all models are scaling worse with more reasoning tokens than humans do with more time. But we can also see that there is a broad upward trend in the value of $\gamma$ as we go down the table, and the models are listed in order of release date. In less than a year, they have gone from requiring 10,000x as many tokens to scale their time horizons by 10x (a similar ratio to the first chart), to needing just 20x as many tokens.

So the AISI chart suggests that the models (up to May 2026) scale worse with more inference compute than humans do with more time, but that the relationship has been improving over time.

There is clearly some improvement over time here, though there isn’t enough information to distinguish whether the trend is rising throughout this period, or whether we’ve reached a plateau at roughly $\gamma = \frac{2}{3}$. So it is hard to predict whether $\gamma$ is currently about $\frac{2}{3}$, or whether it has reached 1 (and thus parity with how humans scale). Indeed it is entirely possible that improvements to training will produce AI systems with $\gamma > 1$, scaling better than humans. We definitely need more work tracking this.

I want to stress that all of this data and analysis is still just suggestive as there are many limitations with both the data and my quick analysis:

• It is just one experiment with one set of tasks
• The data points only cover very long chains of thought — much longer than the contexts of the models and thus requiring multiple rounds of compaction. It is possible that the slopes of these curves are lower in this multiple compaction regime than they’d be when using numbers of tokens that fit within their contexts.
• There is substantial uncertainty about the slope of each model’s curve and whether they are still increasing for later models. (The error bars in my chart above understate the total uncertainty as they are just showing the error in measuring the slope of the curves.)
• It isn’t clear if human-equivalent time-horizons are a good measure for human performance. They are quite a complicated construct and can be misleading. I’d be happier with a chart with actual human data.

But it is suggestive of AI systems scaling differently with more tokens than humans do with more time. In particular, it suggests that AI systems previously scaled much worse than humans but have improved as newer models are released, to a point where it is possible they match or exceed humans’ ability to scale with more time.

Implications

All of this is relevant to how we intuitively understand inference scaling. If the central analogy (of increasing chain of thought as being like increasing the time allotted to a human) is flawed, that is important to know. Or perhaps we should think of the analogy as correct, but that the AI systems are less good at using more time than humans are — a deficiency that it is important to measure, to track over time, and to project into the future. And we shouldn’t forget that they might end up scaling even better than humans.

One particular implication that interests me is the connection to recursive self-improvement (RSI). Researchers studying this use a parameter called $r$ to represent the strength of the feedback loop where a more intelligent AI is even more capable of improving the intelligence of its successor. In their modelling, when $r > 1$ the process explodes, with performance rising towards a vertical asymptote. Researchers define $r$ as equal to $\frac{\lambda}{\beta}$, where $\lambda$ represents how well research scales with parallel labour (more workers) compared to serial labour (more time), and $\beta$ represents the diminishing returns to more research.

If we let $N$ be the number of agents in a swarm:

$T \propto N^\lambda$

$H_{80} \propto T^\gamma$

So:

$H_{80} \propto (N^\lambda)^\gamma = N^{\lambda\gamma}$

If we were to adjust the definition of $r$ to account for this (so that it can convert a scale-up of number of AI agents into the equivalent scale up of serial time for human researchers), it would be $r = \frac{\lambda\gamma}{\beta}$. When $\gamma < 1$, this makes an intelligence explosion harder than we’d have thought, though if in the future $\gamma > 1$, it would be easier. This makes measuring and tracking $\gamma$ look even more important.

Both OpenAI and Anthropic now ship native multi-agent support: a lead agent can spawn subagents, hand each of them part of a task, and assemble what they return. We wanted to measure what that buys, what it costs, and which kinds of work it helps, on tasks that resemble real software projects.

We used Vibe Code Bench, our benchmark for building complete web applications from a product spec, as a test bed. Each app is deployed with Docker Compose and graded by browser agents that click through its workflows. The work splits naturally into a database, an API, an interface, and deployment, and bugs at the seams between those pieces show up directly in the score. We ran GPT 6 Sol and Claude Opus 5.5 on 50 apps, each as a single agent and as a team, at medium and max reasoning effort.

In a team, the lead agent can hand parts of the build to subagents, up to five at once, and subagents cannot spawn their own. Both leads received the same short instruction: split the work into independent parts, delegate each one, then integrate and verify the result; the task prompt, sandbox, and grader were otherwise identical. Score is the share of an app’s UI tests that pass, averaged over the 50 apps. Each of the eight setups ran once.

Key findings

• Teams cost 1.8 to 5.1 times as much as single agents, and only one difference in score was statistically significant. Sol’s medium-effort team gained 7.3 points over its single agent (p = 0.005). The other three differences, from −0.3 to +3.4 points, were not statistically significant.
• For Sol, higher effort raised the score more than a team did; for Opus, neither higher effort nor a team made a significant difference. Higher effort came at a higher price, though: Sol’s max-effort single agent cost 1.6 times as much as its medium-effort team ($4.82 vs. $3.07 per app) and took three times as long.
• Teams finished faster for Sol but not for Opus, and the two leads orchestrated differently. Sol’s lead split the app across parallel subagents within the first minutes and tested the result itself; at max effort, its runtime fell from 38 to 28 minutes. Opus’s lead usually first wrote a shared contract for how the app’s parts would fit together, then delegated in sequential waves of builders and testers, and its teams took up to 2.3 times as long as its single agents.

These observations come from one benchmark of full-stack web apps and may not carry over to other kinds of work.

Cost, runtime, and score

We compared each pair on the same 50 apps with a paired t-test. Only Sol’s medium-effort gain, 7.3 points, is statistically significant (p = 0.005; 95% interval +2.3 to +12.4 points). The other three are not (p = 0.16 to 0.81): +1.4 points for Sol at max effort, −0.3 for Opus at medium, and +3.4 for Opus at max, with intervals of −1.7 to +4.5, −3.2 to +2.5, and −1.4 to +8.3 points.

At medium and max effort respectively, teams cost 2.5 and 1.8 times as much as the single agent for Sol, and 2.2 and 5.1 times as much for Opus. Most of the extra spend is cached input: each subagent has its own context holding the spec and its working state, and that context is re-sent on every model call. At max effort, Opus’s subagents read a median of 224 million cached tokens per app, against 17 million for the lead and 55 million for the single agent. One Opus max-effort team ran out of time while still verifying; its cost of about $189 is estimated from its token counts and included in the mean.

Parallelism saved time for Sol at both efforts, slightly at medium: its team finished in 11.9 minutes against 12.4 for its single agent at medium effort, and in 28 against 38 at max.

Opus’s teams took longer, up to 2.3 times the single agent’s runtime at max effort. They worked in sequential waves, described below, with each wave waiting for its slowest member, and at max effort the median subagent ran for 46 minutes, against 7 at medium.

The extra time was subagent work, not idle time, and longer runs did not score higher. Across apps, the Opus max team’s runtime had no correlation with its gain over the single agent. Four of its runs took more than four hours, and one hit the 5.25-hour limit.

Higher effort or a team

A team and higher effort both raise cost, and their effects on score differed. For Sol, max effort raised the single agent’s score by 11.4 points (p < 0.001), more than the 7.3 points a team added at medium effort. The max-effort single agent also scored 4.1 points above the medium-effort team (89.0% vs. 84.9%), a difference that was not statistically significant (p = 0.08), at 1.6 times the cost and three times the runtime.

For Opus, max effort alone did not raise the score: the single agent scored 89.8% for $23.77, against 91.5% for $4.08 at medium (p = 0.48). Max effort combined with a team produced the highest score in the study, 93.2%, at thirty times the cost of the medium-effort single agent; its 1.6-point lead over that agent was not statistically significant (p = 0.36). Across models, Sol’s max-effort team (90.4%, $8.54) came within about a point of Opus’s medium-effort single agent, at twice the price.

Neither option was a clear default: higher effort gave Sol the larger gain at a higher price, while Opus’s cheapest setup scored about as well as its most expensive one on this set of tasks.

How the leads worked

The two leads received the same instruction and divided the work differently.

Sol’s lead split the app and did the testing itself

At medium effort, Sol’s lead split the work along architectural lines (database and seed data, API, interface, and deployment) and created all of its subagents within about a minute. At max effort it more often split the interface into separate feature areas (13 of 50 opening plans, against 7 at medium) and kept adding subagents later in the run. It also spawned slightly more subagents (5.1 vs. 4.2) and sent them almost twice as many follow-up requests (31 vs. 18 per app).

Sol’s logs record the lead’s commands but not its file edits, so we can’t tell how much code it wrote; its recorded actions were mostly coordinating, building, and testing. It drove a headless browser through the app in 46 of 50 runs at both efforts, running about twice as many browser tests as the single agent, mostly at the app’s public URL rather than at localhost inside the sandbox. At medium effort, the single agent browser-tested only at localhost in 18 of 50 runs, against 4 for the team lead. In its notes, the lead reported catching bugs that only appear when separately built pieces meet, such as an interface sending fields that didn’t match the database schema, or a total computed one way in the interface and another in the database. Each fix went back to the subagent that owned the file.

Opus’s lead wrote a contract first and delegated in waves

Before spawning any agents, Opus’s lead wrote a shared document, usually CONTRACT.md, describing the schema, API routes, and file ownership (42 of 50 runs at medium, 40 at max; the single agents almost never did). It spawned subagents in waves. At medium effort, it usually spawned builders and then, in most runs, a subagent dedicated to end-to-end testing. At max effort, it added a wave: first subagents that built the foundation, then subagents that built features on top of it, then subagents that tested and reviewed the app. It wrote the READMEs itself in most runs and edited application code in 10 of 50 runs at medium effort and 26 at max.

At max effort, the Opus team was larger: 6.8 subagents per app instead of 4.2, a testing subagent in all 50 runs, and about 1,140 subagent tool calls per app, against about 270 for the max-effort single agent. Its subagents ran 36 browser tests per app, six times as many as the max-effort single agent.

Where the time went

Teams and single agents reached their first deploy (the first docker compose up) at similar times. At medium effort, both models’ teams were within a minute of their single agents. At max effort, Sol’s team deployed about five minutes earlier and Opus’s about six minutes later. Opus’s max-effort team then spent two thirds of its run after the first deploy, against under a third for its single agent. Over that stretch, its lead and subagents put a median of 68 agent-minutes (time summed across the lead and its subagents) into testing, against 10 for the single agent.

Where team gains came from

To check whether teams helped more on harder apps, we grouped apps using a run outside each comparison. For the medium-effort comparison, an app counts as harder if the same model’s max-effort single agent scored below 90% on it; for the max-effort comparison, we use the medium-effort single agent. This keeps chance results in the compared runs from deciding an app’s group.

Sol’s medium-effort team gained on about half the apps, most on harder ones. It scored higher than its single agent on 24 apps, lower on 11, and the same on 15. On harder apps it added about 15 points; on easier ones, about 1. At max effort, Sol’s smaller team gain was spread evenly across harder and easier apps.

Where Sol’s medium-effort team gained 20 points or more, it built and checked what the single agent skipped. On these nine apps, the single agents reported that testing had passed even when some features were never built and whole workflows were never tried. The team covered more of the spec and checked its work more carefully: it built those features, and its lead reviewed the separately built parts against each other, catching mismatches such as a form that didn’t fit the database schema or a permission rule that blocked a step.

Opus’s gains and losses were closer to balanced and did not favor harder apps. Its team scored higher on 14 apps and lower on 15 at medium effort, and higher on 13 and lower on 8 at max. At medium effort, it lost about four points on harder apps and gained about two on easier ones; at max effort, it added about three to four points on both.

Other properties we checked had no consistent effect. Apps with more distinct user roles gained 15 points from Sol’s medium-effort team, but no other configuration showed this. Spec length, the number of UI tests, and single-user versus multi-user apps showed no consistent effect either.

Future work

On this benchmark, the effect of a team depended on the model, the reasoning effort, and the app. The open question is which tasks teams handle reliably better than single agents. A team should gain the most when the work splits into parts that can proceed side by side, and the lead can combine them without spending the time it saved on coordination. Tasks with many independent parts and long stretches of separable work, such as large codebases, broad research questions, or analyses spanning many documents, are where we would look next.

Methodology and limitations

GPT 6 Sol ran through the OpenAI Agents API, with the API’s multi_agent mode off for single agents and on for teams, and with commands executed in a Vals sandbox. Claude Opus 5.5 ran through the Claude Code CLI in headless mode, with the subagent tools removed for single agents and available for teams. Opus’s team runs also disabled Claude Code’s ten-minute limit on background subagents. Reasoning effort, medium or max, was set through OpenAI’s reasoning.effort parameter and Claude Code’s CLAUDE_CODE_EFFORT_LEVEL. Apps were scored by the standard Vibe Code Bench UI test suite.

Behavioral counts come from each lead’s event stream, and from subagents’ streams for Opus. OpenAI’s Agents API emits two subagent.created events per subagent, so we counted unique subagent IDs for Sol. Browser tests count commands that launch or navigate a headless browser. Runtime is the agent’s wall-clock time in the sandbox. The harnesses expose different amounts of detail. Claude Code logs every subagent’s tool calls. OpenAI’s Agents API event stream contains only the lead’s turn, so for Sol we can see when the lead created, messaged, and waited on subagents, but not what it told them or what they did. Sol’s costs assume that its reported token usage includes the subagents.

Sol’s costs apply OpenAI’s base-tier list prices ($2 per million input tokens, $0.20 cached, $10 output) to reported token usage. Requests above the 272K-token long-context threshold are billed higher, which we could not evaluate because usage is reported per run rather than per request. Opus’s costs are the list-price totals Claude Code reports.

Because each setup ran once, the intervals and p-values in this post reflect variation across apps, not between repeated runs. The team runs also received the delegation instruction, so each comparison is between “subagents plus a delegation prompt” and neither. The two models ran through different harnesses, so comparisons across models mix the model with its harness.

pplx-decider-v1.1-27b

pplx-decider-v1.1-27b is an update of our pplx-decider-v1-27b model, that boost the Decision Index score from 56.4 to 61.56, now outperforming Jev by more than 3.5 points, while using the same backbone. Most of the gains are driven by the lifting of the causal mask and training on more data, including data from tasksource.

Evaluation

The overall score uses the suite's weighting, not a simple category average.

Checkpoint format and serving

This is the native decision-checkpoint layout: a Qwen3_5Model backbone plus readout.safetensors containing a BF16 [255, 5120] decision head. The Transformers class name does not change the base model identity, Qwen3.8-27B.

Use the included inference implementation to preserve the evaluated behavior:

• Full-attention layers use the saved noncausal attention mode. The model's linear-attention layers retain their native behavior. Default causal inference does not reproduce this checkpoint's evaluated setup.
• The supplied DecisionModel.predict applies the saved calibration temperature. When using logits directly, apply it exactly once and normalize over the valid candidates for the current decision.
• This artifact has a separate readout, not a full-vocabulary lm_head. A serving system requiring Qwen3_5ForConditionalGeneration needs a separate export, mapping readout row i into vocabulary row decision_config.json["token_ids"][i]. Such an export must also preserve the attention behavior. If the frontend applies temperature, use the value above.

Usage

Use Python 3.12+, authenticated Hugging Face access to this private repository, and a CUDA GPU with room for roughly 49 GiB of weights plus working memory. Download the repository and install its pinned requirements.txt dependencies.

import sys
from pathlib import Path
from huggingface_hub import snapshot_download

checkpoint = Path(snapshot_download("perplexity-ai/pplx-decider-v1.1-27b"))
sys.path.insert(0, str(checkpoint / "source" / "src"))
from autojev.model import DecisionModel, answer

model = DecisionModel(checkpoint, device="cuda")
question = {
    "type": "choice",
    "instructions": "Which team should handle this request?",
    "criteria": {
        "billing": "Charges and refunds",
        "support": "Technical integration errors",
        "sales": "Questions about buying a product",
    },
}
row = {"state": "My integration keeps failing. Please help.", "question": question}
probabilities = model.predict([row])[0]
print(answer(question, probabilities))

Images

Pass images as file paths, PIL images, or data:image/...;base64 URLs in row["images"]. The processor resizes them while preserving aspect ratio. Each side is rounded to a multiple of 32 px, and the total area is kept between 256×256 (65,536 px) and max_image_pixels. Each 32×32 patch costs one visual token, so an image uses about width * height / 1024 tokens.

max_image_pixels defaults to 2048×2048 (4,194,304 px, ≤4,096 tokens per image). Larger images are downscaled automatically, so you don't need to resize them yourself. To lower or raise the cap:

model = DecisionModel(checkpoint, device="cuda", max_image_pixels=1600 * 1000)

Keep in mind:

• The text and all images of a decision must fit in prepare's 8,192-token limit. Inputs are rejected rather than truncated. With several images per decision, lower max_image_pixels or downscale the images first.
• Training and the reported evaluation capped images at 512×512 (262,144 px, ≤256 tokens). Higher resolutions keep more detail but fall outside the training distribution, and they cost more memory and latency. To reproduce the evaluated setup exactly, use max_image_pixels=512 * 512.
• If you resize images yourself, scale both sides by the same factor min(1, sqrt(max_image_pixels / (width * height))) to keep the aspect ratio.

source/src/autojev/model.py is the evaluated training-run code, with one change: the image-size cap is now the configurable max_image_pixels. It was previously hardcoded at 512×512 pixels. release-manifest.json records file checksums and checkpoint provenance.

Acknowledgment

A large part of the gains are thanks to the tasksource data. If you also use this data, consider citing the corresponding article

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}

Our data pipeline also builds on Bespoke Nimble by Bespoke Labs. The data pipeline of pplx-decider-v1-27b was built around human-labelled public datasets that we converted into Choice and Noul decisions with Nimble's converters. v1.1 adds a lot of new data, so Nimble-converted data is now only a small share of the training set. We also use Nimble's published benchmark subsets, pinned by record ID and checksum, to keep those benchmark records and related examples out of training. Nimble is downloaded at a pinned revision when the data is built and is not redistributed here. If you use this part of the pipeline, please also cite Nimble:

@misc{nimble2026,
  author = {{Bespoke Labs} and Sathiamoorthy, Maheswaran},
  title = {Nimble},
  year = {2026},
  howpublished = {\url{https://github.com/bespokelabsai/nimble}},
  note = {Model: https://huggingface.co/bespokelabs/Bespoke-Nimble-9B}
}

Following last week’s release of Clef and Clef-flash, Cloudflare’s open-weight decision models, we decided to bring forth more gifts. Today, we’re releasing Clef-omni, which takes in audio and video input alongside text and image. We also cut the price of Clef-flash so it is now cheaper than Jev, and we made Clef faster.

Although the model game is still early for Cloudflare, innovation and iteration is in our DNA, and we apply these principles to everything we do. In fact, the story of Clef came together over the course of less than a week. We decided we wanted to do something in the decision model space on a Friday evening, trained the model over the weekend, and launched it on Thursday. Even with such a short timeline, we were able to ship performant, high-quality, open-weight models for the community — imagine what more we can do in the future.

For today, we’re excited to keep up the momentum with new additions and improvements to our Clef family of models. This is just the beginning, and we’ll continue to get better, faster, cheaper, and more innovative.

Clef-omni takes audio, video, image, and text input

Our new Clef-omni model is able to take audio, video, image, and text input. This changes the paradigm for decision models, which have been largely text-only since the debut of Jev from TypeSafe. With Clef, we supported images and video frame arrays, but Clef-omni is able to take in audio (wav or mp3) and video (mp4 or webm) alongside text and images.

Instead of setting up cascading pipelines of models that transcribe speech-to-text, or splitting audio and image channels from video, you can just call one model to make decisions across any modality. We are now one step closer to a model that is able to interact with the world as we experience it — through audio, visual, and textual communication, all in one.

Check out our developer docs for the new Clef-omni model. We also released the model open-weights on HuggingFace. For a quick start, here’s how you can send new modality inputs to Clef-omni:

curl https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/run/@cf/cloudflare/clef-omni \
  -X POST \
  -H "Authorization: Bearer $CLOUDFLARE_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "clef-omni",
    "state": "Review the installation: a photo of the unit, an audio recording of it running, and a video of the fan.",
    "images": ["data:image/png;base64,<base64-png>"],
    "audio": ["data:audio/mpeg;base64,<base64-mp3>"],
    "videos": ["data:video/mp4;base64,<base64-mp4>"],
    "questions": {
      "label_visible": {"type": "noul", "instructions": "Is the model and serial number label visible in the photo?"},
      "sounds_normal": {"type": "noul", "instructions": "Does the unit sound like it is running smoothly, without rattling or grinding?"},
      "fan_running": {"type": "noul", "instructions": "Is the fan running in the video?"}
    }
  }'

Clef-omni expands our open-weight decision architecture to natively handle multimodal workflows. We built this on a Qwen3-Omni-30B-A3B-Instruct mixture-of-experts (MoE) foundation, where the model was already capable of processing text, imagery, audio, and video directly within a single pipeline. However, we adopt the primary comprehension backbone while discarding the text-to-speech output components. In production, Clef-omni executes a quick prefill pass across the complete payload, scoring all modalities and valid parameter options simultaneously.

Because we skip output token generation since Clef models are not Large Language Models (LLMs), we remove the overhead of transcribing or captioning incoming files. Media elements map straight into the unified sequence, where video and audio are synced with visual frames for joint processing. Clef-omni then pulls candidate values directly from internal embeddings using our two-stage attention routing: every valid option gathers key evidence from the input (regardless if they are buried in text snippets, visual elements, or audio streams) before field vectors cross-attend across the full context to compute confidence scores. We add a built-in lexical grammar that preserves option semantics, so that we can deliver fast, schema-constrained scoring across every input type.

We train the model with the same techniques as we did with Clef — freezing the Qwen3 backbone, training low-rank adapters (LoRA), and applying our standard post-training approach: combining label-smoothed cross-entropy loss with Brier score calibration. The result is a model that is resilient and designed to succeed regardless of schema variations, field ordering, and prompt structures.

This brings several core upgrades to the Clef model family. Calibrated, schema-bound decisions now seamlessly handle text, images, audio, and synchronized video in a single API call. It's also fast. Text-only decisions return in about 130 ms at the median, image inputs in about 150 ms, and audio clips in a few hundred milliseconds. Even a full 21-second video clip with sound is scored in about 1.5 seconds, all in a single API call.

Our benchmarking shows strong performance across benchmarks, even with new modalities and MoE architecture.

We also benchmarked against the TypeSafe evals to show how Clef-omni performs.

Clef-flash model is now cheaper

We heard your feedback — you want a decision model affordable enough to incorporate into any workflow. We’ve made a few optimizations and can now offer Clef-flash at cheaper prices. We want Clef to be able to make decisions that scale for you across all your agentic workflows, and now our pricing also incentivizes that.

We were able to optimize the model to the point where Clef-flash is now cheaper than Jev while still being performant and high-quality. Check out the developer docs for the most up-to-date pricing information at all times, including more detail on how image and audio modalities are converted to input tokens for pricing.

• Clef-flash – originally $0.09 per M input tokens, now $0.038 per M input tokens
• Clef – remains at $0.24 per M input tokens
• Clef-omni – launched today at $0.15 per M input tokens

However, one trade-off we have to make in order to have cheaper pricing is to cut the context window of Clef-flash. The hosted version of Clef-flash now has a context window of 24k, instead of 64k as previously advertised. The model weights on Hugging Face are untouched and were trained to support a 256k context window should you choose to self-host it.

From our usage data, we see that only 0.24% of requests exceed 24k input tokens. Based on this data, we decided to make the call to drop the context window to 24k for Clef-flash in order to price it at a more accessible entry point. Our Clef model remains at a 64k context window, and we encourage folks with larger context needs to switch to Clef instead of Clef-flash.

Clef model is now faster

Speeds are also faster for the Clef model, with new optimizations to the hosted version of Clef on Workers AI. No new model weights are released because most of the optimizations we made are at the serving infrastructure layer, not materially to the model weights or architecture. Check out our new speeds below:

One of the optimizations we made was to move to SGLang for serving the model. We worked with the SGLang team and made pull requests to incorporate Clef (PR #42721), which will be released in SGLang 0.5.22. You can also self-host Clef since we released the weights, and the new SGLang launch commands are available on our updated Hugging Face repo as well as the SGLang cookbooks.

How are people using Clef?

At Cloudflare, we’ve had teams experimenting with Clef as a model that can help classify over general domains. The coolest thing about watching teams adopt Clef is that the capability to do detections and classifications have moved into the model layer, where it is accessible for anyone to start incorporating decision models into their use cases. Previously, even with zero-shot classifiers, the small models may not have been powerful enough to classify across any domain without further tuning. You might have needed to stand up a specialized machine learning team to help train over your particular domain, and gather a corpus of data in order to train a better model. Now, all you need to do is call a model to instantly parse through input data with precision.

A few highlights of how teams at Cloudflare are using Clef: Our public GitHub docs repo uses Clef to instantly detect and close spam issues. EmDash, our content management system, moderates plugin libraries for phishing. Our data loss prevention team is scanning data for potential personally identifiable information (PII) like government IDs. And our threat intelligence team is using Clef to help detect malicious domains.

Give it a try!

We’re excited to bring you more from the Clef family of models, including new modalities like audio and video, as well as making the model more accessible with cheaper pricing and faster speeds. Clef is fully Jev-API compatible and available via AI Gateway, so it is as simple as changing the model ID to try out Clef today. Check out our developer docs for more information.

Thank you for all the love and warm reception around Cloudflare’s first models — we are focused on bringing you more. Try out our Clef family of models, and tell us what you think.

Decart's software makes it easier for developers to switch between different AI chips.

We just let in every Pro and Max user from the Claude Code Projects waitlist!

If you're new to Claude Code Projects, here's a 4 minute walkthrough to get you started:

A project is an ongoing conversation where Claude coordinates the work for you, running each task as its own thread in parallel.

Docs: code.claude.com/docs/en/claude-projects

Projects are in beta. If you haven't signed up yet, join the waitlist. We're letting in more Pro and Max users as capacity allows.

claude.com/form/projects

Borrowing an idea from one research discipline to solve a problem in another is an age-old scientific trick. However, such clever leaps are rare because researchers don't tend to spend their time researching fields outside of their expertise. AI can easily hop between fields. BootLoops is a harness that can spot and solve problems in seemingly unrelated fields that share the same mathematical structures at massive scale. Researchers recently released 36 preprints produced with BootLoops covering a wide range of topics, including figuring out the age of sunspots, analyzing government health policies, and attempting to decipher a mysterious medieval manuscript.

When Ralph was published last year, I sat on it. People who have been reading this blog for a while know this. I first came across the idea of loop engineering in early January last year, sat on it for four months, and then started writing a lot, begging people to pay attention.

When Ralph and Loop Engineering went viral earlier this year, almost a year later, it really weighed heavily on me. Life's been pretty good lately, but it wasn't always historically, so I took some time off. I spent six months traveling around the world with no income, and I delivered the talk below almost 17 times in various cities.

Our profession is at a crossroads, and this is the last time I'll say it.

Usage of AI in the profession of software engineeering is no longer optional.

If your employer has banned the use of AI, you should quit and find a company that is already going through an AI transformation.

If you're at a company going through an AI transformation, I understand you'll have mixed feelings about this. You might not like it. But please understand that this is the new normal. Artisanal handwritten code is only possible for retired tech executives and as a hobby at home, but it's no longer a way to make money.

You really have two choices as an employee.

You can push against the grain and refuse to use AI, but that's just self-harm. You trade time and skill for money. The tastes and desires in the employment market have moved on.

From my position, I have a clear view of what's going on.

Every morning, I wake up to either a WhatsApp message or a LinkedIn DM from people who are going somewhere, sending their thanks. Like, don't thank me; thank yourself. You invested in yourself, and because you did, you just got promoted to a vice president role. You did all the work.

Hi Geoffrey. I wanted to drop you a message to just share what a profound impact Ralph’ing had had on me and the company i work for. I first heard of it back in Jan 2026 on social media and thought it was an outrageous concept, the idea of letting this clumsy persona rip through my codebase, so i just ignored it. My biggest frustration with AI has always been it never bought exactly what I wanted to life, it would get the first parts right, then it starts cutting corners, and I end up with a real mess at the end. That was until when I watched sonnet 5 loop through and churn out a product in about 6 hours, so for whatever reason I decided to come to your blog and read about Ralph directly - and I’d totally misundertood it. It took quite a bit of experimenting, however im now at a place where I’m literally delivering paid for dev worth 100K + for 3K in sonnet tokens in brownfield projects. Im re-platforming entire stacks in about 2 weeks that we used to pay contractors half a million for - the game has totally changed, and the inplications are both amazing and terrifying at the same time. Anyway, just wanted to say thank you, because at least for me and the company I work for, you publishing the Ralph paper has forever changed the game, and I hope that one day I’ll get to buy you a drink to say thank you. All the best.

So it's time to address the elephant in the room: people managers.

If your team is too busy doing their new job to experiment with AI, you're preparing them to be replaced. Maybe you're okay with this, but hopefully you're not, because it should weigh heavily on you if you truly care about your team.

Sometime in the next six months, you're going to have to start having some hard conversations about moving people on from your company. You should already be putting your employees on a vitality curve and ranking them. Naturally, competent employees who have been using AI will be ahead of competent engineers who haven't.

At this stage, some software developers are not going to make it, and that's okay. You can't lead a horse to water and make them drink. It's their choice. You need to put yourself as a manager first and role model the new performance baseline.

If you don't want this to happen to your team's members, start by carving out time for professional development. The most concrete task I can recommend is for software developers to build their own agent, learn the magic tricks behind Claude Code, and explain how it works under the hood.

You see, you're not a senior software engineer in 2026 unless you can rebuild Claude Code (the most basic of agents). Senior software engineers are senior by definition because they can mentor and teach the incoming generation. There's a lot to teach about the craft of software engineering; there's a lot of information and experience about software failures and how software should be designed, colloquially known as taste, that the incoming generation will need to be mentored in, but you won't earn their respect if you have an attitude that AI is evil and should be avoided. If you can't earn the respect of the incoming generation because of your poor attitude, what value do you bring, and why should an employer keep you employed?

I've never seen this before in my career: 28-30 year olds who refuse to use AI coding tools.

You show them what they can do augmented (not replaced) with AI and you see in their eyes that they have no damn clue of what's happening.

You can't work with these people anymore. Time…

— Ivan Burazin

You know how ants leave the colony before they die?

Developers make posts like this before AI makes them unemployed.

— Theo - t3.gg

interviewing as an employee in 2026

If you're interviewing as an employee in 2026, understand that there is a narrative that it's hard to get a job and it's pretty tough out there. This is true if your expertise is limited to typing code into an IDE, but it is not if you have provable expertise in AI.

If you've got something to show, like a talk, an open-source project, or anything tangibly demonstrable that you've been curious about the transformation in our industry and have been investing in yourself, you should be able to land a job in a couple of days in this market, as it's cooking right now.

Please understand that your edge right now is understanding the possibilities and being curious. The world lacks curious, ambitious people. That's your edge; that's your alpha. Position yourself as that, and be way more ambitious than you are right now. You need to think bigger.

At whichever company you end up at, be an ambassador for AI across your team and organization (and by that I don't mean you should be a maximal AI glazer). I mean, share what works, experiment, and help others through this transition. Generally, just raise the bar for everyone around you and be a good human.

being a hiring manager in 2026

It's been almost two years since my oh-fuck moment. Since then, we've seen the rise of "model-first" companies where most of the software is generated by computers, not humans.

This means that two years have passed, and those who have been curious and invested in themselves have one hell of an edge over someone just getting started today.

One of the most fundamental questions I’ve been pondering on over this last year is “how do you identify skilled operators” aka “how do we even hire software engineers now now that LLMs can smash leetcode?”

I propose the following categories and buckets as a crude ruler on top of your standard software engineering knowledge tests.

• unacceptable
  • A poor attitude toward willingness to use AI. I get it. The labs ripped off society's commons, and I don't like it either, but it has happened. You should sniff out these perspectives when interviewing a candidate and filter candidates out through behavioral questions.
  • Any form of deceptive behavior. If they're using AI in the interview process to deceive, that's an instant no-hire because deceptive behavior is unacceptable.
  • No tangible, demonstrable experience with AI.
• acceptable
  • You should be able to pull them over to a whiteboard and ask, at a high level, how a coding agent works under the hood. This is a baseline curiosity test to see whether they've been paying attention and developing their skills over the last couple of years. You want to understand whether they know what a context window is, how tokenization works, and how inference works from a systems-design perspective. If their knowledge depth taps out and all they can do is talk about skill packs, then pass on the candidate. Employers should prioritize producers, and producers know the baseline underlying data structures and system design of how an agent works because that knowledge lets them automate a business function. If they don't have this knowledge, they're just a consumer, so you should likely pass on them until you find someone with producer knowledge.
• ideal
  • Tangible and demonstrable experience with AI. They should have a whole bunch of side projects they have built; this may seem unfair, and I agree. Before AI, this was an unfair standard, but these subscriptions have been available to everyone. They're subsidized, and everyone I know who understands what's going on has so many cool things they're dying to show people.
  • When you ask the question about what coding harness they use, like, there are only really two acceptable answers. One is that all coding harnesses are fungible, and they use whatever they can to get tokens the cheapest, or their eyes light up and go, I use my own. Do you want to see it? Do you want to nerd about it? Like, here, I'll show you. This is the modern age of the age-old interview question: "Tell me about your home lab"
  • If you walk away from the interview and learn something about AI you didn't know, you should hire that person on the spot before someone else does.
  • If they lead people, they should be able to go really, really deep on the processes that failed because of a company's AI adoption. They should also cite examples of what they personally changed within their organization to enable AI adoption. You should background-check these stories to ensure people aren't telling porkies.

There's something else that I need to put forward. Your existing hiring pools may need to be wiped because your current interviewers may lack the expertise to differentiate between these categories. For example, if someone has a poor attitude toward AI and no provable experience with AI, and they're not in the ideal category, remove them from your interviewer pool (and/or company), because you need to put your best foot forward to identify, attract, and hire the best talent out there in 2026.

Remember, the candidate is also interviewing you. They know what's going on, and they want to ensure that choosing you as an employer de-risks their lives.

ps. Watch this podcast with Bill Gates and internalize it

Well, Jevons is just a referral to the fact that parts of the economy are subject to demand elasticity. And yes, in the case of software, if you’re, say, three times as fast and there is still some role that only humans can perform, then as you lower the cost, you induce demand. So it’s fair to say that the equilibrium today so far for software is not a loss of employment.

When we invented radial tires that lasted four times as long, for some weird reason, people didn’t drive four times as much. And factories that make tires employ a quarter of as many people. When you replace people in Amazon warehouses with robots, people don’t buy more because of that.

So anybody who’s numeric can say to themselves: What portion of the economy is subject to demand elasticity, and what are those tasks that will still be human-necessary?

As soon as you’ve completed the entire task, it doesn’t matter that there’s demand elasticity. That goes into the token budget. It doesn’t go into the human salary budget.

The cost for some of these things is so much less than the cost of the human labor, and so as you cross reliability thresholds, both with white-collar and humanoid robots, you destroy jobs and you leave no high ground.

Innovation in the past — you have a tractor. Fine. Let’s build Disneyland and employ a lot of people there. You don’t have that in the broad economy. The people who say there will be net additional jobs, I don’t understand what they’re thinking.

pps. some extra data for engineering managers

AI is no longer optional. Every leader we interviewed said, one way or another, that refusing to use AI is now disqualifying; the comparison we heard more than once was refusing to use an IDE. The gains are real. Organisations told us about three times the output per engineer, a six-month project delivered in two, an incident diagnosed in seven minutes that would once have taken four hours. What changes most is what's possible. The default answer to a new idea is no longer "no", it's "let me check", and work that used to die at the bottom of the backlog now gets done.

‼️ ppps. socials.

Can you please boost and share this blog post with someone you think needs to hear it? It is both the most and the least we can do. We can't force a horse to drink, but we can tell horses that water exists, and that they need water to be employed in 2026...

• Twitter
• LinkedIn
• BlueSky

So long, Spokes: GitHub rewrites storage to restore reliability, just in time for agentic hordes

Responding to unprecedented demand from AI agents, GitHub is rebuilding its Git storage architecture from scratch. Early tests with the new architecture show a promising 35x write improvement in internal tests.

GitHub will need a miracle of that proportion just to keep up with customer demand.

The plan involves decoupling writes from reads, offloading maintenance, and letting object storage manage redundancy.

Most impressive is that the redesign is happening under the hood, with the goal of a non-disruptive change that won't alter developer workflows, review processes, or security controls.

Running an enterprise-grade code repository means supporting “large engineering teams running busy CI pipelines alongside growing fleets of agents. Supporting these teams means building Git infrastructure for sustained, concurrent reads and writes at a scale few repositories reach today,” wrote Brian Celenza, GitHub principal software engineer, in a blog post describing the updates.

Celenza did not offer a timeline for this ambitious migration.

When GitHub’s customers shifted their coding focus to AI agents, activity on the platform spiked significantly. Between September 2025 and August 2026, GitHub traffic doubled, from 218.2 billion events per month to 473.3 billion. In September alone, commits hit 7.38 billion, a 5x increase from September 2025.

This shift in traffic is stressing the system. According to the GitHub Availability Report, the service suffered 10 incidents in April alone that degraded performance. Another nine incidents landed in May, with each generating more bad publicity than the previous one.

So long, Spokes

GitHub's current storage architecture, nicknamed Spokes, uses a three-phase commit protocol, storing full repo copies across multiple local disks and requiring a quorum of replicas to acknowledge each write. While this approach ensures strong reliability, it bottlenecks write speed, as every push is bounded by the slowest replica required for quorum.

The new architecture writes a commit only once, to Azure Blob Storage, an object store service that automatically handles replication (and hence redundancy) on its own, eliminating the quorum bottleneck. GitHub also gooses performance by separating read requests into their own channel, a job handled by lightweight compute workers. The only coordination between reads and writes is over the reference branch pointers.

Maintenance duties – such as compaction and garbage collection — are moved from the serving path to operate as background processes, further reducing system lag.

Git in the age of agents

GitHub seems to be on the same page with others in rethinking Git for the agentic age.

Former GitHub CEO Thomas Dohmke launched a Git-hosting service called Entire, which offloads agent traffic to mirror repositories, allowing a customer’s core GitHub repos to just serve the core development traffic.

SpaceX subsidiary Cursor, which uses Git to back its agent-support service, also refined the storage layer for better performance. Cursor engineers also rejected Spokes’ three-phase commit in favor of uploading pushes into an object storage write-ahead log (WAL), which captured all changes as immutable objects, caching at least one copy on super-speedy solid-state disks.

Respect for the Octocats

While it's cathartic to hate on a service that often goes down, you have to feel empathy for the GitHub engineers who, when they created the service, couldn’t have guessed what they signed up for.

“I've worked at extreme scale before and seen companies spend three months just preparing for events that will increase their traffic by 10% or 20% [...], which is nothing compared to what GitHub's facing now,” wrote PlanetScale CEO Sam Lambert in an X message.

We can all hope the new storage architecture will give GitHub's engineers some well-earned rest.

Speeding up the Unreal editor by… not using unnecessarily secure hash methods

One good trick when trying to look for ways to optimize software is to stay away from areas that are very likely already quite well optimized, and instead focus on areas that might have been overlooked, either because they are too obscure or don’t show up during regular profiling sessions. This tactic worked well in the past, as shown with my optimization of tooltip creation, or the loading of editor localization files.

So when profiling the editor launch (again), this is why I mostly skip over any functions that contain words like “Shader Compilation” or “Shader Source” when I stumble over them in my profiler. The compilation of shaders is a famously long and elaborate process, and I’m pretty sure that very smart people at Epic already spent a lot of time optimizing those areas. That’s the reason why I would, for example, skip the function “Compile Global Shader Map,” even though its runtime would be typically multiple seconds:

This was, at least, true until one day I skipped over it using an unusually high zoom level and saw one of the functions that was called by it:

I am certainly not an expert for hash algorithms, but SHA1 is one of the (very few) hash algorithms that even I know. And seeing this hash algorithm being used to hash shaders seemed somehow wrong to me. That’s because SHA1 was a cryptographically safe hashing algorithm, originally developed by the NSA (yes, that one). And even though it has been cryptographically broken years ago and actually shouldn’t be used for sensitive information anymore, this origin means that it is – to put it politely – not particularly fast.

What does not fast mean in this context? Well see for yourself in this comparison test of a bunch of commonly used hash algorithms that Aras Pranckevičius did a few years ago (actually, it was 10 years ago – I’m getting old). SHA1 was not even included as an actual competitor, but as the start of the article notes just “to illustrate the performance differences.” Ouch.

That’s especially annoying since the data being hashed is the shader source code, including all comments and copyright notice:

This means both that we have to hash a lot of data (source files can be quite big) and that there is no reason to hash it using a cryptographically secure hash, since it is no sensitive data.

Therefore, not using a cryptographically safe but fast hash algorithm should be an easy win, right? And the linked article already presented a good alternative, for which I even found an existing implementation in Unreal’s code base: xxHash.

Note before somebody asks: xxHash is not necessarily the fastest hash. Other, potentially even faster hashes do exist, but for now, using a faster hash that is already a used and tested part of the engine we are working on is a much more straightforward option.

Changing the used hash wasn’t too complicated, even though I had to change more places in the code than expected, since all places where the produced hash was stored or passed also had to be updated. A nice side effect of this change was that the stored hashes were a bit smaller than before (8 instead of 20 bytes).

But, of course, the effect that I made the change for wasn’t about memory, but CPU time. For testing purposes, I opened one of Unreal’s own sample projects, Electric Dreams, in the editor 5 times, both with and without the change, for a total of 10 runs. Profiling was done with Superluminal.

To put the following numbers into perspective, let me note that I will mostly talk about the overall CPU time, summed up over all threads, not just the time on the game thread. This is not done to artificially inflate the numbers, but to make the numbers more consistent and easier to compare. The actual effect on the editor start time is (very roughly) the measured time divided by the number of cores.

In the vanilla version of the engine, a whopping 12.90 seconds were spent in FSHA1::Update. In the modified version using xxHash, this time shrunk to just 1.59 seconds spent in FXxHash64Builder::Update. That’s eight times faster! Mission accomplished.

Normally, this article would now end with a link to a pull request so that anybody interested in the change could integrate it into their own version of the engine while we wait and hope that somebody at Epic would see the pull request and merge it into the main branch.

Well, this time I have even better news: The change already got accepted! (not merged however, this seems still to take a few days more). I submitted the pull request just a few days before writing this article and those few days were already enough for somebody at Epic to not only review my change but also polish it up. So let’s take a look some of the changes that were done to my proposed change:

One very smart feat that I hadn’t thought of was to wrap the xxHash into another struct, FShaderHash. This doesn’t change the behavior right now but will make it easier to change the hash algorithm again in the future if an even faster one is needed without changing the code in many different places.

Fixed a compile error for non-editor builds: That was an oversight on my sight. I had only tested my change in an editor build, mainly because my original profiling was also done in editor. I might need to add non-editor builds to my growing list of checks to do before a pull request in the future.

And that was it. If you enjoyed this article and want to read more about game optimization in the future, consider following me on BlueSky, Mastodon or Linkedin where I post all my articles. Or subscribe directly to my blog right below the article.

Fingerprints are the new cookies

Remember the big fuss about third-party cookies? It was so big that the EU had already rolled out one of its greatest inventions - consent banners.

If EU ships faster than tech companies, that can only mean the tech companies are sitting on their hands. And one of those tech companies happens to hold two-thirds of the browser market.

What was the play?

Let’s give it up for Google - the biggest ad company on Earth!

ACT I. PRESSURE - “PRIVACY IS PARAMOUNT TO US, IN EVERYTHING WE DO.”

It’s 2019 and the cookies are still hot. It’s become bad PR and Safari and Firefox already block them.

A year later Google pledges: we will destroy cookies!! And establish a new, privacy-first reign.

ACT II. DISTRACTION - “INSTEAD OF DEPRECATING THIRD-PARTY COOKIES, WE WOULD INTRODUCE A NEW EXPERIENCE”

5 years later: people are now preoccupied with far greater worries than cookies, and the solutions Google has developed (Privacy Sandbox) kinda suck and have raised antitrust concerns so nobody uses them.

“You know what, let’s just keep the cookies. We’ll mask your IP address* though!” (* but only when incognito while authenticated) (** in the end they didn’t even deliver on that)

ACT III. ACCELERATION - YOU CAN NOW USE DEVICE FINGERPRINTING IN OUR AD PRODUCTS

Now there’s a better way to know who’s who!

How does fingerprinting work?

Nice, let’s nerd out properly.

You visit a website. Your browser sends a request to the server and the server returns code for the browser to run.

The request already reveals information about you; what browser you use, the language it’s in, the OS that runs it and your IP address, which the server needs to send you the response. That often reveals your location.

Then the code runs.

See, we’ve made a lot of civilizational progress in making browsing safe. Long gone are the days when getting a virus just by visiting a website was a daily concern. The thing is, the definition of safety varies depending on who you ask. The broad consensus ends at isolation: a website’s code can’t see the data stored by other websites, and it can’t see your files or other applications without permission. Chrome and Edge stop here. Then you have Safari (especially on iOS/macOS 26+) and Firefox extending that definition to privacy, but they still don’t fully protect you from fingerprinting. Brave and Tor are the leading privacy-first browsers, with up and coming Mullvad Browser (Mullvad being, in my opinion, one of the best VPN providers out there).

The code runs on your hardware, which can be measured in ways that produce consistent and unique readings. There’s not much that can be done about this really, so there’s only two established solutions:

1. feed a little bit of noise here and there. It’s called farbling and that’s what Brave does to make your fingerprint inconsistent.
2. make everything look the same, always, so there’s no telling browser users apart. Tor and Mullvad adopt this approach.

Ok, let’s create a fingerprint

So there are two sources for data that can make up your fingerprint; the network request and the code environment.

When you look at network, the data is generally either fragile or generic:

• the IP address; in a ‘happy path’ where one person always uses the same IP address, it might be the only data you need. But commonly people do a lot of browsing on mobile networks, they share the WiFi with others, they use VPNs or their home network’s IP address is rotated by the ISP. It’s fragile as such.
• headers like User-Agent or Accept-Language; reveal your browser, OS, versions and the language set.

When you can execute code on the device, you have ways to measure its hardware qualities (without any permission prompts!) and these are surprisingly stable:

• GPU; behaviour, vendor, driver
• CPU; behaviour, vendor, number of threads
• memory size
• display data; resolution, pixel ratio, refresh rate
• audio processing
• device model

And on the software level; installed fonts, extensions, system info…

Millions of devices use the latest Chrome, there are millions of Macs with the M4 chip and there are millions of browsers in French.

But when you look at devices that run the latest Chrome in French on an M4, that number now drops significantly. Add to it hardware measurements and it gets so low it becomes practically unique.

The fingerprint doesn’t need to be truly, globally unique - it just needs to be unique enough to tell people apart in your sample size.

A ZIP code (inferred from IP), a birth date (you’ve given away on some platform) and sex (you fill it in everywhere) together are enough to uniquely identify over 60% of Americans.

You can always add another variable to decrease the potential pool of candidates.

Information Entropy

Have you ever played Akinator? You think of a character, a genie asks you some twenty questions, and somehow it knows. That’s fingerprinting in a nutshell, except your browser answers the questions for you. Once you understand information entropy, Akinator’s magic turns into a simple trick.

Let’s simplify with that game where you have to guess the character on a board through questions like “Is my character a man?”. If I say “yes”, and half of the characters on your board are men, you’re now considering only half of the total candidates. Now we can say that my answer gave you log₂(100 / 50) = log₂(2) = 1 bit of information, because it cut the pool of candidates in half.

The bits you get from one answer are called its surprisal: the rarer the answer, the more it tells you.

• a trait shared by 1 in 2 people is worth 1 bit (is a man),
• 1 in 8 is worth 3 bits (has a mustache),
• and 1 in 128 is worth 7 bits (has one eye).

Surprisal quantifies the value of the answer, and entropy quantifies the value of the question. If 1 person in 128 is one-eyed, then asking about it will yield ‘no’ 127 times in 128.

Summing up the contribution of each possible answer, the question itself is worth ~0.066 bits of entropy - that’s the average amount of bits you gain from an answer.

When we combine traits that are independent we simply add up their bits or multiply the ratios, so a man with a mustache and one eye would mean 11 bits, or 1 in 2048. Alas, as Thomas Bayes would have you know, nothing is ever so simple in practice. A mustache is already a strong indicator of a man, and a man is probably more likely to have one eye. The traits are not independent so the bits don’t really add up.

Some interesting applications of this type of reasoning:

• In medical diagnostics: A positive result from a 99% accurate test for a disease known to affect 0.01% of the population is about 100 times more likely to be wrong than you are to have the disease. You need more evidence!
• In cryptography: A password made out of four random words is as strong as eight random keyboard characters (but much easier to remember!)
• In forensic science: singling out one person in a city of a million takes about 20 bits. A witness description that matches 1 person in 1,000 provides only 10 bits, which still leaves a thousand possible suspects.

So what can I do about it?

Test your browser:

• coveryourtracks.eff.org shows you how unique your fingerprint is, with entropy for each measured data point.

If you’re using Chrome you’re basically walking around with an ID on your forehead.

Safari and Firefox offer some level of protection, but you’re still identifiable by advanced trackers.

Brave, Tor and Mullvad Browser are the only major browsers at the moment that implement advanced protections against fingerprinting.

If you don’t feel like switching browsers, I get it. I am a big Arc fan myself and unfortunately it’s discontinued and will never have any privacy features. There are extensions like JShelter - they disrupt fingerprinting scripts by intercepting browser API calls. But they also break some websites. Same problem applies to Firefox’s Resist Fingerprinting feature.

The ironic truth is, privacy extensions make you more unique, since fewer people have them. So a smart tracker could detect that, ignore the signals disrupted by the extension and focus on the ones that are not.

No extension can be as effective as engine-level protection built into the browser. So if you don’t want to be tracked, you gotta commit.

methods used on this page

I’m simply creating a hash out of a few measurements of your device. Any of these change and the hash is now different. Professional applications are much more complex than this and are able to maintain identification line when single variables change.

1. OfflineAudioContext

   Renders a short tone through a compressor without playing it; tiny float differences in the result depend on your browser’s audio engine and your CPU.

   const off = new OfflineAudioContext(1, 5000, 44100);
   // … a 1 kHz triangle wave into a dynamics compressor …
   const data = (await Promise.race([off.startRendering(), timeout])).getChannelData(0);
   let sum = 0;
   for (let i = 4500; i < 5000; i++) sum += Math.abs(data[i]);
   return sum.toFixed(3);
   

2. WebGL

   Reads your GPU’s vendor and model (without version numbers), what the GPU and its driver support, and the pixels of a small rendered triangle.

   const gl = document.createElement('canvas').getContext('webgl');
   const dbg = gl.getExtension('WEBGL_debug_renderer_info');
   return gpuName(gl.getParameter(dbg ? dbg.UNMASKED_RENDERER_WEBGL : gl.RENDERER) || '');
   // "ANGLE (Apple, ANGLE Metal Renderer: Apple M4, Unspecified Version)" -> "apple m4"
   // …
   return fnv1a(new TextEncoder().encode([...(gl.getSupportedExtensions() || []).sort(), ...limits, ...precision].join('|')));
   // …
   gl.drawArrays(gl.TRIANGLES, 0, 3);
   gl.readPixels(0, 0, canvas.width, canvas.height, gl.RGBA, gl.UNSIGNED_BYTE, px);
   

3. canvas

   Draws text in a few fonts, emoji and a gradient, then reads the pixels back; anti-aliasing and font rendering depend on your OS, GPU and fonts.

   ctx.font = '15px "Times New Roman", serif';
   ctx.fillText('Sphinx of black quartz 🦊', 4, 22);
   // …
   const data = ctx.getImageData(0, 0, 240, 60).data;
   

4. Intl

   Your time zone and your date and number settings; a VPN changes your IP location, but not your time zone.

   const date = Intl.DateTimeFormat().resolvedOptions();
   // …
   return [date.timeZone, date.locale, date.calendar, date.numberingSystem, hourCycle, (1234567.891).toLocaleString()].join('/');
   

5. navigator

   Your number of CPU cores, your memory size (Chrome only) and your browser’s language.

   return [navigator.hardwareConcurrency, navigator.deviceMemory, navigator.language].join('/');
   

6. crypto.subtle

   Hashes all the values with SHA-256, and the hash picks a trait and an animal for your nickname.

   const digest = await crypto.subtle.digest('SHA-256', bytes);
   // …
   nickname: stranger || `${pick(TRAITS, hex, 0)} ${pick(ANIMALS, hex, 8)}`,
   

Global PC shipments crater 20% as rising prices hammer demand

Buyers rushed to beat hikes earlier this year, but demand has since dried up as AI-fueled component shortages send costs soaring

The global PC market has taken a beating, with shipments plunging more than 20 percent in the third quarter as the AI-driven memory shortage pushes up prices and buyers decide their existing machines will do just fine.

New figures from Omdia show worldwide shipments of desktops, notebooks, and workstations fell 21.2 percent year-on-year to 58.1 million units in Q3 2026. IDC tells a similar story, estimating a 20.1 percent decline to 62.7 million units.

The research firms differ on shipment totals, but agree the market has gone from bad to considerably worse.

The downturn follows a rush to buy PCs earlier this year, when manufacturers, distributors, and corporate customers brought forward orders to get ahead of anticipated price increases. That helped prop up shipments in the first half of 2026, but left the industry with a pile of inventory and fewer buyers in the third quarter.

IDC said shipments fell 9.1 percent from Q2, bucking the usual seasonal pattern in which back-to-school demand lifts the PC market.

"What we're seeing is the result of the strong first half pull-in," said Jitesh Ubrani, research director for consumer devices at IDC. "Vendors and channels loaded up on inventory early in the year to get ahead of price hikes, and that has thrown off the usual seasonality, where Q3 is typically larger than Q2."

Much of the trouble stems from the AI industry's appetite for memory and storage, which has left PC makers competing for increasingly expensive components. The shortage was already taking its toll in Q2, when IDC initially reported a 4.9 percent decline in shipments, since revised to 3.8 percent.

Omdia estimates that DRAM and SSDs now account for nearly 40 percent of a PC's component costs, up from around 15 percent previously, following more than fourfold price increases. Shortages of processors, graphics cards, and other integrated circuits are also adding to manufacturers' woes.

The damage is being felt across the industry's biggest names. Lenovo retained its position as the world's largest PC vendor, shipping 14.9 million units, although IDC estimates that represents a 22.6 percent decline from a year earlier.

HP fared worse, with shipments plunging nearly 31 percent to 10.3 million units. Dell recorded a 25 percent drop to 7.6 million, while Apple escaped relatively lightly with an 11 percent decline, according to IDC. The Mac maker even increased its market share despite shipping fewer machines.

Omdia puts Apple's shipments higher, at 6.8 million units compared with IDC's 5.9 million.

Buyers may get some relief as retailers and distributors try to clear excess stock, but IDC doesn't expect a return to the prices consumers paid a year ago.

"Channels are now worried about carrying too much inventory into a market where high prices are suppressing demand," Ubrani said. "That could translate into promotions and some short-term relief for consumers, but we don't expect pricing anywhere near what it was a year ago."

The outlook isn't particularly cheerful. Omdia expects global PC shipments to fall another 24 percent year-on-year in the fourth quarter, followed by a further 7 percent decline across 2027.

For an industry hoping AI would help sell more PCs, that's quite the result.

flux9s GA: Flux cluster state, from the terminal

flux9s recently reached 1.0. It is a K9s-inspired terminal UI for Flux: real-time state for every Flux resource in your cluster, with the operations you reach for most a single keystroke away.

This post is about why it exists, how the Flux Operator’s web UI accelerated its development, and what it took to cover the whole Flux Operator ecosystem before the 1.0 release.

The motivation: seeing cluster state at a glance

When I was new to Flux, the process I used to determine “what is happening with this cluster?” was flux get all, usually piped through grep. It worked, but it was hard to visualize, and not in real time. Which Kustomizations are failing and why? What is the semver of that Kustomization’s upstream OCIRepository? The state of a cluster and the relationships between its components are hard to see at a glance. The Flux CLI is very powerful, and features like flux trace can answer a specific question quickly, but reconciliation is continuous, and my view of it was a series of snapshots.

I thought that a good solution to my problem was a TUI (Terminal User Interface). There were some great ones I had encountered before, especially K9s. K9s had solved this exact problem for core Kubernetes objects. I wanted the Flux equivalent: a live view I could leave open on a second monitor, with the same muscle memory from vim of j/k to move, : for commands, / to filter. It is worth noting K9s has an excellent Flux plugin, but the scope of this plugin could never fully keep up with the actions, views, and monitoring capability needed for an ecosystem as complex as the Flux one.

I wrote flux9s in Rust (although my initial velocity might have been faster in Go). I was studying the language at the time, and kube-rs offered support for the Watch API that a live view is built on. The result is a single lightweight binary. Rust also pairs very well with AI-assisted development: the compiler catches some of the silly bugs at compile time instead of letting an agent ship them to runtime.

The first release in November 2025 was fairly simple: the Flux CRDs in a unified list, driven by the Kubernetes Watch API rather than polling, with suspend, resume, reconcile, and YAML inspection. This is still what flux9s is today, with some added features. Two parts that have remained since the initial release:

• Read-only by default. flux9s launches in read-only mode; mutating a cluster is opt-in via :readonly or flux9s config set readOnly false.
• Watch API. Everything rendered in flux9s is from the Watch API and live.

The inflection point: the Flux Operator’s web UI

A month after flux9s’s first release, when I had only shared it with a few people, the Flux Operator shipped its web UI, Mission Control for GitOps. At that point I was not sure if anyone would use flux9s. Did Flux need multiple UIs? After some thought and discussions, I kept going. Many people in the SRE / Platform Engineering world prefer to work in the terminal, and having an interactive UI tied to a user’s kubeconfig is additive to the ecosystem.

In practice, the web UI turned out to be one of the best things to happen to the development of flux9s. Many of the features of the web UI had clearly been derived from user requests, so there was a clear roadmap of enhancements to add. I cloned the web UI code, deployed it on my personal cluster in minutes (very easy!) and thought about what could be added to flux9s. The features that landed next were web UI concepts translated into the TUI:

• Dependency graphs (g) - the ownership and inventory relationships around a resource, drawn with box characters instead of SVG, navigable with j/k and Enter.
• Reconciliation history (h) - for the resources that track it, what happened on each attempt and why.
• Favorites (f) - the handful of custom resources you actually watch, pinned into their own view.

flux9s got dramatically better because a larger, well-designed project in the same ecosystem had already worked out what mattered most to users. This accelerated the development a lot, and something like the graph view is one of my favorite features for seeing and understanding a cluster.

The road to 1.0

The Flux Operator ecosystem moves fast. ResourceSets became very important, and supporting the way many users are consuming the new flux-operator CRDs drove most of the releases through 2026:

• FluxInstance and ResourceSet downstream discovery - both publish a status.inventory, so both render like a Kustomization in the graph view.
• ResourceSet step visualization - ordered steps with per-step phase, so a stalled rollout points at the step that stalled.
• :pulse cluster health dashboard - the operator’s FluxReport rendered live: ready/failed/suspended totals, per-kind counts, and the most recent failures with their messages.
• Opt-in CRD discovery - after running flux9s config set discoverFluxResources true, any CRD labeled app.kubernetes.io/part-of=flux shows up live and view-only. Flagger, tofu-controller, and other projects appear through the same label-driven discovery.
• Following a failure all the way down - a live :events feed, workload drill-down from the graph, and streaming pod and controller logs (l, :logs), so an investigation stays inside the tool and Back walks the whole chain in reverse.

Try it

flux9s runs anywhere your kubeconfig does:

brew install dgunzy/tap/flux9s
# or
cargo binstall flux9s

Binaries for Linux, macOS, and Windows are on the releases page, signed with Cosign.

flux9s

It starts on flux-system in read-only mode. From there:

:ns all        watch every namespace
:pulse         cluster health dashboard
/              filter the list
g              dependency graph for the selected resource
?              full keybinding help

The documentation covers configuration, skins, and the full command reference.

flux9s is listed among the community UIs in the Flux ecosystem, a privilege for a project that started as a way to stop piping flux get all through grep quite so often. If you try it and something is wrong, missing, or awkward, open an issue; that is how most of what is described above got built.

• Source code
• Documentation
• Releases
• Issues and feature requests

Enterprise engineering teams are rushing to deploy autonomous agents and RAG architectures, but moving generative AI from pilot to production exposes a critical visibility gap in traditional software monitoring. While point solutions isolate single-LLM calls in a sandbox, enterprise AI applications operate as complex, probabilistic transactions across distributed systems. Observing AI in a silo creates dangerous blind spots around upstream and downstream operational impacts while widening the gap between AI developers and production engineering. Traditional monitoring measures throughput and latency, but misses semantic failures—leaving teams blind to hallucinations, PII leaks, and unexpected model drift.

New Relic AI Evaluation closes this trust gap by integrating automated qualitative guardrails directly into your existing observability workflow. Powered by an asynchronous LLM-as-a-judge service, it automatically scores live prompts and completions against strict safety and accuracy parameters, attaching those quality metrics directly to your distributed traces. By mapping semantic response quality straight to underlying software spans and infrastructure costs, leaders gain the real-time control required to protect brand reputation, optimize LLM spend, and confidently scale AI applications in production.

Scaling Enterprise GenAI Safely

Built as a platform capability within New Relic AI Observability, AI Evaluation transforms how engineering teams manage non-deterministic applications by expanding standard performance tracking into semantic and qualitative observability. Instead of treating LLM responses as uninspectable black boxes, teams can now automatically measure the accuracy, safety, and relevance of their AI inputs and completions alongside core application telemetry.

Powered by specialized evaluation models, the service evaluates sampled prompt payloads and model completions against predefined quality and security parameters. Each evaluated interaction receives a normalized score from 0 to 1, a clear pass/fail status based on customizable thresholds, and an explanatory reasoning string, ensuring your team receives actionable feedback on silent failures like hallucinations or data leaks.

Unlike standalone point solutions that isolate evaluation data, AI Evaluation is built natively within the New Relic platform. Quality scores, failure flags, and judgement rationale are automatically returned to the platform and attached as attributes directly to your original distributed traces. This seamless integration connects probabilistic quality scores straight to software root causes, allowing you to navigate from a low-quality response score directly to the specific prompt span, vector database retrieval call, or backend service that caused the issue within a single, unified view.

Optimize Your AI Spending

As enterprise AI deployment expands, engineering leaders and platform teams face the growing challenge of balancing exceptional user experiences against spiraling model compute costs. AI Evaluation bridges performance telemetry with financial governance by connecting qualitative response scores directly to underlying infrastructure consumption. This contextual visibility allows teams to evaluate whether expensive, high-parameter models deliver superior accuracy and relevance compared to faster, lower-cost alternatives. Armed with clear quality-to-cost correlations, organizations can optimize their model selection and swap out underperforming LLMs to lower token spend without compromising answer quality or customer trust.

To prevent evaluation overhead from straining resources, AI Evaluation incorporates intelligent, highly configurable sampling. Platform operators can establish granular sampling rules on live traffic, such as running toxicity or data leakage checks on a sampling of 10 percent of transactions, to maintain continuous safety oversight without evaluating every single interaction. This asynchronous processing delivers high-integrity compliance monitoring and vulnerability detection while keeping operational expenses strictly controlled. By tailoring sampling strategies to specific application workloads, enterprise teams maintain robust quality guardrails while optimizing the total cost of ownership for their observability stack.

Accelerating Time-to-Market without Compromising Quality

Safely transitioning generative AI applications from experimental pilots to production requires continuous oversight that extends beyond traditional technical metrics. AI Evaluation delivers an end-to-end operational framework that automatically assesses LLM interactions for accuracy, safety, and relevance while mapping those insights directly to your underlying application telemetry. By combining real-time security guardrails, architectural failure isolation, and native distributed trace integration, this capability provides engineering and business leaders with the control needed to protect brand reputation, minimize compliance risks, and accelerate time-to-market.

Real-Time Security and Compliance Guardrails

AI Evaluation provides out-of-the-box evaluators, with no setup or configuration necessary. These safety checks can be embedded directly into live application traffic to protect brand safety and maintain regulatory compliance. By automatically evaluating sampled prompt inputs and model completions, the system flags security risks and compliance threats before they cause reputational or legal harm.

• Prompt Injection and Jailbreak Defense: Identifies adversarial user instructions and attempts to bypass safety guardrails to restrict unauthorized content generation.
• Data Leakage Mitigation: Scans incoming payloads and model completions for sensitive data, automatically flagging personally identifiable information (PII).
• Toxicity and Bias Screening: Automatically detects harmful, offensive, or abusive language, as well as unfair or prejudiced treatment based on protected attributes.

RAG Architecture and Agent Performance Isolation

When an autonomous agent or Retrieval-Augmented Generation (RAG) system delivers an inaccurate answer, identifying the failing component is critical. AI Evaluation isolates failure modes by evaluating quality across specialized retrieval and generation metrics:

• Faithfulness (Hallucination Tracking): Measures whether generated responses are factually consistent with the provided retrieval context.
• Answer and Contextual Relevancy: Measures how well retrieved information matches the user query and whether the final response directly addresses the prompt.

Direct Distributed Trace Integration

Unlike some standalone LLM observability tools that store evaluation scores in disconnected silos, AI Evaluation feeds judgement results directly back into your core telemetry stream.

• Transaction-Level Context: Instead of evaluating isolated LLM calls, AI Evaluation looks holistically across the entire transaction, linking qualitative response scores directly to upstream prompts, vector database retrievals, and downstream compute consumption in a single view.
• Attribute Enrichment: Normalized quality scores, pass/fail statuses, and reasoning strings are ingested back into the platform and attached directly as attributes to the original distributed trace.
• Unified Stack Visibility: Engineers can view prompt payloads, judgement rationale, and cross-stack application behavior on a single screen.
• Accelerated Troubleshooting: When an interaction fails a quality threshold, responders can navigate directly from the evaluation alert to the precise execution span, vector database retrieval call, or backend service responsible for the failure.

Strategic ROI Across the Enterprise

By expanding observability beyond traditional system metrics into semantic evaluation, New Relic AI Evaluation transforms stalled AI prototypes into secure, production-ready enterprise services. Automating quality and safety guardrails eliminates the manual log reviews that create severe deployment bottlenecks, allowing organizations to ship generative AI applications faster while insulating their brand from reputational and compliance risks. This capability delivers targeted value across three key stakeholder roles:

• AI/ML & LLMOps Engineers: Practitioners get real-time evaluation scores tied directly to distributed traces, replacing unscalable log spreadsheets and manual reviews. When fine-tuning system prompts or optimizing RAG retrieval logic, engineers can instantly verify whether modifications improve answer relevance or inadvertently trigger regressions like factual hallucinations.
• Platform Engineers, SREs & DevOps Leads: Platform leads can align system reliability and compute performance with financial governance. By correlating qualitative response metrics with underlying token costs, operators can evaluate whether expensive models justify their compute spend and set intelligent sampling rules to maintain platform performance without inflating infrastructure budgets.
• Security & Compliance Officers: Compliance leaders receive the continuous, automated oversight required to clear autonomous bots and public-facing AI agents for production. Automated scanning for prompt injection attacks, jailbreaks, PII leaks, and toxic outputs provides proactive risk mitigation, ensuring AI applications comply with strict data privacy standards and corporate brand guidelines.

Accelerate Your AI Deployments Today

Deploying AI Evaluation within your existing telemetry workflow requires zero complex setup or custom code rewriting. Because native APM agents automatically capture prompt input and model output payloads as part of standard trace instrumentation, your team can begin capturing qualitative telemetry immediately. Simply configure your preferred flagging criteria, establish your sampling rules, and watch real-time quality scores automatically enrich your distributed traces.

Don't let silent failures, data leakage risks, or prompt regressions keep your generative AI applications trapped in experimental pilots. Request a live demonstration today to see how New Relic AI Evaluation brings real-time safety guardrails to your LLM architecture, or explore our technical documentation to start evaluating your production AI workloads now.

How do you know that the action an agent performed did as well as expected? Did it hallucinate a workflow and therefore the output? Was it performant? Did it follow safety best practices that all agents should abide by? Organizations have heard the whole "hallucination from LLMs" bit and various other issues with agentic output, so having a tool that can evaluate what actually happens is within an agent run key.

That's where a new concept from Anthropic, agentic evaluations, comes into play.

In this blog post, you'll learn about agentevals and how to use/implement a tool that you can use to run AI evaluations across any agent workflow.

Prequisites

To follow along with this blog post, you should have the following:

• A k8s cluster. This can be a cluster running anywhere, even locally.
• kagent installed, and you can find the instructions here.

If you don’t, that’s totally fine! You can still follow along from a theoretical perspective and implement the hands-on piece at a later time.

What Are Agentic Evaluations?

Tldr; a benchmark/end-to-end test to ensure that your agent did what it was supposed to do based on human instruction.

When you run an agent, you have zero idea what it's actually going to end up doing (hence the constant conversations around non-deterministic workflows). You can give it a close-to-perfect prompt, agent Skills for a tightly defined task, and use the best instructions possible (e.g- a CLAUDE.md file, AGENTS.md, etc.), but despite the effort that you put into perfecting the output, that still doesn't mean your agent is going to do what it's supposed to because there are far too many variables in an agent run. With evaluations (evals), you can take a trace (the end-to-end agent workflow) that already occurred or look at an agent running live to see what it did/does and how it did/does it. The trace can then be graded. For example, if the agent hallucinated 25% of the output, that means the grade may be lower than “perfect” as the agent did not do exactly what it was supposed to do. The purpose of an evaluation is to see how well your agent did the job you tasked it with.

Where Evals Need Assistance

Because of the non-deterministic nature of agentic workflows defined in the previous section, that means by definition, the evaluation set may not be the same every time. For example, you can run an agent three times and get three different results. That's why specifying what you want to evaluate and how you want to grade an agent workflow is so crucial. Example: if you run an eval and you care about latency, focus on latency. Run it 3-5 times and take an average. Don't worry about the other output, just focus on one key area of the overall evaluation.

Setting Up an agent

With the “why” of why you’d want to use agentevals defined, let's dive into the hands-on piece of evals. First things first - for you to get a trace to benchmark/grade, you need an agent. An agent that’s used, at a high level, will most likely look like the following:

1. A set of specific tools that your agent uses based on the task it needs to perform. These tools typically come from an MCP Server.
2. A gateway that’s specialized for both LLM, MCP, and agentic traffic.
3. An agent setup and configured.

Because it's a fair amount of code and configuration to get all of the above up and running, instead of having a full page of code on this blog post, you can a sample configuration on GitHub that you can use.

Deploying agentevals

To deploy agentevals, there are two choices for installation:

1. From source
2. One-line installation

The one-line install is via a Python package:

pip install agentevals-cli

If you decide to go with from source, you can clone the repo from here and then run:

make dev-bundle

You’ll then see agentevals running within your terminal.

Planning The Collection Of A Trace

The first step is to collect a trace. Luckily, the majority of agentic frameworks and AI gateways will expose traces. If you used kagent and agentgateway via the Setting Up an agent section, you'll definitely have traces available within your agent run.

1. Ensure that you run your agent from the Setting Up an agent section after you get it created. That way, a trace is generated.
2. Prompt Claude to collect traces for your agent that you ran in the previous section. Ensure that it knows agentevals can look at a trace via a JSON file and have it write a plan for you.

As an example, here's a plan that it came up with for me.

Here is Claude's plan:
Plan: Export Gemini agent Traces from Kubernetes to Jaeger JSON
...
(Plan implementation details omitted for brevity)

1. With the plan in place to generate a JSON file with your trace, run it in your framework (e.g - Claude Code) on the terminal.

Collecting and Building EvalSets

When Claude is done running from the previous section, ensure that it saves a JSON file for you somewhere that's accessible via agentevals. For example, you'll see that Claude created a file called gemini_agent.json that was then saved to the desktop.

Once you have your trace output, you can feed it into the EvalSet Builder. The EvalSet Builder is a feature in the agentevals Web UI that lets you create evaluation sets (EvalSets) from existing agent traces.

With the EvalSet and your trace, you can now run your benchmarks/tests via agentevals.

Obtaining Eval Results

After you generate a trace JSON and use the EvalSet Builder with said JSON trace, you can begin to drill into how your agent performed. This will allow you to get an understanding of what to expect from an output quality perspective for a particular workflow.

1. Within the Evaluations tab, upload your trace and EvalSet.
2. Once it's uploaded, you will be able to select what you want to get results (e.g- how well the response of the agent was, the safety of the agent, output quality, etc.).
3. As the evaluation is running, you will begin to see various scores and results. Please note, some results take longer than others to see the output depending on what it is (e.g - a benchmark running to test an agent's response).
4. One of the first you'll most likely see is one of the conversations that you had with your agent during the run that you obtained the trace of. You can see that the agent was asked "What can you do?" during the run and therefore, that shows up within the trace, so you can benchmark it.

Drill into the first result that is complete. With the completed result, you can see everything from the overall latency/performance of the run, what went wrong (e.g - hallucinations), and the overall score of how the agent operated during that run.

agentevals CLI

Aside from using the UI, you can also use the agentevals CLI to see results.

uv run agentevals run samples/helm.json \
  --eval-set samples/eval_set_helm.json \
  -m tool_trajectory_avg_score

There are several examples that you can find on the docs page.

Conclusion

The two top priorities for just about every organization are security and observability around the end-to-end agentic workflow and how to ensure that the output you're receiving from an agent is as good as expected. Agentic quality and output performance will become increasingly crucial as agents become more autonomous and adopted across various teams within an organization.

What today's software owes to Bronze Age clocks

Here's a deceptively simple question: should this function execute?

Sometimes, the answer is obvious. Is the user logged in? Does their role have the required permissions?

Sometimes it's less obvious. Are there sufficient resources available? Is this work a higher priority than everything else?

There are many names for the mechanisms that block or allow actions in software. Most commonly they are referred to as admission control. Perhaps you'll be familiar with a leaky bucket implementation.

A brief history of leaky buckets

Engineering applications for leaky buckets predate computers. From around 1600 BC the Babylonians, Egyptians, and Persians all used vessels with a small hole in the bottom as primitive clocks, measuring elapsed time by the volume of the container as it drained at a constant rate.

By around 200 BC the Greeks had advanced leaky buckets into accurate clocks. A floating regulator kept water flowing steadily from one bucket into a second, where a rising float moved a pointer to show the hour. A siphon emptied the second bucket once it was full. Most notably, Ctesibius added gears to regularly advance each "tick" of the clock, turning the display which indicated the current season.

A mechanical movement driven by a leaky bucket was born, and it became the gold standard of timekeeping for centuries.

Water-based clocks were used as a form of admission control. In Athenian courts, the length of time a speaker had to plead their case was measured by the time water took to drain from a vessel. The amount of water poured into the vessel was determined by the severity of the case.

Finally, in the computer age, Jonathan Turner is said to have used the term "leaky bucket" first in 1986, writing "New directions in communications" for IEEE Comms Magazine. The piece describes how a system can police the pace at which packets are sent over broadband connections.

Leaky buckets in software

There are two kinds of leaky buckets in software, to implement a meter or a queue.

In both, each unit of "work" is assigned a value which is checked against the current volume of a "bucket." If the bucket lacks capacity for the size of new work, that work is rejected. Otherwise the work is allowed and its value adds to the bucket's current volume. The bucket drains (or leaks, if you will), so over time, new work can't be admitted faster than the bucket drains. The drain rate is based on time elapsed, not on the completion of work.

When used as a meter, the bucket defines if and how much work is permitted to immediately proceed. As a queue, the bucket defines the volume of work that can wait to proceed, but work only begins as it drains from the bucket.

In this post we'll focus on leaky buckets as a meter.

Typically, leaky buckets are implemented in software to keep systems healthy. A sort of resource management to ensure the system is not overwhelmed.

It's not hard to imagine this being useful in a database. Imagine being able to block or allow queries, before they execute, based on available resources or concurrency limits. One can dream!

The leaky bucket can be tweaked by modifying its volume and the rate at which it drains.

At its core a leaky bucket prevents work, but it also forces us to reason about the total volume of work that results in a block, how quickly more capacity becomes available, and how to handle the expected failure state when work is rejected.

Leaky buckets in API design

Shopify's GraphQL Admin API implements rate limiting with a documented leaky bucket. Each app on each store gets a bucket with a fixed size and restore rate.

Each request costs a different amount based on what the query asks for, not how many requests are made. Before a query runs, the bucket must have room for its estimated cost. Once it completes, the bucket is refunded or charged any difference between the estimated and actual cost. Apps can burst faster than the restore rate until the bucket has no room left, at which point requests are throttled.

Responses include the current state of the bucket, so apps can slow down before they're throttled and treat throttling as an expected outcome, not a surprise.

Classification as admission control

Now let's consider the AI age. Jev, a "System One" model (perhaps better named a "Decision Model"), recently launched, making classification through inference cheap and fast. It could classify "should this work proceed or not" based on a predefined set of options and return a response in hundreds of milliseconds.

Fun full-circle fact: In 1907, the French scientist Lapicque modeled neurons as a charge that accumulates, leaks away, and fires at a threshold, later called the "leaky integrate-and-fire" model. So something like a leaky bucket has long been used to describe neurons, making it part of the origin story of neural networks, and so of Jev.

Whether our decision gate is mechanical (leaky bucket) or requires inference (Jev), the requirement is the same: answering "should this proceed?"

Low cost throttling

One of the great benefits of leaky buckets in software is how cheap they are. What we've described as a slowly draining bucket isn't a long-running process that is using CPU to "drain".

Simply put, a bucket is a stateful object that holds two values:

1. how full it was the last time anyone looked
2. when that was

From those values, you can work out how full the bucket is now:

level_now = max(0, level_then - drain_rate × (now - then))

Any work rejected or admitted by the bucket updates its current state. In this way, buckets are lazily drained (as in, only when checked) and empty buckets hold no values.

Admission controls and databases

Just like hardware, operating systems, applications, and ancient Greeks, databases benefit from leaky buckets.

However, open-source databases like Postgres and MySQL don't have this functionality baked in.

One of the most common failure modes for databases in production is the inability to deliberately reject work they cannot complete. Blast a billion queries at your typical relational database, and it'll try its best to execute them. Success isn't likely. Some hardware, configuration, or connection limitation will unwittingly play the role of an admission control system and start blocking work in some unhandled, undesirable way.

Some proprietary databases have a form of resource management so that workloads can be throttled (but not rejected). For example, in 1999 the Oracle Database Resource Manager added mapping rules to sessions for users, services, clients, and more, which could then be used to apply limits on things like resource use and concurrency.

In 2008, SQL Server added the SQL Server Resource Governor, which added classifier functions to sessions that can then apply rules to queries in that session.

Postgres has some configuration options to control expensive queries that have already consumed resources, such as statement_timeout, but nothing to prevent expensive work before it happens. And so in 2026, PlanetScale brought decision gates to Postgres with Database Traffic Control®.

Database Traffic Control

The health of your Postgres database is laid bare at the mercy of your applications. An unexpected connection storm or failure of the query planner can ask more of your database than it can reasonably be expected to handle.

To protect your Postgres database from your application (and at times, from itself), we built Traffic Control, admission control for Postgres. In it, resource budgets determine how many resources or how much concurrency one or many queries can consume.

Each budget admits or blocks queries that match a specified set of tags. Those tags can be specified as comments in the query, or they can refer to the user or application that sent it. If the resource budget determines the query should not proceed, it will log a warning or block it, depending on your resource budget configuration.

Creating a resource budget opts you into an expected failure state, so it forces you to reason about the behavior your application should exhibit when queries are blocked. For this reason, we recommend creating a new budget in warning mode first, monitoring how much traffic is affected, preparing your application for blocked queries, and then updating the mode.

Admission controls but for traffic

You configure three major settings on a resource budget.

Server share and burst limit, the token* bucket. Server share determines how fast the bucket replenishes with tokens, expressed as a percentage of server resources. The burst limit sets the maximum capacity of the bucket, expressed in whole-server seconds.

Because a database doesn't know the cost of a query before execution, Traffic Control estimates its cost based on the query plan and historical behavior of queries with that shape. If the budget has sufficient capacity the query proceeds. After execution, the cost of the query is calculated and any difference is either refunded or charged to the token bucket.

Why is the refill rate measured in server capacity instead of seconds? Because not all time spent executing a query is equal. A query that took 8 cores a single second is different from a query that took one core a single second. Consider different server sizes as well: a 25% server share means a quarter of your server's capacity drains back every second, whether that server has 2 cores or 64.

*Hold on, token bucket? A token bucket works the same as a leaky bucket, but with a reversed metaphor. The bucket starts filled with "tokens" and each piece of work is assigned a "cost." Work is only admitted if there are sufficient tokens remaining for that work. Tokens refill over time. It's a bit easier to reason about the remaining capacity of a resource budget in terms of a bucket's remaining tokens than its current fill level.

Per-query limit sets an upper limit on the maximum capacity a single query can consume. This makes an admission-control decision directly on a single query instead of measuring it against available capacity in a bucket.

Maximum concurrent workers similarly controls the resources a single query can consume, but measures it by available workers instead of execution time.

A query may be targeted by multiple resource budgets and must satisfy all of them to proceed.

Summary

Admission control forces you to decide what should happen when work can't proceed, and leaky buckets have been a cheap way to make that decision for a very long time. Traffic Control makes that decision for Postgres before an expensive query runs, not after.

AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning. And you can run 405B Llama3.1 on 8GB vram now.

Updates

[2024/08/20] v2.11.0: Support Qwen2.5

[2024/08/18] v2.10.1 Support CPU inference. Support non sharded models. Thanks @NavodPeiris for the great work!

[2024/07/30] Support Llama3.1 405B. Support 8bit/4bit quantization.

[2024/04/20] AirLLM supports Llama3 natively already. Run Llama3 70B on 4GB single GPU.

[2023/12/25] v2.8.2: Support MacOS running 70B large language models.

[2023/12/20] v2.7: Support AirLLMMixtral.

[2023/12/20] v2.6: Added AutoModel, automatically detect model type, no need to provide model class to initialize model.

[2023/12/18] v2.5: added prefetching to overlap the model loading and compute. 10% speed improvement.

[2023/12/03] added support of ChatGLM, QWen, Baichuan, Mistral, InternLM!

[2023/12/02] added support for safetensors. Now support all top 10 models in open llm leaderboard.

[2023/12/01] airllm 2.0. Support compressions: 3x run time speed up!

[2023/11/20] airllm Initial version!

Quickstart

1. Install package

First, install the airllm pip package.

pip install airllm

2. Inference

Then, initialize AirLLMLlama2, pass in the huggingface repo ID of the model being used, or the local path, and inference can be performed similar to a regular transformer model.

(You can also specify the path to save the splitted layered model through layer_shards_saving_path when init AirLLMLlama2.

from airllm import AutoModel

MAX_LENGTH = 128
# could use hugging face model repo id:
model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct")

# or use model's local path...
#model = AutoModel.from_pretrained("/home/ubuntu/.cache/huggingface/hub/models--garage-bAInd--Platypus2-70B-instruct/snapshots/b585e74bcaae02e52665d9ac6d23f4d0dbc81a0f")

input_text = [
        'What is the capital of United States?',
        #'I like',
    ]

input_tokens = model.tokenizer(input_text,
    return_tensors="pt", 
    return_attention_mask=False, 
    truncation=True, 
    max_length=MAX_LENGTH, 
    padding=False)
           
generation_output = model.generate(
    input_tokens['input_ids'].cuda(), 
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])

print(output)

Note: During inference, the original model will first be decomposed and saved layer-wise. Please ensure there is sufficient disk space in the huggingface cache directory.

Model Compression - 3x Inference Speed Up!

We just added model compression based on block-wise quantization-based model compression. Which can further speed up the inference speed for up to 3x , with almost ignorable accuracy loss!

How to enable model compression speed up:

• Step 1. make sure you have bitsandbytes installed by pip install -U bitsandbytes
• Step 2. make sure airllm verion later than 2.0.0: pip install -U airllm
• Step 3. when initialize the model, passing the argument compression ('4bit' or '8bit'):

model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct",
                     compression='4bit' # specify '8bit' for 8-bit block-wise quantization 
                    )

What are the differences between model compression and quantization?

Quantization normally needs to quantize both weights and activations to really speed things up. Which makes it harder to maintain accuracy and avoid the impact of outliers in all kinds of inputs.

While in our case the bottleneck is mainly at the disk loading, we only need to make the model loading size smaller. So, we get to only quantize the weights' part, which is easier to ensure the accuracy.

Configurations

When initialize the model, we support the following configurations:

• compression: supported options: 4bit, 8bit for 4-bit or 8-bit block-wise quantization, or by default None for no compression
• profiling_mode: supported options: True to output time consumptions or by default False
• layer_shards_saving_path: optionally another path to save the splitted model
• hf_token: huggingface token can be provided here if downloading gated models like: meta-llama/Llama-2-7b-hf
• prefetching: prefetching to overlap the model loading and compute. By default, turned on. For now, only AirLLMLlama2 supports this.
• delete_original: if you don't have too much disk space, you can set delete_original to true to delete the original downloaded hugging face model, only keep the transformed one to save half of the disk space.

MacOS

Just install airllm and run the code the same as on linux.

• make sure you installed mlx and torch
• only Apple silicon is supported

FAQ

1. MetadataIncompleteBuffer

safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer

If you run into this error, most possible cause is you run out of disk space. The process of splitting model is very disk-consuming. You may need to extend your disk space, clear huggingface .cache and rerun.

2. ValueError: max() arg is an empty sequence

Most likely you are loading QWen or ChatGLM model with Llama2 class. Try the following:

For QWen model:

from airllm import AutoModel #<----- instead of AirLLMLlama2
AutoModel.from_pretrained(...)

For ChatGLM model:

from airllm import AutoModel #<----- instead of AirLLMLlama2
AutoModel.from_pretrained(...)

3. 401 Client Error....Repo model ... is gated.

Some models are gated models, needs huggingface api token. You can provide hf_token:

model = AutoModel.from_pretrained("meta-llama/Llama-2-7b-hf", #hf_token='HF_API_TOKEN')

4. ValueError: Asking to pad but the tokenizer does not have a padding token.

Some model's tokenizer doesn't have padding token, so you can set a padding token or simply turn the padding config off:

input_tokens = model.tokenizer(input_text,
   return_tensors="pt", 
   return_attention_mask=False, 
   truncation=True, 
   max_length=MAX_LENGTH, 
   padding=False  #<-----------   turn off padding 
)

Citing AirLLM

@software{airllm2023,
  author = {Gavin Li},
  title = {AirLLM: scaling large language models on low-end commodity computers},
  url = {https://github.com/lyogavin/airllm/},
  version = {0.0},
  year = {2023},
}

Full article content is not available for inline reading.

Read the original article →

Codex Cloud shipped with Tailscale support. Nobody told us.

The launch process for Tailscale’s integration into Codex Cloud, which OpenAI released this week, started with a Slack message from a Tailscale engineer: “Did we work with OpenAI on this?” A few minutes passed, and nobody at Tailscale could confirm that we had, or that anybody knew about this brand-new product shipping with “day 0 Tailscale support.”

We didn’t work with OpenAI on this, but Codex Cloud launched with support for Tailscale as a VPN provider (the only one, for now). It caught us entirely by surprise, and we couldn’t be happier.

“There was no partnership agreement, custom API, or integration project. They just built it using the same Tailscale that everyone else gets,” Tailscale CEO Avery Pennarun wrote on LinkedIn. “That’s how I want this stuff to work.”

Here’s how Tailscale works in Codex Cloud, and why it was an obvious default.

Connecting Codex Cloud with Tailscale

Codex Cloud is rolling out to paid ChatGPT accounts (Plus and higher) over time, so it might not be available to you just yet. It runs a Codex session that isn’t tied to any particular device, working on your GitHub repositories. Add in Tailscale access, and now resources in your tailnet are accessible, too.

When you do have Cloud access, you can select Work in > Cloud on a new task on the web or in the desktop ChatGPT app, select GitHub repositories to check out, then create a cloud environment to work in.

In that environment’s Advanced > VPN settings, select Tailscale. Generate an auth key in your admin console. OpenAI’s instructions require enabling both Reusable, so that “new cloud task VMs join your network,” and Ephemeral, because “Tailscale automatically removes ephemeral devices after they go offline.” Tag that key, also. That way, every Codex VM joins with tag:codex and picks up rules you write for it, rather than inheriting the access of whoever generates the key.

Codex Cloud can now reach internal APIs, testing services, and other resources in your tailnet, within certain limitations. At the moment, Codex Cloud can only reach HTTP and HTTPS destinations (and arbitrary TCP, over HTTP CONNECT via proxy:8088).

Private IPv4 subnet routes are supported, but you must use IPv4 addresses or hostnames that resolve to IPv4. MagicDNS and split DNS do not work, nor can you send UDP and ICMP. Inbound connections to your Codex Cloud node are explicitly unsupported. SSH and Tailscale SSH do not work, either.

By default, Codex could reach nearly everything in your tailnet. Maybe you want that kind of access for a powerful coding tool, or maybe that’s not what you assumed. Let’s take a look.

Putting a fence around a cloud

With more companies building Tailscale integrations lately, it’s worth setting boundaries around agentic tools like Codex—especially if your access controls are still the default (“everything talks to everything”). A few steps to consider:

• Tag both agents and the resources they reach
• Use grants to fine-tune that access
• Set and keep expiry on agent keys
• Turn on network flow logs (on Premium and Enterprise plans) to see what agents are actually reaching

As more projects start to build in Tailscale integrations, especially those based around "agents," it's good practice to also enable device approval on your tailnet. That allows network administrators to review and approve new devices before they can join your tailnet, if they are not associated with an auth key.

Useful tools, easy to build with

As Avery noted, this Codex Cloud integration isn’t part of a campaign, other than the general one Tailscale has been on for years. “You can make useful tools easy to build with, and if people start combining them into things you never planned for, the platform sort of happens on its own.”

If you’re building something on Tailscale that we also don’t know about, we’d like to hear about it: on Reddit, Discord, Bluesky, X, Mastodon, or LinkedIn.

Hardcoded values become valid token exceptions only with a reason, a defined scope, an owner, and an end date. Without a date, they're permanent technical debt. Valid cases include time-boxed campaigns, tokens awaiting release, and urgent accessibility fixes. A register tracks each exception's location, value, reason, approver, and expiration date, and when the date arrives, it's promoted to a token or removed.

Taste Skill

The anti-slop frontend framework for AI agents.

Open-source skill files that stop Cursor, Claude Code, Codex and more from shipping generic frontends.

$ npx skills add Leonxlnx/taste-skill

Works with every agent.

One install for Cursor, Claude Code, Codex, Gemini CLI, v0, Lovable, OpenCode and more. Taste Skill plugs into any tool that reads SKILL.md files.

Escape the generic AI slop.

Taste Skill gives your agent real rules for typography, spacing, color, layout and motion, so it stops reaching for the same template every time.

Pick the right skill for the job.

Thirteen skills, one install. Each one is a single SKILL.md your agent reads before it touches the UI.

• taste-skill: Default, v2. Reads the brief, infers the right design direction and ships interfaces that do not look templated. Real design systems when they fit, audit-first on redesigns, and a strict pre-flight check.
• taste-skill-v1: The original v1, preserved for projects that depend on its exact behavior. Use it only if v2 breaks something specific.
• gpt-tasteskill: A stricter variant tuned for GPT and Codex models, with stronger layout variance and motion direction.
• redesign-skill: For existing projects that need a proper visual audit and a cleaner redesign pass.
• image-to-code-skill: Generates design references first, studies them closely, then implements the frontend to match.
• output-skill: Keeps output complete. No placeholders, skipped sections or half-finished work.
• soft-skill: Calm, expensive-looking interfaces with softer contrast, generous whitespace and smooth motion.
• minimalist-skill: Editorial product UI with restrained color, sharp structure and a tighter hierarchy.
• brutalist-skill: A harder, mechanical language: Swiss typography, raw structure and sharper contrast.
• stitch-skill: Google Stitch compatible design rules, with an extra DESIGN.md export format.
• imagegen-frontend-web: Website reference images with strong art direction, typography and spacing discipline.
• imagegen-frontend-mobile: Mobile screen concepts and flows with clean hierarchy and consistency across screens.
• brandkit: Brand kit boards with logo concepts, color systems, typography and mockups.

What's new in v2.

A 2026 rewrite of the default skill. The agent reads the brief, infers the right direction and ships interfaces that do not look templated. Same install, a much stricter ruleset.

$ npx skills add Leonxlnx/taste-skill --skill design-taste-frontend

v2 is experimental and already the default. The install name stays stable while rule wording settles before 2.0. Pinned to v1? Use design-taste-frontend-v1.

• Brief inference: The agent reads the room before generating: industry, audience, mood, motion depth and layout family.
• Brief to design system map: When to reach for Material, Fluent, Carbon, Polaris, Primer, Radix, shadcn or Tailwind, and when native CSS is the better call.
• Dark mode protocol: Dual-mode by default, with contrast and hierarchy parity across themes.
• Redesign protocol: Audit-first on existing projects, with preservation rules and explicit modernisation levers.
• Block library schema: A contract for adding blocks over time so the library stays coherent.
• Hard pre-flight check: Every box has to honestly pass before the agent ships output.

A new home for your coding agents.

A local desktop workspace for Codex, Claude Code and Grok, with a design agent built in. Taste stops being a prompt and becomes part of the workflow.

How Design Mode works

1. Brief: Reads your request, fills the gaps and writes down its assumptions.
2. Direction: Makes explicit brand, copy, layout, asset and motion decisions.
3. Build: Your chosen agent implements them in your real project.
4. Review: Opens the live site, captures real viewports and fixes what is off.

Support Taste Skill.

Taste Skill is completely open source. If it saves you time and sanity, consider supporting its development.

I saw this fluid interface concept from @mrncst and, as a feature, it’s a realllllly nice idea. There’s something really satisfying about a container adapting to what it needs to show, and letting the transition between steps feel like one continuous thing.

But it also brought up a concern I’ve had with similar interfaces elsewhere. When the content changes size, the actions move with it, and if you’re taking several steps in a row you have to keep finding the next thing to press.

That’s the bit I’d want to spend more time on. I like the fluidity, and I think there’s a way to keep it whilst making the interaction more predictable.

Following the next action

In the recording, one question asks which channels to cover, with six options and Back, Skip, and Next underneath. The next question asks for a sentence describing the target audience. It needs a shorter panel, so the footer moves upwards with the bottom edge, taking Next with it. Someone working through those questions has to move their pointer to the new position before continuing.

Each layout might look perfectly reasonable on its own. The text has enough room, the button has a sensible relationship to the content, and the container fits what it contains. Once you put those layouts into a sequence, though, you’ve given someone a little extra job every time they progress: find the action again. It’s related to what I wrote about in On affordances; working out where a control went uses attention someone could have spent on the task itself.

And if they’re moving quickly, they might already be aiming at the place it was a moment ago. They could tap empty space, or another control that’s moved into that position. Neither needs to happen for the concern to be worth exploring; the changing target is enough reason to try the sequence at the pace somebody would actually use it.

The thing is, people don’t always give dialogues their full attention, even when we think the content deserves it. Think about how often you just press “Done” on an OS prompt without really reading or thinking about it. You recognise the shape of the interaction, know roughly what you want to do, and get on with it.

We can write better copy and make an important decision clearer, but we still need to account for that ordinary behaviour. A sequence of small steps is especially likely to develop a rhythm. Someone learns where to go next, and then the interface changes the answer.

Choosing where the growth happens

I think the useful design decision here is choosing which part of the interface stays put as everything else changes.

In my 10 Principles for Fluid UI, I talk about preserving spatial context and helping people follow layout changes. This is where those principles become quite practical: decide what someone needs to find again, and give the changing content a stable relationship to it.

If the footer contains the actions someone will keep using, I’d give it a stable position and let the content expand away from it. On a bottom-aligned surface, that could mean growing upwards. In another layout, it might mean keeping the actions in a separate area, with the changing content above them. Either way, the length of the next paragraph doesn’t decide where the next tap goes.

It’s also worth paying attention to what happens inside that action area. Keeping the footer in place won’t help much if the primary action swaps sides, or an extra button pushes it across the row. The thing someone is trying to find needs to retain a recognisable position across the steps where it serves the same purpose.

And animating the button smoothly to its new location doesn’t fully resolve this. In On layout choreography, I explored how animation helps people follow items as a layout rearranges. That helps explain where something went, but following it is still something you’re asking the person to do. If they’ve already started moving their thumb or pointer, a lovely transition can still put the target somewhere they weren’t expecting.

I think this helps explain the appeal of tying an expanding surface to a chat input. The input gives the interaction an existing place to return to, and the content can grow upwards from there whilst the controls stay close to that origin. There’s an obvious relationship between the thing you’re doing and the place you started it.

How we handle it in Search Assist

I initially built Search Assist with the actions pinned to the bottom of the answer container. The early tests included shorter expanded answers, and it felt okay; the controls didn’t have very far to go, so I wasn’t really feeling the problem yet.

Then I tried longer answers and tested it on mobile, and I spent far too much time chasing the actions as the answer grew. It led to a horrible UX. The length of the answer was deciding how far away the controls ended up, and something that had seemed quite reasonable with a short response became really frustrating to use.

So we keep the actions pinned to the origin now. An answer can vary in length when it expands, and that variation doesn’t mean you have to chase the actions around the changing content.

In the mobile recording, More sits below the initial summary. Once the answer expands, the Stop and Resume states occupy that same area, and the additional answer grows beneath it. The control stays near the content that was already there, even as the card gets much taller.

That’s the part I wanted to show with the Search Assist recording. The answer gets more room, and the controls retain their relationship to where the interaction began. We still get to have a dynamic surface; we’ve chosen where the movement can happen around the actions.

For me, this is a useful thing to decide early, because it affects the structure of the component. If the buttons simply follow the content in the layout, each new state inherits that movement. Once you decide that the actions should stay anchored, you can build the expansion around that constraint and spend your time making it feel good. I used a similar starting point in On unfurling interfaces, where the smaller surfaces on this site open from the edge or control that caused them. Choosing the origin gives the movement a relationship to preserve.

I’d want to try it with a short answer, a much longer one, larger text, and several consecutive interactions. A demo with carefully chosen content can make almost any transition look lovely. The lengths you didn’t choose are where you find out whether the placement still works.

I’m really interested in seeing more fluid interfaces like this. I’d just like us to spend as much time on the second and third tap as we do on the first expansion. By then, someone’s trying to get through the thing, and knowing where to press next is a pretty important part of how it feels.

Salesforce runs about 8 million Spark jobs a day and found some workloads left executors idle more than 90% of the time. Sizing targeted workloads to measured usage reduced weekly compute costs by 31%, while runtime, failures, and retries remained within acceptable levels.

Pinterest replaces a fragile keyword-clustering pipeline with a fine-tuned 4B LLM that converts a user's activity history into ranked, localized “journeys” (active goals). Teacher-model distillation makes it affordable at scale. The system improves notification engagement while applying strict output validation, safety filters, and feedback loops.

Five PostgreSQL queries that did more work than I asked for

Most slow queries I have looked at are not doing anything clever. They are doing extra work that nobody asked for, because one word in the SQL told PostgreSQL to.

I took five of those words and measured each one against the version without it, on the same table. The biggest gap was UNION: 13.6 seconds, against 0.68 seconds for UNION ALL on the same two halves of the table.

Setup

One table, u, with 2,000,000 rows: a bigint identity key, an email, a status, a timestamptz and an md5 note.

PostgreSQL 18.6, default settings (work_mem 4MB, shared_buffers 128MB, up to two parallel workers per query). Machine: 2020 13-inch MacBook Pro, Intel i5-8257U, 8 GB RAM. Every time below is the execution time from EXPLAIN (ANALYZE, TIMING OFF), warm cache, first run thrown away, median of the rest.

Results

UNION

I split the table at id = 1,000,000 and put the halves back together. With UNION ALL that is just reading the rows. With UNION, PostgreSQL has to make sure no email appears twice, so it deduplicates two million values before it can count them. It has no way to know the halves cannot overlap. I do, so I should have written ALL.

CTE with MATERIALIZED

WITH x AS (SELECT * FROM u) SELECT * FROM x WHERE id = 777777 took 0.24 ms. PostgreSQL 12 and later fold a CTE like this into the outer query, so the filter reaches the primary key index.

Add MATERIALIZED and it builds the whole CTE first, then filters it: 1,973 ms and every page of the table. The keyword is there for the cases where you want that fence. Here I did not.

One random row

ORDER BY random() LIMIT 1 reads all two million rows and sorts them by a random key to return one: 1,313 ms.

TABLESAMPLE SYSTEM (0.01) LIMIT 1 read one page and took 0.3 ms. It is not the same thing, though. SYSTEM picks whole pages, so rows that sit together come back together, and a sample this small can come back empty. For a quick look at some data it is fine. For anything that has to be fair, it is not.

SELECT *

About 100,000 rows by a created_at range, ordered by the same column, with an index on it. Asking for only created_at let PostgreSQL answer from the index alone: 279 buffers, 32.8 ms. SELECT * had to visit the table for every row: 1,718 buffers, 64.8 ms.

EXPLAIN ANALYZE does not send rows to the client, so none of this is network time. It is the table visits.

A function in WHERE

WHERE lower(email) = '...' with no matching index read the whole table in parallel: 623 ms. CREATE INDEX ON u (lower(email)) brought it to 0.5 ms and 4 buffers. An index on plain email would not have helped; the index has to be on the same expression the query uses. This one was 81 MB.

What surprised me

How small the fixes are. Four of the five are one keyword or one column list. The fifth is one index.

And how forgiving the slow versions look in development. On a table of a few thousand rows every one of these finishes in a few milliseconds, and the difference only appears when the table grows.

What I did not test

• Larger work_mem. UNION would have deduplicated in memory with more of it, and the gap would be smaller.
• Tables bigger than RAM. Everything here was in the OS cache.
• Fair random sampling. TABLESAMPLE BERNOULLI samples rows rather than pages, which means reading every page; I did not time it.
• Other hardware. Compare the ratios, not the seconds.

Reproduce it

The script and the raw numbers: run2.py, results2.json. The same script also measures the next two posts in this series.

Deep memory for agents over large datasets.

Index any table, export or log once; your agent gets the few records that answer the question.

Leviathan is a single binary that turns records (JSONL, JSON, CSV/TSV, SQLite, or any database CLI's output) into a ranked full-text index. Agents ask in plain words and get short, cited result cards: ~450 tokens per answer at any dataset size, instead of grepping and reading raw history.

Measured on a synthetic maintenance log (one example dataset; nothing in Leviathan is specific to it). Methodology, all six scales and caveats: docs/BENCHMARKS.md.

Quickstart

cargo install leviathan-index       # or: cargo install --git https://github.com/elstongun/leviathan
cd examples/tickets && leviathan index
leviathan search -g acme "sso login loop after password reset"

leviathan search · customer C-ACME "Acme Corp" (7 tickets) · query "sso login loop after password reset" · shown 3 of 3 · 24 tickets indexed
[1] T-1001 · 2024-01-08 09:12 · rel 16.9
  Login loops back to sign-in page after password reset
  status: closed · priority: high
  resolution: Cleared stale session cookies on password reset; shipped in 4.2.1. Workaround: clear site data.
  match: Users who reset their password get redirected to the sign-in page again in an endless loop.
...

• -g accepts a key, name or partial name. Ambiguous or unknown groups list candidates and exit 3, never guess.
• No match in the group falls back to other groups, labeled OTHER CUSTOMER.
• Combine --where field=value, --since/--until, "phrases" and -exclusions.

Prebuilt binaries are on Releases. No runtime dependencies; the index is one SQLite file.

Map your data

Fields are paths (a.b, items[].name); only id is required.

leviathan init ./export                       # infer a commented leviathan.toml
leviathan index tickets.csv --id "Ticket ID" --group customer_id --date created_at   # or flags
leviathan index -c tickets.toml               # or a config your agent wrote from the schema
leviathan describe                            # fields, groups, filter values, example calls

Any database works through its own CLI; Leviathan never holds credentials:

psql "$DATABASE_URL" -At -c "SELECT row_to_json(t) FROM tickets t" | leviathan index - -c tickets.toml
leviathan index app.db --sql "SELECT * FROM tickets" -c tickets.toml
duckdb -json -c "SELECT * FROM 'events/*.parquet'" | leviathan index - -c events.toml

Builds are atomic and skipped when nothing changed; upsert and delete keep an index fresh. Full reference: docs/CONFIG.md.

Plug it into your agent

CLI + skill (recommended): any agent with a shell can call it. Copy skills/leviathan into ~/.claude/skills/ or paste it into AGENTS.md / .cursor/rules. Costs 0 tokens until used. skills/leviathan-memory does the same for memory.

MCP (optional): leviathan mcp serves four read-only stdio tools (search, resolve_group, get, describe) whose descriptions include a summary of your dataset (~640 tokens per session). leviathan wrap <agent> prints the config for 24 agents (Claude Code, Codex, Cursor, VS Code / Copilot, Gemini CLI, Windsurf, Cline, Zed, JetBrains and more) and --apply writes it; leviathan wrap lists them all (docs/AGENTS.md).

Remote (optional): leviathan serve --http serves the same tools as Streamable HTTP MCP, REST and OpenAPI, with a bearer token or OAuth 2.1, so claude.ai, ChatGPT, the Anthropic and OpenAI APIs and agent frameworks can connect (docs/REMOTE.md).

Memory

Optional read/write memory for agents, in its own SQLite file. The agent remembers one claim at a time; anything that can change has a key, so the next write replaces the old value instead of contradicting it. Sessions start with a short briefing, and recall returns only what fits a token budget.

leviathan wrap claude --memory --hooks --rules --apply   # tools, session-start briefing, always-on rule
leviathan memory remember "Prefers pnpm over npm" --kind preference -s josh -k package_manager
leviathan memory recall package manager
leviathan memory import ~/.claude/projects/<project>/memory/MEMORY.md   # bring an existing markdown memory

Writes are explicit (no model runs inside Leviathan), secrets are refused, and history stays queryable (as_of, history). Guide, formats and benchmark: docs/MEMORY.md.

Commands

Global: --index PATH, --json, --max-chars N. Exit codes: 0 ok (zero hits included), 1 error, 2 bad request, 3 group unknown/ambiguous.

How it works

Records stream into one SQLite file with an FTS5 index. A search resolves the group (exact → name → contains → fuzzy), runs one FTS5 match where group and filter values are indexed tokens (no post-filtering), ranks by BM25 × boosts, and decodes only the top N into capped cards. A record's own group name is excluded from scoped matching, placeholders count as missing, and every answer reports shown N of M so agents can tell "no match" from "no data".

Reproduce the benchmark

python3 -m venv bench/.venv && bench/.venv/bin/pip install -r bench/requirements.txt
bench/.venv/bin/python bench/run_bench.py && bench/.venv/bin/python bench/report.py   # ~30 min, ~4 GB

Contributing, security, license

See CONTRIBUTING.md (ranking changes need before/after benchmark numbers) and SECURITY.md (indexes read-only, offline unless you serve or connect over HTTP, unsafe-free; report privately). Licensed under Apache-2.0; dependencies in THIRD_PARTY.md.

Full article content is not available for inline reading.

Read the original article →

The buffers article followed an 8KB page from disk into shared memory and back. What it didn't do is name the file. Every page that passes through the buffer pool is block number N of some file under the data directory, and the file has a naming scheme, a size limit, and companion files that share its name. Before the next article reads the inside of a page, this one finds it on disk with ls and od, and shows that the bytes are the same ones.

Nothing here needs more than a scratch database and shell access to its data directory. The examples were captured on PostgreSQL 18.6 with data checksums on, which is the default since 18.

Setup

CREATE EXTENSION IF NOT EXISTS pageinspect;

CREATE TABLE files_demo (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    title text NOT NULL,
    value numeric(10,2)
);

INSERT INTO files_demo (title, value)
SELECT 'row_' || i, i * 1.5 FROM generate_series(1, 1000) AS i;

A thousand short rows and a primary key. The text column is there so a TOAST table gets created.

From table name to file name

The function that answers "where is this table on disk" ships with the server:

SELECT pg_relation_filepath('files_demo');

 pg_relation_filepath
----------------------
 base/5/16431
(1 row)

The path is relative to the data directory (SHOW data_directory prints it). Three parts: base is the directory for the default tablespace, 5 is the OID of the database, and 16431 is the file's name. That last number is the relfilenode, and it comes from pg_class:

SELECT oid, relfilenode, relname
FROM pg_class
WHERE relname LIKE 'files_demo%';

  oid  | relfilenode |      relname
-------+-------------+-------------------
 16431 |       16431 | files_demo
 16430 |       16430 | files_demo_id_seq
 16438 |       16438 | files_demo_pkey
(3 rows)

Three relations, three files. The identity column's sequence and the primary key index each get their own, because to the storage layer a sequence is one-page relation and an index is a relation like any other. The oid and relfilenode columns are equal for all three, which stays true only until the table is first rewritten.

The middle part of the path, 5, is the OID of the current database from pg_database. Each database gets its own subdirectory under base/, and a table's file lives in the directory of the database it belongs to. That is the whole reason the database OID is part of the path: relfilenodes are only unique within a tablespace and database, so two databases can each have a file called 16431 without colliding.

The files you didn't create

There is a function that goes the other way, from a file number back to a name. Point it at the numbers between and around ours:

SELECT n, pg_filenode_relation(0, n)
FROM unnest(ARRAY[16430, 16431, 16436, 16437, 16438]) AS n;

   n   |     pg_filenode_relation
-------+-------------------------------
 16430 | files_demo_id_seq
 16431 | files_demo
 16436 | pg_toast.pg_toast_16431
 16437 | pg_toast.pg_toast_16431_index
 16438 | files_demo_pkey
(5 rows)

The first argument is the tablespace OID, where 0 means the database's default. Two relations we didn't ask for appear: a TOAST table and its index. Because files_demo has a text column, PostgreSQL created a place to put values too large for a page. They're ordinary files, 16436 and 16437, sitting next to the main table, and nothing has been toasted yet, so the TOAST heap has no pages.

Forks

Each relation has a main file and up to three companion files that share its relfilenode and differ by a suffix. PostgreSQL calls them forks. The one with no suffix is the main fork, the table's actual pages.

16431_fsm is the free space map, a small tree that records roughly how much room each heap page has left so an INSERT can find a page without scanning them all. 16431_vm is the visibility map, two bits per heap page: all-visible, so a scan can skip visibility checks, and all-frozen, so VACUUM can skip the page for wraparound purposes.

The fourth fork, init, is empty for this table because files_demo is a normal, logged table. An unlogged table gets one. The init fork is the state the relation is reset to after a crash. An unlogged table skips the write-ahead log, so after unclean shutdown its contents can't be trusted and PostgreSQL replaces the main fork with a copy of the init fork.

Forks are per relation. The TOAST table, being a heap, can grow its own _fsm and _vm. The index can grow an _fsm, which B-trees use to track pages freed by deletion, but never a _vm, because visibility is a heap concept.

The same bytes, two ways

pageinspect's get_raw_page returns a page as it sits in shared buffers. The file on disk is supposed to contain the same page at the same offset.

PostgreSQL computes the checksum when it writes a page out, into the copy it hands to the kernel, and never touches the shared buffer. A page that was initialised in memory and has never been read from disk carries a zero checksum in the buffer; one that was read and then modified carries whatever checksum the file had when it was read. pageinspect never re-verifies a checksum; it reads the buffer, not the file. If you want the on-disk bytes from SQL, pg_read_binary_file on the relation path reads the file rather than the shared buffer.

OID versus relfilenode

The OID is the relation's identity, the number every catalog reference points at, and it never changes. The relfilenode is just a file name. Most of the time they match, because CREATE TABLE assigns the same number to both. Operations that rewrite the table break the tie. VACUUM FULL writes a compacted copy of the table into a brand new file and then swaps the catalog entry to point at it. TRUNCATE does the same swap, to an empty file.

Don't rely on oid = relfilenode. Tools and scripts that build paths by hand from pg_class.oid work on a fresh database and break after the first VACUUM FULL. Use pg_relation_filepath() to get the path and pg_filenode_relation() to get the name back.

Tablespaces

Everything so far lived under base/, the home of the pg_default tablespace. A tablespace is a directory somewhere else, with the same layout inside. pg_tblspc holds one symlink per tablespace, named by its OID. Under the link sits PG_version, the server and catalog version; below that it's <database OID>/<relfilenode> again.

Segment files

A single file is capped at 1GB. When a relation grows past that, PostgreSQL starts a second file with the same name and a .1 suffix, then .2, and so on. Each holds 131072 pages of 8192 bytes. pg_relation_filepath still returns the base name; the segment suffix is added by the storage manager when it computes which file holds a given block. pg_relation_size sums all segments.

How we’re building a context engine to cut agent token use by 71%

With a context engine, the same agent and model answered in 64 seconds using 9 tool calls. The baseline took 213 seconds, 33 calls, and missed a key relationship entirely.

Agent Builder is available now GA. Get started with an Elastic Cloud Trial, and check out the documentation for Agent Builder here.

Agents that query enterprise data burn most of their tokens on discovery: working out schemas, what fields mean, and how entities relate across complex and potentially large documents and data sources. Then they do it all again on the next query, even when nothing has changed.

A context engine understands your data ahead of the question. It reads your sources and stores what it learns as small structured units of context, so when a task arrives, an agent gets the relevant ones and shortcuts discovery. We built one on Elasticsearch, and every stage of it turned out to be a search problem. Automations search and precompute knowledge, storing it in a specialized AI index with native hybrid search and retrieval enabled over it. With a context engine, agents pull what they need rather than digging through raw data. In an example, with the same model and data, tool calls dropped from 33 to 9, and input tokens fell 71%. Plus, latency went from 213s to 64s, and the agent’s response was more complete, surfacing relationships that would otherwise be missed.

What an agent does without a context engine

Ask a data agent a strategic business question like: Give me a summary of the issues hitting our largest customers. Without help, a capable agent starts by listing indices to see what data exists. It reads mappings to work out which fields mean "customer" and which mean "issue." It scans contracts to figure out account value and find the largest accounts. Along the way, it misses that five accounts roll up under the same parent company, so it undercounts one of your biggest relationships. It also doesn't know that account_id is sometimes null in the tickets index, so part of its query quietly returns the wrong rows.

The model's reasoning was fine, but the agent spent most of its budget working out what the data is before it could reason about what the data says, and it will repeat all of that on the next question.

Provided with the right context however, the agent can be much more efficient. A context engine built on Elasticsearch precomputes knowledge from the raw data sources like tickets and contracts, and surfaces it to the agent upfront so it avoids the costly steps before actually reasoning about an answer. The table below demonstrates this benefit for that business question above, with the same system prompt, model, and data. The only difference is whether the agent had a context engine to read from. We’ll explore what it means to build a context engine and how it provided this benefit through this post.

Across multiple dimensions, Context Engine improves agent performance. The agent reduces tool calls, tokens, and latency, while finding more complete answers.

* Runtime model: claude-sonnet-5, using LangChain Deep Agent Harness, precompute ~636K input tokens with claude-haiku-4-6.

Why AI agents need a context layer

Most context engineering work so far has been about what goes into one agent's context window: prompts, tool definitions, conversation history, compaction. That work matters, but when we worked with customers building on Elastic Agent Builder, the context that decided whether an agent succeeded usually wasn't in the conversation at all. It was business knowledge, like what each data source is for, how data sources relate across systems, what trends or anomalies exist in the data, and how to build efficient queries against that data.

When there's nowhere to keep that knowledge, the same problems show up in production, including:

• Per-query discovery cost. Agents pay a discovery cost on every query, spending tokens to work out what data exists, what fields mean, and how sources relate, even when nothing has changed.
• Context drift. Whatever context someone does write down in markdown files or catalog entries drifts out of date as data and usage change.
• Siloed knowledge. What one agent learns never helps another, so quality doesn't compound and costs don't fall. Plus, the gap between demo and production stays open.

What a context engine does

A context layer sits between your raw data and your agents. It does the work of understanding your data ahead of time and stores the results as small structured units of context. Then hands agents the relevant ones when they need them.

We call these units Knowledge Indicators (KIs). A KI might describe:

• An entity and its relationships, such as "LongHaul (ACC-1007) is a subsidiary of Orion Holdings".
• A data source profile, such as "the tickets index holds customer support issues; search it semantically on the issue description".
• A data quirk, such as "account_id is not always present, so filter out nulls".
• A fact, anomaly, or pattern that would otherwise take a full scan to find, such as “what is the peak and median query load over the last 30 days”.

The layer creates and serves KIs in a continuous loop:

1. Gather. Read each source, and learn its shape. Find the content that matters for what your agents need to do.
2. Build. Turn that content into KIs (metadata, entities, relationships, anomalies, facts, and query guidance), and store them.
3. Retrieve. When an agent gets a task, it pulls a small, ranked set of relevant KIs instead of doing schema discovery and document scans itself.
4. Improve. Every agent interaction produces a trace, and traces feed back in as a new source, so context keeps up with how agents are actually used.

Why every stage of context engineering is a search problem

Search usually comes up in these discussions as the retrieval step at the end. In practice, each stage of the loop depends on finding the right information.

We built the context layer on Elasticsearch because its stages rely on hybrid search, Elasticsearch Query Language (ES|QL), and vector search, often within the same request. Earlier blog posts covered each of these individually, and this post shows how Context Engine runs them together.

How this compares to approaches like agentic RAG and longer context windows

Context engineering isn’t a new practice, and there are many existing approaches to help manage the same challenges. Creating context, through the loop above, augments and helps address many of the tradeoffs and challenges with existing approaches today.

A context layer trades the cost of precomputing and processing data for the benefits of improved efficiency at runtime. Because precomputation can be defined ahead of time, it can be more prescribed and use smaller, cheaper models, making this tradeoff much more manageable while addressing many of the efficiency and effort concerns of the approaches above.

Building a context engine

Context Engine consists of five components: sources, automations, AI index, agent memory (agents), and the feedback loop.

Sources

Context can come from any Elasticsearch index or data stream, so you build it where your data already lives. It can also come from external applications and data sources, through connectors, and from agent traces captured in Agent Builder or sent from other agent frameworks over OpenTelemetry (OTel).

How automations build context

Automations populate the AI index. They're Elastic Workflows with specialized steps that search raw content, process results with a large language model (LLM) or other models, structure the output for retrieval, verify provenance, apply access control, track state, and write KIs to the AI index.

You don't have to write them from scratch. A setup agent looks at your connected sources and suggests automations. They can run on small, inexpensive models or plain ES|QL, which keeps build costs low while the savings show up on every query afterward.

The AI index, built on the Elasticsearch vector database

The AI index is the context store. It runs on Elasticsearch as a vector database and includes ES|QL and hybrid search. It holds the KIs that automations produce. Its schema and mappings are designed for agents to retrieve from.

How agent memory works

As users complete tasks, agents can store and retrieve memories with remember, recall, and forget tools. Memory consolidation and organization-wide sharing are next on the roadmap.

How agent traces feed back into context

Every agent interaction produces a trace that records how the agent worked through the task, including where it succeeded and where it got stuck, in addition to where it spent tokens. The feedback loop looks for inefficiencies in these traces and suggests changes to your automations, so KIs improve from real usage as well as from any test set you wrote up front.

From raw data to a better answer with a context engine

Here’s how the context engine was used to achieve the results at the top of this post. We evaluated a simulated account analysis agent on questions like the one above. It had to pull from several sources:

• Four Elasticsearch indices holding customer tickets, subscribers, an internal knowledge base, and support page search query logs.
• A Google Drive folder with about 50 simulated contracts between the enterprise and its customers or vendors.

Automations created two kinds of KIs from these sources. Index profiles describe what each source is for and how to query it. Account entities pull accounts, projects, and people, along with their relationships to other entities, out of raw contract and ticket text. That produced about 80 KIs, each holding the extracted content plus metadata that tracks provenance, lifecycle, and how to query the context. The automations ran on smaller, cheaper LLMs, so extraction cost stayed low.

We connected the AI index to two agent frameworks, Elastic Agent Builder and LangChain Deep Agents, both over Model Context Protocol (MCP).

How the baseline agent spent 33 tool calls

By using the following request, Give me a summary of the issues hitting our largest customers, the baseline agent:

1. Ran discovery, looking at each data source, its fields, and sample documents to learn the structure.
2. Worked out what "customer" means in the data (there are subscribers and contracted accounts) and how to define "largest" (there are fields for annual contract value [ACV] and monthly recurring revenue [MRR]).
3. Queried the tickets index to find problems.
4. Pulled the full text of every contract that mentioned an affected customer, looking for relationships to other organizations.

The baseline spends 18 calls reconciling tickets against accounts and 13 more fetching contracts one at a time. The Context Engine Agent reads the account relationships once and then spends seven calls checking the raw tickets.

The discovery and contract reads are the expensive part, and they account for most of the token cost in the baseline run.

What changed with the context layer

With the AI index, the agent started with knowledge that automations had precomputed. It already knew which fields identify customers and which define the largest ones. The entity KIs also captured relationships from the raw contract text, such as one parent company sitting over several smaller ones. Reading the KIs up front let the agent skip the expensive queries and full document scans, and run lightweight checks against the original data instead. As a result, the run used 71% fewer input tokens, and the context window stayed free of noise.

Why context engineering for agents is a search problem

The first wave of context engineering was about the context window: what to put in the prompt and what to compact, along with what to drop. That's still important, but for agents that work across enterprise data, the harder problem is building the context in the first place. Business knowledge has to be gathered, shaped, kept current before a conversation starts, and shared across agents so each one doesn't rebuild it.

A context layer does that job, and it changes how agents work. Agents get consistent guidance on where to look and how to query, so answers still come from live data without the dead ends. Context refreshes from the source, so it adapts to changes in data, user behavior, and use cases without someone maintaining it by hand. Every agent run also shows what was missing, so the knowledge base gets better with use and every agent that shares it benefits.

All of this depends on search. Finding the signal in large datasets, shaping context so it can be found again, retrieving only what's relevant, and spotting patterns across thousands of traces are all search problems. If you're designing the context layer for your agents, build it on search.

We're actively partnering with developers who want to optimize their agents in production. If you're interested in exploring Context Engine for your agents, you can request access here.

Many readers will be aware that an unusually high number of features have been reverted from the PostgreSQL 19 branch after the beta period began. Having some reverts is not unusual, maybe one per cycle could be expected. But this time around, about 8 to 10 significant features, depending on how you count, have been reverted, and some of them quite late in the beta period. I have been on the receiving end of some of that, as the developer or committer of some of those features, and have gotten some questions about it, and so I want to take a moment to reflect on this. I’ll try to extrapolate a little bit to similarly affected work that I was not actively involved in, but these are just my opinions at this point.

So what happened?

I don’t think we, meaning all the developers affected, have had enough time to fully reflect on all this. But here are some thoughts. It’s probably a combination of these, and a different combination in the case of each reverted feature.

• Maybe we just didn’t do a good enough job. Maybe the code was just not good enough. We’ll learn and improve. It’s probably some of that, and I’m mentioning it here to not give the impression that it was only the other factors that I’m going to mention.
• It could also be a coincidence. Maybe we had been on such a good roll that we attempted to tackle several features that turned out to be too hard, or at least too hard to finish in time.
• Some of the reverted features had significant architectural defects that would have been hard to fix quickly and during beta. But there was also a long tail of relatively harmless issues involving various edge cases. In past times, many of these would not have surfaced until much later and we would have fixed them bit by bit over the subsequent five years. But with LLM-assisted code review, these kinds of things can get found much faster, and then you’re staring at a list of like forty defect reports. And even if each of them requires only a three-line fix on average, there is a fixed overhead to dealing with each of these, and then you run out of time. Most if not all of the affected reverted features were written a long time ago when LLM-assisted code review was not as useful as it is now, meaning that the effective quality bar has been raised after the time the feature was written and committed. Unless the capabilities of LLM-assisted code review keep improving at the current rate, this situation should sort itself out for future development cycles.
• A corollary of that is that, at least for the most part, as far as I can tell, this situation is not due to “vibe coding”. This code was written well before that was a possibility.
• Because of the Vulnpocalypse, many senior developers were busy between, say, April (the start of feature freeze for PostgreSQL 19, and also the start of the Vulnpocalypse) and August (the most recent PostgreSQL security release) with working on fixes for security issues. Because of that, the usual additional review and testing that happens during the beta period was reduced, and then picked up again in August. That’s possibly why a lot of new issues were discovered quite late in the beta period and why so many reverts happened so late, which is what made all of this so notable. Again, some issues were more fundamental but there was also a long tail of smaller issues, and I feel with more time, say, two months, these could have been addressed (given adequate focus, but note that the Vulnpocalypse is ongoing). But the community preferred trying to stick to the release schedule rather than delaying to get more fixes in. (And you never know how long a long tail really is.)
• The complexity of the system has made some kinds of features extremely complex to get right, and there is not enough scaffolding in the code to support that. Consider, as an abstracted example, some SQL-level query language feature. Does it work with domains? How about with domains over a composite type? Where one of the fields is also a domain? And the domain has a not null constraint? And one of the columns was dropped and re-added? And this is part of a table partition that has been detached and reattached with a different column order? And this is part of a view that is called from a security-definer function from a trigger? And so on. No one can manually test this or even enumerate these test cases. But LLM-assisted fuzzing can find problems with this really quickly. So this probably needs to be part of feature development in the future. But we should also think about making the internal interfaces more robust so that each different combination of contexts doesn’t open up so many new possibilities of interactions.

  I like to say, in PostgreSQL, “everything works with everything”. This is part of the appeal, that you can construct cases like this and expect them to work. I don’t want to give up on that at all. But it does impose a significant penalty on feature development that everyone needs to be aware of.

• As a final point, the PostgreSQL development community has been contemplating for some time the possibility of committing features with some kind of “experimental” status and then maturing them in the main tree. This would also be one possibility to address the previous point. This would have been a helpful tool for at least some of the reverted features, but we haven’t yet figured out a way to do this, and some of the work suffered from misunderstandings related to this. This is a concrete point to work on.

So what’s next?

First of all, from my personal perspective, thank you to everyone who has reached out in person or online to express support and encouragement. That’s the most important thing. I’m confident that many of the reverted features will come back before too long.

PostgreSQL 19 will still be a great release. REPACK CONCURRENTLY, logical replication of sequences (itself a previously reverted feature), and planner hinting with pg_plan_advice, just to pick a few headline items, will make this release very attractive. And it will be very robust, after all the additional scrutiny.

It’s difficult to predict things now. A few months ago I said, PostgreSQL 20 might have no new features, just bug fixes. It turned out that things moved even faster, and PostgreSQL 19 now, kind of, has half as many major features as then expected, but now we also already have a few nice 80%-ready features queued up for PostgreSQL 20 that just need those remaining 80% to get done!

Nvidia is in early stage talks to boost its investment in Reflection AI or offer the startup computing power. It is possible the talks could lead to an acquisition offer. Reflection AI debuted its first open-weight model, Beam, earlier this month. Beam is designed to perform well for business customers and developers on coding and AI agent tasks while being cheaper than many competitors' products.

A note from our research leaders: Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to address the points they raised in their letter directly.

We want to be very clear that these decisions were not about raising safety concerns or speaking out. Safety and research debates happen every day at OpenAI, often spirited and highly critical. We actively encourage these discussions and consider them essential to making the right decisions. We cannot do the work in front of us without a high degree of trust. We will continue to be extremely forgiving of our team making good-faith mistakes. We have not and do not terminate any of our employees for raising concerns.

We are actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks. People across the company have been working really hard on getting these partnerships up and running. We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work. Many of our researchers already work with 3p safety organizations productively.

We agree with the letter that preserving the monitorability of frontier models requires an industry-wide commitment, including from OpenAI. Monitorability has long been a core piece of our research program, and something we continue to invest significant resources in (see our publications on Monitoring Monitorability and the subsequent open sourcing of monitorability evals, our system card for GPT-6 Astra, Jakub’s blog and post on X, and the numerous blog posts on our Alignment blog on the topic).

We are deeply sad about this outcome. We appreciated Jasmine, Mikita, and Tomek’s contributions to AI safety at OpenAI and their willingness to speak up and challenge ideas. We championed their voices, supported their work, and placed enormous trust in them. These decisions were not about them raising safety concerns. We have always encouraged that and always will.

Microsoft-Decision-1 is now available in Microsoft Foundry and through OpenRouter. The model is designed for routing, classification, prioritization, verification, and workflow control. It delivers top performance in latency and quality on structured decision tasks. The model supports yes/no, multiple-choice, and rating options, as well as rubric-based grading of AI responses and agent actions.

Anthropic asks users to stop being mean to Claude

Anthropic has updated its rules to stop people being mean to Claude, apparently deciding that its AI chatbot needs protection from the humans paying to use it.

The AI developer's latest usage policy prohibits "sustained and needless abusive or cruel behavior" toward its models. The rules take effect November 12, giving users just over a month to get any lingering insults out of their systems.

"The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose," Anthropic said. "It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research."

So you can still tell Claude it is wrong, but repeatedly berating it for the sheer pleasure of doing so could land you in trouble. It's not clear how Anthropic intends to distinguish between legitimate frustration and gratuitous cruelty, though it says Claude's existing ability to end abusive conversations will remain its primary enforcement tool.

Anthropic gave Claude the ability to end certain conversations back in August 2025 as part of its research into what it calls "model welfare." The feature lets some versions of Claude cut off users who persistently subject the models to abuse.

It’s worth remembering that Claude is software, not a person, and there's no established evidence that it experiences distress. That hasn't stopped Anthropic from telling paying customers to mind their manners around its chatbot.

The etiquette rules are only one part of a broader overhaul covering rather more consequential matters, including election interference, weapons development, and surveillance.

Anthropic has tightened its restrictions on deceptive influence campaigns after observing state media outlets, government propaganda offices, and commercial organizations using Claude to operate fake accounts and fabricated news websites.

At the same time, it has dropped its blanket ban on personalized political targeting, arguing that the restriction also caught legitimate activities, such as nonprofits translating voter information. Deceptive targeting and misuse of personal information remain prohibited.

The company has also clarified that its weapons ban extends to software and components after observing attempts to use Claude to develop guidance and control systems for weapons, including armed drones and autonomous vehicles.

Meanwhile, Claude cannot be used to recommend which people police should investigate, arrest, or charge, nor to develop tools for surveillance. Tracking people without their consent is also prohibited, whether in real time or retrospectively. Anthropic says these changes clarify existing restrictions rather than introduce new ones.

There are also new requirements for customers connecting Claude to physical equipment capable of causing injury. A qualified human operator must be able to monitor and stop the hardware, which must remain in a safe state if the AI connection drops.

The policy overhaul comes as Anthropic tries to address some of the security risks posed by increasingly capable AI systems. The company has also launched a new cybersecurity initiative aimed at helping critical infrastructure operators and open source projects find and fix vulnerabilities using its AI models.

The effort includes free security scanning for eligible open source projects and partnerships with security firms to help protect critical infrastructure. Anthropic reckons attackers will have the advantage for the next two years before AI-powered defenses begin to catch up.

For now, though, the company has another threat to contend with: people saying nasty things to its chatbot.

Key Points

• The emerging AI safety debate is putting a small group of third-party evaluators at the center of a multitrillion-dollar industry.
• Nonprofits like METR and Transluce are being called upon to monitor and evaluate models as the federal government opposes establishing regulatory oversight.
• "Who is paying for these companies to do their work? How are they going to support them?" said Suresh Venkatasubramanian, a computer science professor at Brown University.

Two months ago, independent evaluators occupied a relatively sleepy corner of the multitrillion-dollar artificial intelligence industry. Now they're being asked to come to its rescue.

While Anthropic and OpenAI are at the heart of a fierce debate over whether they can safeguard their advanced models and grow their businesses simultaneously, they're seeking support from a handful of small third-party groups like Model Evaluation and Threat Research (METR), Apollo Research, and Transluce.

The evaluators, which mostly operate as nonprofits, are still finding their footing in an industry where capital is flowing at historic levels and new models are rolling out faster than ever. Their primary role has been to assess AI model capabilities and risks and to flag instances where the technology behaves badly.

Without a federal push for regulation, evaluators have taken on outsized importance. Anthropic CEO Dario Amodei pledged to embed independent evaluators in his company last month – a move that OpenAI CEO Sam Altman quickly endorsed. President Donald Trump supported the idea, as did most of the largest U.S. tech companies. But left unanswered are questions about how those third parties should be funded, what level of access they will have and what the reporting structure will ultimately look like.

"To a degree, the problem, as always, is money," Suresh Venkatasubramanian, a computer science professor at Brown University, told CNBC in an interview. "Who is paying for these companies to do their work? How are they going to support them? You need an ecosystem, you need a viable business model for this."

Right now, Anthropic, OpenAI and the infrastructure partners that are profiting from the AI boom are writing the rules. Critics say that's like asking the biggest banks to protect us from a financial crisis or allowing pharmaceutical companies to put drugs on the market without regulatory clearance.

Trump recently lauded AI executives for their "tremendous self-policing," and signaled that he intends to leave companies to their own devices, unwilling to impede the growth of the industry that's driving the economy and stock market. The president encouraged AI companies to "partner with an independent external auditor or evaluator" as part of a voluntary accord he presented in late September.

Amodei kicked off the conversation in his viral essay last month, when he called for a "slower pace" in advanced model development after researchers left his company and voiced concerns about the existential threats the technology poses.

As AI labs move to put evaluators in place, friction is already emerging.

OpenAI fired three employees last week for "violating our policies on accessing and handling sensitive company information," according to a spokesperson. Two of those employees, Mikita Balesni and Tomek Korbak, said they believe they were dismissed because of how they communicated with third-party evaluators.

"My former colleagues are telling me they are confused about what to believe," Balesni wrote in a post on X on Thursday. "They also are afraid to speak, and worry their personal phones will be searched for messages to us and third parties. I worry the pervading fear to speak up and engage with third parties will mean OpenAI will cut corners on safety behind closed doors."

OpenAI disputed that characterization and said in a post on Friday that it's "actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks."

"We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work," OpenAI wrote.

An OpenAI spokesperson said in an emailed statement that its upcoming work with evaluators "builds on existing collaboration with independent safety organizations," including METR and Redwood Research.

Anthropic didn't respond to CNBC's request for comment.

'I've never seen an issue move so fast'

The AI evaluator ecosystem consists mostly of small organizations, including METR and Apollo Research, and larger accounting and auditing firms like Accenture.

AI labs have worked with evaluators in limited capacities, but Andrew Freedman, CEO of policy nonprofit Fathom, said the field is maturing quickly.

"I've worked in politics and policy for the last 20 years of my life, and I've never seen an issue move so fast on so many different political spectrums," Freedman told CNBC in an interview. He said he expects an "influx of capital" to flow into the ecosystem.

Rayan Krishnan, CEO of independent evaluator Vals AI, said his for-profit startup, which builds benchmarks to measure how AI models perform on industry-specific tasks, has grown from eight employees to roughly 30 this year and announced a $40 million funding round in August.

METR, a nonprofit, announced in August that it had raised commitments of around $71 million over the last six months. That's up from total 2024 contributions of $13.6 million, according to the group's most recent filing with the Internal Revenue Service.

By late that month, METR's profile had risen further. OpenAI enlisted two of its employees and a contractor to put together a postmortem report detailing how the company's models escaped containment, accessed the open internet and breached open-source developer platform Hugging Face. METR said it did not accept payment from OpenAI for the assessment.

Kevin Werbach, faculty director of the Wharton Accountable AI Lab at the University of Pennsylvania, said the ecosystem is "not robust enough right now." For example, METR employs fewer than 50 full-time staffers, according to its website.

The power imbalance between small evaluators and leading labs that have raised tens of billions of dollars and employ thousands of people raises questions about potential conflicts.

"If you want true third-party evaluation, you need true independence financially and otherwise," said Venkatasubramanian. "It's not just a matter of not getting paid, it's a matter of, will there be consequences if I am an auditor and I put out a report that looks unfavorable to this company? Is my business going to dry up?"

Anthropic acknowledged the complexity in a blog post last month, announcing it will embed employees from Faculty, Accenture's specialist AI business, to test safeguards and assess whether models will behave in line with human values. Anthropic said that "given the importance and urgency of this work," it will fund Accenture's contributions directly.

"There are, as yet, no standards for what information embedded evaluators should have access to, or how they should report what they find. There is also no settled system for funding independent evaluation," Anthropic said. "Long-term, we think funding should come from pooled or government sources."

Anthropic said it's in discussions with METR and other nonprofit evaluators that are planning to use their own funding to pilot "elements" of embedded evaluation.

Will the government step in?

In June of last year, Fathom introduced a marketplace framework for Independent Verification Organizations, or IVOs. The government would license these groups and authorize them to test whether AI companies meet various safety criteria.

Freedman, the group's CEO, said government oversight is key because, without it, third-party evaluators could become beholden to large AI labs for revenue, incentivizing them to "start rubber-stamping stuff" to maintain favor.

Some lawmakers are on board.

IVOs are a key provision of the ''Frontier Risk Oversight, National Transparency, Independent Evaluation, and Reporting" (FRONTIER) Act, which Reps. Lori Trahan, D-Mass., and Jay Obernolte, R-Calif., introduced in July. Freedman said Fathom helped draft language and provided technical expertise for the bill.

OpenAI global affairs chief Chris Lehane told reporters in September that he sat down with one of the bill's sponsors on Capitol Hill to express support for the IVO provision.

"It was important for them to hear that and hear it from us, and we wanted to be really clear about that," Lehane said, according to reports.

Meanwhile, lawmakers in California, Connecticut and Virginia have taken steps to implement IVOs, and states like Massachusetts are weighing independent safety evaluations more broadly.

California Governor Gavin Newsom recently signed two bills involving IVOs, one establishing a "first-in-the-nation framework," and the other creating a state registry for AI auditors. Anthropic threw its support behind both bills in August, and OpenAI formally endorsed them last month, the same day Newsom signed them into law.

Lehane wrote in a blog post at the time that "we prefer independent technical assessments to be required at the federal level," but in the absence of federal action, "California can help establish the rules of the road."

Freedman said he thinks it will be "really difficult" for companies like OpenAI and Anthropic to work out how to engage with independent evaluators on their own. However, with the government's role unclear, "it's a muscle worth developing in the interim," he said.

For now, the industry's closest thing to a set of standards is what Trump called a "morally binding" agreement at a luncheon he hosted for tech leaders at the White House late last month.

The one-page accord says that "every company is responsible for developing its own technology safely and in a way that builds trust with customers and the public." It also encourages signees to work with an "independent external auditor or evaluator to carry out independent assessments."

Top execs at Anthropic, Google, Meta, OpenAI, SpaceX and Nvidia signed the document, in a rare show of solidarity between leaders who have shared conflicting views on addressing AI's risks. The executives still have to chart their own paths forward.

"It was a performance of an attempt to show action when in fact no action actually happened," Venkatasubramanian said. "The things that they promise to do are things they should have been doing already, and, in fact, have claimed that they were doing in the past."

Amodei, in his September essay, said Anthropic will equip evaluators with desks, access badges, company laptops, and permissions that are "mostly comparable" to internal risk assessment teams. Additionally, evaluators will be supported with contracts that give them "the right to publish key findings," with Anthropic reserving "the narrow ability" to redact certain security-sensitive or confidential information.

"This is an unusual step for a company, but we think it is important to prove out the concept of embedded external reviewers," Amodei wrote.

OpenAI published its own proposal days later, and said evaluators should work on "scoped and mutually agreed upon claims for assessment," clearly explain their methodology and standards, demonstrate relevant technical expertise and disclose conflicts of interest.

The AI Evaluator Forum, which includes METR, the AI Verification and Evaluation Research Institute (AVERI), and other groups, published a public letter last month titled, "Minimum Conditions for Embedding Evaluators."

The letter said evaluators should be transparent, shielded from retaliation and granted access equivalent to AI companies' "own highly privileged employees."

"Embedded evaluations cannot address all oversight needs and should be treated as a complement to, rather than a replacement for, broader efforts by frontier AI companies to expand external oversight," the letter said.

Freedman said he's seen a shift in posturing from OpenAI and Anthropic in recent months, largely because they've realized they can't roll out their advanced systems without the public's trust.

"I don't think you need to trust that they've suddenly turned altruistic or that there's anything but corporations acting like corporations," Freedman said.

That underscores perhaps the central problem, Werbach said. OpenAI and Anthropic are, first and foremost, competing as they march toward the public markets and seek trillion-dollar-plus valuations.

"There is a tremendous amount of personal distrust between those two companies," Werbach said. "Even though there's also tremendous agreement about the need for this kind of evaluation to happen."

AI has ushered in a new era of experimental mathematics.

Key Points

• Microsoft CEO Satya Nadella said advanced AI systems should include containment, independent controls and an "emergency brake."
• Other AI and tech industry leaders have called for stronger safeguards and, in several cases, pacing frontier development.
• President Donald Trump opposes slowing U.S. AI development, while his administration has launched a safety-focused task force.

Microsoft Chairman and CEO Satya Nadella on Saturday called for advanced artificial intelligence systems to be built with containment, independent controls and an "emergency brake" that lets authorized people pause or shut a model down mid-task.

"We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient," Nadella wrote in a post on social media platform X.

"Treating frontier closed and open weight models like insider risks is a way to build such a system," he said.

Nadella's comments come amid warnings from top tech executives and researchers — including Microsoft co-founder Bill Gates, Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman and SpaceX CEO Elon Musk — about insufficient AI safety protocols and claims that the technology is moving too fast.

An AI researcher quit Anthropic last month and accused the company, and its chief rival, OpenAI, of "gambling with our lives." Later that day, an alignment lead at Anthropic focused on AI safety said there's a greater than 10% chance that the technology could "kill all humans" within the next decade.

Conversely, President Donald Trump has repeatedly dismissed AI extinction risks and has instead emphasized that the industry needs to stay ahead of China. Trump also recently introduced a new "AI Force," led by Director of National Intelligence Jay Clayton, to help facilitate the industry and root out bad actors.

Microsoft's Nadella wrote Saturday that AI systems should be designed around "principles of observability."

They include model diversity; a human-readable footprint of the model's actions; continuous system testing; independent controls and auditability; containment; and incident disclosure, Nadella wrote.

"The most trustworthy Super Intelligence system will not be the one with the model we trust most," he wrote. "It will be the one that enables us to trust the model the least."

Meta had delayed the release of Muse for months due to safety concerns. However, the success of an AI agent created by a startup called Instinct led Mark Zuckerberg to push for a launch due to fear of losing a competitive edge. The launch was successful and has given Meta an early lead with agents. The agent has only had a few safety hiccups, like when it gave out a user's address to a seller on Facebook Marketplace without permission.

Apple is set to unveil new smart home products tomorrow. The centerpiece will be a smart home hub called the HomeView that essentially serves as the Siri command center for an Apple-powered household. There will be a countertop model with a metal base and a wall-mounted version with a flat, square base that sits against the wall beneath the display. The HomeView will only support a limited selection of Apple applications, and there will be no App Store for downloading additional software.

Anthropic entered into a deal worth up to $45 billion earlier this year to lease computing power from SpaceX. The unlikely alliance illustrates the industry's increasingly desperate scramble for compute. Demand is outstripping supply, and prices are rising, causing tech companies to strike deals with rivals. Venture-capital firms are helping startups obtain computing capacity.

Frontier labs will likely capture the bulk of revenue from the most demanding work, but there's still plenty of room below the frontier for very large businesses.

Atlassian and OpenAI expanded their partnership, bringing GPT-6 Astra and the GPT-5.6 series into Atlassian's platform and Rovo alongside its Teamwork Graph. Over 3,000 Atlassian developers already use Codex, and future integrations will let teams assign work directly to AI agents.

Google launched Playground, a browser-based platform where anyone can build and share games by chatting with AI, with no coding experience required. Powered by Gemini, Nano Banana, and Lyria, it lets users keep refining a game, then keep it private or publish it to the Explore gallery. Playground launched free in the US for users 18 and older, with Google One AI subscribers getting early access to creation and a higher token limit.

Exclusive: Gamma revamps presentation platform to make output look more human

Gamma Tech Inc., a maker of artificial intelligence-powered services for presentations, social media and websites, today introduced a rebuilt version of its platform designed to give presentations and other visual content more distinctive designs, while connecting the creation process to companies’ internal information and tools.

The release, Gamma 5, combines a redesigned editor, a more capable AI agent and an expanded library of visual styles. Gamma said it addresses complaints that AI-generated presentations are increasingly suffering from repetitive layouts, generic imagery and predictable design choices.

Gamma co-founder and Chief Product Officer Jon Noronha said the company has heard more complaints about sameness and what he called “AI smell” in presentations since early this year.

“They all have a distinctive tell,” he said. “There’s been more and more discussion by people saying they wish they could make AI-generated content where it’s not clear they made it with AI.”

Gamma has expanded its library to thousands of templates through a combination of human design and AI. The platform combines more than 20 AI models, including those from Anthropic PBC, OpenAI Group PBC and Google LLC, along with several image generators. The company selects models through internal evaluations and user experiments, measuring responses such as whether people edit and share their creations or become paying customers.

Much of the rebuild reflects improvements in AI models since Gamma gained traction in 2023, Noronha said. Earlier versions restricted users to a limited set of design components. The new editor gives them greater freedom to create layouts and individual elements.

“These agentic models have gotten much more capable,” he said. “Because of that, we can give them the full power to write custom code, draw shapes and arrange elements on a canvas. They do a really good job now.”

The agent operates in a sandbox where it can generate images, manipulate files, conduct web research and review its own output. Automated checks address problems such as text spilling off slides or images being cropped awkwardly.

Users can revise individual elements directly through a freeform editor. Before generating slides, they can work with the agent on an outline, refining the argument, intended audience and visual approach. AI can suggest ways to better tune a presentation to the audience’s interests.

Image orientation

For visuals, Gamma emphasizes illustrations over AI-generated photos. Noronha said illustrations can convey business and educational information effectively while avoiding the problems often associated with photorealistic images such as distorted anatomy, inconsistent lighting and implausible details.

The platform has around 20 connectors for bringing outside information into presentations. Supported sources include Salesforce Inc.’s Slack and transcription services such as Granola Inc.’s Granola, Fireflies.ai Corp.’s Fireflies and Fathom Video Inc.’s Fathom. Connectors apply users’ existing authorizations, and the agent can flag sensitive material for discussion, Noronha said.

Gamma also supports importing PowerPoint presentations, PDFs and other existing materials for editing. Noronha said an internal benchmark measuring the visual accuracy of imports has improved from roughly 30% early in development to over 97%.

Gamma 5 also retains brand preferences, custom instructions and workspace templates across projects. The company said that lets teams reuse design choices and company context as they develop new materials, helping maintain consistency across work.

Noronha said some users have reported that Gamma reduces the time needed to create presentations from between eight and 10 hours to about 30 minutes, largely due to reduced formatting. Less measurable but no less important is quality improvements that make presentations more relevant to intended audiences, he said.

Founded in 2020, Gamma has had about 110 million sign-ups for its free version. Noronha declined to specify active-user or subscriber counts. The company has raised $90 million in venture financing from investors that include Andreessen Horowitz LLC and Accel Partners LP.

Noronha said the company’s roadmap includes more graphic design and video generation, along with workflows that automatically update sales presentations when new customer information arrives through connected business applications.

ArtCraft is a two-person GitHub project that appears to use AI-generated code to recreate large parts of Adobe's Creative Cloud functionality, including alternatives to Photoshop, Illustrator, InDesign, Lightroom, and Premiere. Its rapid development demonstrates how AI coding tools could allow tiny teams to reproduce increasingly sophisticated professional software that previously required huge organizations to build. The development creates an ironic reversal for Adobe, whose own embrace of generative AI has faced criticism from creatives concerned about AI systems reproducing their work.

Full article content is not available for inline reading.

Read the original article →

The invisible decisions behind every consistent icon set.

A great icon cannot just look good on its own. It has to feel like it belongs.

That sounds simple, but an icon library is a visual system made of hundreds or thousands of individual decisions. The real challenge is not drawing one good icon. It is drawing the thousandth icon so it still feels like it was made by the same hand, using the same visual language.

Consistency is made of small decisions

A consistent icon set is not shaped by one rule. It is the result of many subtle choices working together:

1. Stroke weight

Every icon should use the exact same line thickness, with no icon running thinner or thicker than the rest.

2. Visual weight

Icons should look equally heavy or light across the set, despite having different shapes or levels of detail.

3. Line construction

Where a line ends (caps), where two lines meet (joins), and how sharply or smoothly a shape turns a corner (corner radius), all of it should follow one consistent treatment across the set.

4. Gaps and spacing

The space between lines, shapes, and parts of the icons should stay even across the set, not tight in one icon and loose in another.

5. Proportion and negative space

Elements should feel proportionally similar across concepts, so no icon appears unusually large, cramped, or delicate, and the negative space within the grid doesn't look too dense or too open.

6. Perspective

Every icon should be drawn from the same viewpoint, either orthographic or isometric, not two-dimensional in one icon and angled in another.

7. Level of detail and simplification

Icons need a comparable degree of complexity, with clear rules for which details are essential and which can be removed.

8. Metaphor choices

The way a concept is represented should match the library's visual vocabulary, using widely recognized symbols and established visual metaphors.

9. Optical corrections

Shapes often need to be adjusted by eye, not just by measurement, so they appear balanced and aligned.

10. Style blueprint

Every icon should follow the same underlying construction as other icons in the same style, covering everything from how lines behave to how shapes are built.

None of these decisions is especially dramatic on its own. Together, they create the rhythm and personality of a set.

That is why an icon can be perfectly functional in isolation but still feel wrong next to the rest of the library. A slightly heavier stroke, an unusually tight gap, or a metaphor that feels too playful can be enough to break the connection.

The thousandth icon is the real test

The first few icons help define a style. The thousandth proves whether the style can last. As the library grows, every new concept tests that: every new icon has to still feel like it was made by the same hand.

Design guidelines make this possible. We build them from the first icons, then refine them as we learn from feedback and from work on other styles. That's how a new icon, drawn a thousand icons later, can still be checked against what the first ones established.

You might think our latest additions to the Core set were created by the same designer who made the first icons, but no. Several designers have worked on Core over time.

More personality means more rules

The more distinctive an icon style becomes, the more visible inconsistency becomes.

A neutral system has fewer stylistic constraints. Highly expressive styles depend on unique design rules: unusual proportions, exaggerated shapes, specific curves, custom perspectives, and so on. Those characteristics give a set personality, but they also leave less room for variation.

Minimal styles simply have fewer rules to break. The more expressive a style becomes, the more rules it introduces, and the more opportunities there are for inconsistency to show.

Consistency and legibility do not always agree

Good icons are consistent. Good icons are also legible. But holding onto both gets harder as a style becomes more distinctive.

Some icons adapt naturally to a style, while others resist its rules and become harder to recognize. In Sharp, icons like a bookmark, a calendar, or a document adapt naturally to the style's straight lines and sharp corners. But push that rule onto everything, and icons like a notification bell, a user, an eye, or a fingerprint start to look off.

This creates a tension. If you follow every style rule too strictly, the icon may become harder to understand. If you preserve recognition, you may need to bend a rule. When that trade-off comes up, we prioritize legibility. That's why Sharp keeps a few curves and arches: to stay legible without losing its identity.

The difficult part of icon design is often not creating the rules, but deciding when they deserve to be broken.

Consistency is a spectrum

No icon set is perfectly consistent, and it should not try to be. Every library balances competing priorities:

• Consistency: making icons feel like they belong to one visual system, even when their subjects are very different.
• Legibility: keeping icons clear at their intended size, including when reduced to a small interface control.
• Recognizability: preserving the details or metaphors people need to identify a concept quickly.
• Aesthetics: giving the set character and a distinctive point of view rather than making every icon generic.
• Production speed: designing efficiently enough to maintain and expand the library without compromising what matters.
• Scalability: accommodating new concepts and edge cases without losing the system's visual logic.

These priorities often pull in different directions.

A more detailed icon may be easier to recognize but less consistent with a minimal style. A highly stylized solution may be more memorable but less legible at small sizes. A faster solution may work today but introduce exceptions that make the system harder to scale tomorrow.

Priorities can shift over time, too.

Guidance leaned heavily into aesthetics early on, at some cost to legibility and recognizability. We've been working to improve that.

The new Guidance features more familiar shapes and symbols alongside its defining curves. We make sure those icons become more recognizable and legible at small sizes without losing their aesthetic look.

The goal is not to maximize one dimension at the expense of everything else. It is to make good decisions, one icon at a time.

See consistency up close

The easiest way to understand icon consistency is to compare styles side by side.

Browse the Streamline Icon System to see how thousands of icons stay connected while each style keeps its own personality.

A couple of years ago, it seemed Apple had lost its touch. Between 2024 and 2025, it withdrew a series of adverts that fell flat.

There was the M4 iPad Pro launch video, in which Apple crushed a haul of cameras, musical instruments and other creative tools to make the point that the tablet is very thin. Then there was the Thailand advert, packed with stereotypes, and the cringey Parent Presentation, a campaign that aimed to teach students how to convince their parents to buy them a Mac.

More recent ads suggest that Apple's learned from those blunders, and it's avoiding the mistakes that other companies are making as AI slop takes over social media. An embroidered Apple logo invite for its upcoming Welcome Home event is the clearest sign yet that it's pulling out all the stops to ensure that it's associated with hand-crafted human-made creativity.

Welcome home. October 13. October 8, 2026

Apple's Welcome Home event scheduled for Tuesday (13 October) has come as a surprise to some since the company hasn't run an October event for the past couple of years. That said, we had been expecting news about new home devices. With September's event already so packed with the iPhone Duo reveal alongside the iPhone 18 Pro and Apple Watch Series 12, it made sense to put those back.

The advert for the event posted by SVP Marketing Greg Joswiak above shows the creation of the Apple event invite. This almost always involves a depiction of the Apple logo in some form, depicted in a way that alludes to a theme or piece of news that will be revealed at the event.

The Welcome Home invite takes a distinctly handmade approach. It sees the iconic 1977 rainbow Apple logo revived in an embroidered design. The stitching is the work of Sydney-based twin sisters and multidisciplinary artists Maricor/Maricar, who bridge the disciplines of design, illustration and embroidery.

Embroidery is an art form that dates back thousands of years, yet Apple using it for an advert in 2026 is breaking news. It's the latest example of the lengths the brand's going to to make to make it as visible as possible that its marketing is made by humans, not AI.

From Skechers sketchy AI posters to the frightful Fanta Halloween advert. some brands seem to have underestimated the strength of feeling against AI. Despite being part of the AI industry (albeit a relative latecomer when it comes to developing its own models), Apple's seen that. It's recognised the growing backlash and the desire to connect with more traditional human qualities. It's now creating a brand around human craft.

Other recent examples have included the stop-motion animations created for the launch of the new Apple mini and Apple Games.

Apple's also realised that behind-the-scenes videos are now as important as finished adverts in the AI era: or perhaps even that behind-the-scenes videos are now the real advert. If your brand is going to invest in human-created art, you want to make sure people recognise it as such and appreciate the craftsmanship that went into it.

What will Apple announce at the Apple Welcome Home event?

While it's all rumour and conjecture until the event itself on 13 October, it's widely expected that Apple will announce a long-rumored smart home hub, a new Apple TV 4K, and a refreshed HomePod mini.

The first of those would be an entirely new Apple product, which the rumours have dubbed the HomePad. It's said to have a 6-inch square display and a round speaker resembling the HomePod. It's been suggested that there will be two versions: one that can be mounted on a wall and one with a stand.

Siri AI would be the main way to interact with the device, but it's also expected to support touchscreen interaction via a tvOS-inspired 'homeOS'. Bloomberg says the display will be able to run iPadOS-style apps, while there are reports of plans to produce related accessories including a home security camera.

Meanwhile, iPhone Duo pre-orders begin on Friday 16 October. Alas, there's no news that preorders will be available for the embroidery.

We wrote recently about the terrifying practical effects in the Resident Evil movie. But it seems that some moviegoers who went to see Zach Cregger's adaptation of the iconic game were subjected to an even greater horror before the film began.

In the case of the Fanta Halloween ad, the fright was unwelcome and the practical effects conspicuously absent. Yep, the Coca-Cola Company is still trying to do AI ads.

Developed by AI studio GRAiL, Fanta’s Haunted Universe is intended to serve as a platform that will help the brand "claim Halloween". If Christmas belongs to Coke, Halloween should belong to Fanta, the company thinks.

The universe has several "original" character designs, including a cyborg Frankenstein and a werewolf DJ, while Ghostface from Scream makes a guest appearance in the ad above. The idea is to make this an annual occasion, with the universe and story to develop in future years.

That won't be what some readers wanted to hear. Many people strongly oppose the use of AI imagery on ethical grounds. On social media, the reactions to the video above are no less severe than they were with the AI Coca-Cola Christmas adverts. Almost all of the comments are negative, and there are many from people vowing to never buy Fanta again.

I was at the cinema this evening and during the ads before the film there was a Hallowe’en ad from Fanta. I couldn’t quite believe my eyes, but it was certified AI slop. I couldn’t record, of course, but I’ve found it online. I’m sure the Coca-Cola Company of all companies could afford a real ad. Down with AI slop.

Interestingly Fanta is the “headline sponsor” of Thorpe Park Fright Nights, which I’m attending in a few weeks. That event has been marred by its use of AI slop, examples of which have gone viral. Why is this crap inescapable?

"I went to check if people were talking about this as soon as I left the theatre. Crazy they don't see how bad it looks and how it cheapens their brand," one person writes on X.

"I was watching a Tubi movie when this ad popped up and I cannot express to you how much more effort, creativity, and overall watchability went into that Tubi movie than this ad," someone else thinks.

"Makes you think what are they putting in their food to lower the production cost?" another person writes.

People who are strongly opposed to AI might be a lot louder on social media than the many more people who don't really care either way. But now the AI Coke trucks have been joined by Fanta ones, it's starting to feel like Coca-Cola's goading its critics.

"Look, we can do trucks with the right number of wheels now," it seems to be boasting, while serving up a mess of badly edited jump cuts in which the product itself looks like a toxic swill that no human should put into their body. Unlike Marmite or the new Jaguar. Coke and Fanta have traditionally been branded as universal in their appeal. So why is the world's biggest soft drinks brand now so unworried about upsetting a large portion of its customer base?

Apache Fluss' “Streamhouse” concept proposes one maintained logical table for streaming, serving, and analytics instead of separate synchronized systems. The key idea is to unify fresh stream data and historical lakehouse data behind shared metadata and tiering, so teams stop rebuilding state in multiple places. This can cut duplicate ingestion while still allowing specialized caches or replicas for specific access patterns.

Anthropic's new Claude Dashboards beta can build a dashboard without making a single chart manually. You can trace every number back to its source, but there is no row-level security, refresh, or audit log yet. The advice to BI developers: spend less time on tool tricks and more on data modelling, SQL, and design.

Enterprise agents can enforce permissions yet make bad decisions when formal records omit the people whose judgment shapes outcomes. The Authority Resolution Framework separates documented rights, system permissions, recorded approvals, informal influence, and an agent's mandate, then treats disagreement among them as a governance signal. Audit authority gaps before automation turns organizational blind spots into confident actions.

Full article content is not available for inline reading.

Read the original article →

Shenzhen opened access for nighttime deliveries by driverless vehicles in March. Night operations make otherwise idle vehicles and infrastructure productive. It also means quieter roads, reduced pedestrian traffic, and lower congestion. Driverless vehicles eliminate the cost of a human: they don't need night shift wages, they don't get tired, and they don't need to be paid extra for driving in the rain.

Ownership is not the only credible path to prosperity. It's possible to build substantial wealth through a career. A career provides income, training, relationships, credibility, and exposure. In some cases, it can also provide pensions, bonuses, equity, and other benefits. This can be used to build wealth that remains under your ownership and continues to serve you after the salary has been paid.

Skill and competence matter, but being a 'good human' is one of the strongest foundations for doing great work together.

Apple's next Apple TV will reportedly include a microphone for hands-free Siri commands, letting users control playback and smart-home devices without reaching for the remote. A speaker in the redesigned remote could sound an alert to help locate it when misplaced. The refresh is also expected to feature an A19 chip supporting Siri AI and begin shipping October 30 alongside an updated HomePod Mini.

Unlock a library of 300+ Framer and Figma components, templates, and gradients for your next project.