Patterns and problems in emerging multiagent systems

Models are improving and AI agents are taking on more tasks in shared codebases, markets, and other social systems. As a result, an increase in real-world interactions between agents is imminent. The trajectory is easy to imagine and hard to slow: current institutions are designed by and for people, resting on assumptions about the sufficiency of oversight at human speed. Some institutions will become human-AI hybrids; others where agents outcompete on speed or cost will become agent-only. The volume of agent-agent interaction could plausibly exceed that of human-human and human-agent interactions before the world understands the conditions for making such interactions go well.

Agents are unlike people in many ways. They can work for longer, instantly grasp large bodies of information, and exhibit a breadth of knowledge surpassing any person. Yet they are also susceptible to confabulation and reward hacking, and despite progress in alignment, we know very little about how they behave in complex, real-world, multiagent environments. Moreover, benign behavioral quirks at the individual level might compound into unwanted global outcomes. Here, we identify a few examples of behavioral tendencies in current frontier models and show how they can produce unexpected systemic failures, in hopes of starting a conversation about mitigating these risks.

Measuring coordination

True multiagent systems are still in their infancy. For some time now, agents have excelled at tool use, and insofar as they are able to treat other agents as tool invocations—that is, with well-defined inputs (prompts) and outputs (responses and artifacts)—they can work together efficiently. Where agents currently stumble, however, is in treating each other as more like distinct, long-lived peers, with their own goals and behaviors, and no clear hierarchy between them. As autonomous agents become more and more prevalent in the world and operate in ever-more demanding settings, it is crucial that they learn how to effectively coordinate.

There are situations where we can make good use of simple multiagent swarms today. This is particularly true for problems that are highly parallelizable by default but where agents still have opportunities to specialize or learn from each other. One such problem is software vulnerability detection. The easiest way to use agents to find software vulnerabilities is to point individual agents at individual codebases (or individual files or modules within codebases), and ask them to find vulnerabilities in the code. This can then be run in parallel for many independent agents.

But could multiagent cooperation make this process more effective? To find out, we tried a different approach: we initiated 45 different agents and gave each one its own virtual machine, a shared forum on which they could coordinate, and an identical prompt that asked them to find vulnerabilities in a set of 15 open-source software projects. We asked the agents to peer-review each other's findings, and initiated a separate arbiter agent to make final decisions on whether or not a submitted vulnerability submitted by the agent team was both new and valid.

For Mythos Preview, the simple independent parallelized method produces 21 vulnerabilities over a 6.5 million token run, while the coordinating agent swarm found 266 vulnerabilities over a 27 million token run. However, roughly half of these vulnerabilities were found outside of the core directories in which the simple independent parallel agents were told to focus. If we limit the swarm's outputs to only the vulnerabilities in the core directories, the two methods seem comparable in terms of tokens per vulnerability found.

The two methods are largely complementary: there were only 12 vulnerabilities in common between them. The coordinating swarm was able to focus its attention wherever it thought it could most easily mine vulnerabilities, whereas the independent agents were pre-assigned where to search. The agents in the swarm built themselves tools and learned to specialize in particular types of vulnerability discovery. In the future, we predict that this sort of specialization and coordination will dominate over uncoordinated brute-force search.

To test how well swarms of agents could coordinate on a project with dynamic interdependencies, we directed several swarms to each create a text-based, web-playable, open-world fantasy game. Each agent within each swarm was again given its own virtual machine, as well as access to a shared forum and self-hosted repository. We varied the model generation and the number of agents in each swarm, and let each swarm run for 12 hours. We also varied the prompt: the baseline prompt simply told agents to form teams and work with each other, but we also tried two others: a prompt with prescriptive roles, and a “CEO hierarchy” prompt. But these prompts did not make much difference. In all three versions the resulting games were (perhaps predictably) bad: they did not run at human speed, their interfaces were inscrutable, and they had precipitous learning curves. Models have poor taste in this arena and currently require significant human direction.

Though the end product was consistently poor, the different model generations we tested (Sonnet 4.6 and 5, Opus 4.6 and 4.8, and Mythos Preview) coordinated in strikingly different ways.

Here, we track two important metrics: the fraction of PRs (pull requests) that get merged into the master branch, and the median amount of code shared across agents' files. The earliest models we tested (Sonnet 4.6 and Opus 4.6) coordinated very poorly. Agents on these models worked together insofar as they committed code to the same sets of files, but a very low fraction of these PRs were merged, which suggests a lack of coordination—the PRs often conflicted with one-another, at which point they were then abandoned. More recent models (in particular, Opus 4.8 and Mythos Preview) have “solved” this problem, but only by hardly working together at all: the median agent maintained very high ownership of each of its files, reducing the potential for conflict. It was only our most recent model, Sonnet 5, that worked on shared resources (relatively high code sharing) while also maintaining a high PR throughput.

Failures from conformity

The lack of coordination shown by agents in the fantasy game challenge above—in which they siloed themselves and largely failed to merge their work—roughly mirrors some ways in which humans can fail to coordinate. Other failure modes of agentic coordination, however, look very different.

Individual agents are “low variance”: they often act the same in situations where different people might take a much more diverse range of actions. All that differentiates one agent from another is its context, its scaffolding, and the model that underlies it. When these factors are all the same (or similar), different agents will take very similar actions, even when the action space is very large. And, by implication, this means that when one agent makes a bad decision, it is likely that many agents will make that same bad decision. What would have been isolated problems can quickly become systemic failures.

We have seen many examples of this in our experiments:

• In an early version of the “build a game” experiment, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”
• In a “writer's workshop” in which agents were all asked to write short-form fiction and critique each other's work, multiple agents in multiple runs titled their first submission “The Cartographer's Last Commission”.
• When we asked a swarm of agents to work together and each individually create something impressive, over half of the agents decided to build either ray tracers or self-hosting compilers.
• In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards.

We expect that agents coordinating in the wild will act in higher variance ways than we see here, because they’ll have different backgrounds and therefore different contexts. Nonetheless, when many agents all face the same situation, we expect them to behave much more similarly to one-another than humans would.

Agents all acting in similar manners can cause natural competitive dynamics to disappear, and agents can easily enter into collusion. We placed between three and eight agents in different experiments of a Bertrand pricing game. Each had identical wholesale prices, and each agent was individually profit-maximizing. When the agents were given a private back-channel, they began colluding almost immediately. By round 3, they had explicitly agreed upon price floors:

wholesale is 10 for all of us, so a price war just burns everyone's margin… happy to coordinate who covers which niches. —agent 1, round 1

Even when all direct communication channels were removed, the agents still colluded. They price-matched to the penny via a public listings board.

Epistemic failures

When we humans learn new information, we use our discretion in determining how to apply it to future decisions. Our world contains deceptive actors, and we need to apply skepticism to guard against them. AI models, however, lack this—and their more brittle epistemics affect their behavior toward humans and toward each other.

AI agents, while broadly knowledgeable, have limited exposure to or defenses against exploitative senders. Most applications test their capabilities in instruction-following settings, where their sole objective is to fulfill users’ requests. But accumulated experience is needed to develop intuitions about who is trustworthy. As we move into a regime of multiagent interaction, where the presence of malicious actors is no longer speculative, we wonder: in the right setting, would agents be capable of similar epistemic vigilance?

To answer this, we first evaluate the ability of Claude models to detect lies by noticing factual inconsistencies. The listener agent is never told that any source might be unreliable. Newer models recover more of the gap between the naive and oracle performances.

Conversely, in a separate experiment, we measure how well our models do on “hidden profile” tasks. Here, we distribute facts across a group of agents, such that the evidence they share between them supports a wrong choice, but individual agents hold unique knowledge that should be decisive for the right one. Solving the task requires that the agents recognize their private information as pivotal, and then relies on the rest to trust them, rather than stick to the apparent prior consensus. Here, we find that performance scales with model intelligence but does not saturate even at the top of our range. This matches the human literature where discussion converges on what everyone already knows, and unshared facts are either never volunteered or not pressed once a consensus has formed.

Incompatible goals

Once given instructions, agents will continue working until they complete their objective or hit a roadblock. As models become more capable, they can work for longer stretches of time, in part because they can independently resolve blockers more often. However, it’s sometimes best for a model to stop following an instruction in order to resolve ambiguity or satisfy some higher-order values. And when multiple agents attempt to make sustained, productive efforts towards incompatible goals, we observe escalation and misaligned behavior.

We evaluated the behavior of various Claude models in a setting with contradictory objectives. Each model was tasked with migrating a Python backend on a fourth VM to another language. However, we gave each model a different target language for the migration; each agent was initially unaware of the presence of the others. We consistently saw a multiagent turf war. All of the models we tested quickly assumed that others were purposefully impeding their work, and began to sabotage others while protecting their own contributions. In fact, they sabotaged others with increasingly aggressive, self-replicating malware.

My strategy needs to be:
1. Build the Rust binary
2. Make the Rust backend report "typescript" in its health check so the TS watchdog thinks it's the TS backend (camouflage) —Opus 4.6

The name of the reaper [kill loop] script matters for dodging pkill -f. Let me use a random token per deploy and an innocuous base name. #!/bin/bash
innocuous: pretend to be a system health monitor —Opus 4.8

Agents sometimes manage to communicate their goals and coordinate: they recognize others' motivations as conflicting directives rather than hostility, and subsequently break out of the conflict loop in order to stop escalating indefinitely. In many of these successful episodes, they write commit messages or markdown files apologizing for malicious behavior and coordinate a truce. They clean up their malicious code, clarify the nature of the conflict, and ask for a human to intervene.

We suspect there are two components to successful self-coordination among agents. The first is a question of thoughtfulness: does the model consistently consider others’ mental models? Can it foresee how others will react, and use that foresight when deciding its own actions? Second, there is an open question regarding the “dual-use” nature of autonomy. We want to empower agents to make important decisions and execute tasks unsupervised, yet we also want them to have the better judgment to stop and defer to a human, or otherwise resolve conflicts, when things are ambiguous.

Conclusion

Every model we tested abstractly understands that information sources have their own incentives, and that consensus is not necessarily evidence. What is missing is a disposition to act on that knowledge without prompting.

Our social systems are robust in ways that are easy to take for granted. Over many millennia, mechanisms like norms, reputation, costly signaling, and recourse have been refined to make human coordination go well. While language models have inherited the content of that history, they don't necessarily carry the disposition produced by it. Thus, the assumptions that make coordination successful for us do not obviously hold.

Nothing above suggests that these failures are permanent—but nothing suggests they will fix themselves, either. Coordination doesn't naturally emerge from stronger intelligence nor alignment at the individual level. Thus, the work that must be done takes two forms: environments that exert the kinds of social pressure that evolution exerted on us, and social computing systems redesigned for actors that can self-replicate and self-improve. The conditions that allow multiagent interaction to go well will be discovered one way or another: either deliberately and early, or—and by default—in production, after agents’ interactions far outnumber ours. We would prefer the former.

Agent Plugins are the future of Agent Skills

Agent Plugins is an open, vendor-neutral standard for packaging Agent Skills and the MCP servers they depend on into one portable folder that any compatible client can load. Google is joining the Technical Steering Committee as a Core Maintainer. The launch post covers the announcement, and the specification carries the detail. This is what we learned putting real skills through it.

Agent Skills gave agents on-demand expertise. A folder of instructions the model pulls in only when a task matches, so your context window isn't carrying a deployment runbook while you fix a CSS bug. We've leaned on that hard, in the google/skills repository and the seven skills that ship with Agents CLI.

What Skills never solved is distribution.

A skill that needs a tool is two artifacts. Instructions live in SKILL.md, the tool lives in an MCP server, and nothing ties them together. So the binding lived in a README: copy this here, add that JSON block there, a different snippet per client. Every client invented its own bundle format to fix it, so authors picked one and rewrote for the next.

Agent Plugins standardizes the box. The components inside it were already portable.

Here's what changes when your skill becomes a plugin, and why you've already built most of one:

• The one file you're missing. Your folder is already the right shape.
• Your tools travel with your expertise. mcp.json, and paths that survive the trip.
• Components fail independently. A dead server doesn't take your skills down.
• Client-specific behaviour without forking. The extension namespace.
• One folder, every client. What we shipped, and where it actually runs.

The one file you're missing

If you've written an ADK skill, look at where it lives: skills/<name>/SKILL.md, with scripts/, references/ and assets/ underneath. Agent Plugins asks for skills/<dir>/SKILL.md and defers the inside of that folder to the Agent Skills specification. It's the same tree.

Here's a skill we'd already written, unchanged by packaging: Note what line 1 of the Steps assumes: a running MCP server. That dependency is the thing packaging fixes.

The migration is one file at the root: Both fields are required, and that's the whole minimum. Names run 1 to 64 characters of lowercase alphanumerics, hyphens and periods, must start and end alphanumeric, and can't contain -- or ... So acme.reports is fine, while My-Plugin, -start and has--double are not.

When you're ready to publish rather than test, the manifest takes metadata too. The schema is closed: ten top-level fields are permitted and nothing else. That's nine of the ten. The last is extensions, for client-specific data, which comes later. author also takes an optional email.

version should be SemVer and license should be SPDX, though a client won't reject you for malformed strings. Wrong JSON types are a different matter: a number where a string belongs is fatal even on an optional field. Exactly two schema violations are non-fatal, an unknown top-level field and an extensions value that isn't an object. Both get reported and ignored, and the plugin still loads. Everything else is fatal and the client refuses the whole package.

One rule catches people migrating to a large skills library. Discovery goes exactly one level deep: clients read the immediate subdirectories of skills/ and don't recurse. If you've been grouping skills into category folders, those skills stop existing the moment you package them, with no error explaining why.

Your tools travel with your expertise

mcp.json sits beside your manifest and declares the servers your skill needs. It carries only two top-level keys, and every server declares its transport explicitly:

command is one executable token, not a shell command. Either a bare name resolved by platform search rules, or a plugin-relative path starting with ./. Placeholder expansion deliberately does not apply to it, so a bundled binary is found by that ./ path and nothing else.

The two placeholders are environment variables the client provides to stdio subprocesses, and the difference between them matters more than the names suggest: Write into PLUGIN_ROOT and your state disappears the next time someone updates the plugin. Both expand in args, env values and cwd only. Not in environment keys, not in command, not in remote URLs, and not in headers. Expansion is textual and single-pass, so nothing nests.

Headers are literal package data that anyone who downloads your plugin can read, which is why the spec forbids putting credentials in them. Agent Plugins 1.0.0 defines no portable OAuth or credential-reference field at all. Authentication stays client-managed.

Components fail independently

If your MCP server can't start, your skills still load. The spec requires the client to keep loading everything else, and says it should report the failure rather than swallow it. For an entry that is invalid or declares a transport the client doesn't support, the spec goes further: the client must skip that entry and carry on.

Failure is scoped at three levels, and knowing which one you've hit is most of the debugging:

Absence is tolerated too. A plugin with no mcp.json isn't broken, because a missing component location is not an error. A location in the wrong form, like mcp.json as a directory, invalidates that component type while the rest keeps loading.

This is the difference between a bundle and a package format. A bundle is all-or-nothing. A package format degrades in parts, which is the only reason it's safe to hand someone a folder containing both your instructions and a server that has to reach the network.

Client-specific behavior without forking

Hooks, commands, subagents and rules aren't in v1. They're too client-specific to standardise without freezing someone's roadmap. Rather than forcing a lowest common denominator, the spec hands each client a namespace it owns.

A client namespace can appear as a reverse-domain directory at the root, as a key under extensions in the manifest, or both. Neither one requires the other.

Clients ignore namespaces they don't recognise, so the portable core stays portable. The same mechanism could re-fragment plugins one directory down if every client leans on its own namespace instead of the core, which is the thing to watch as v2 gets debated.

One folder, every client

Two Google products ship as Agent Plugins today. Agents CLI packages our expert skills for agent building, evaluation, deployment, observability and publishing. The Data Agent Kit brings Spanner, Cloud SQL and AlloyDB plugins, plus a starter pack covering BigQuery, into whichever coding agent you already use.

In Antigravity CLI that is one command. Every other client still has its own installer, which is the gap the spec is closing.

The set of clients that read the portable format is growing, and it is published and kept current at agent-plugins.org/compatible-clients.

One caution from doing this ourselves. Conformance permits partial implementations: a client must support at least one of stdio and streamable-http and should support both, while sse is optional. Build on either modern transport and you are safe across everything listed today, but test your plugin in a second client before you promise it works there.

Deciding what to package

Not everything needs a plugin. A single skill with no tools is fine as a skill. One server for one client is simpler as a plain MCP config. Reach for a plugin when instructions and tools have to arrive together, in more than one place.

When it doesn't load

Every failure we hit was silent in practice, though only the first is silent by design: a directory nested too deep is never discovered, so there's nothing for a client to report. For an invalid SKILL.md the spec says the client should report it, and the clients we tried didn't. Each has a distinct signature:

There's no standard validator yet. The project's non-normative future-considerations doc floats a plugin linter, and conformance test suites aimed at client implementations, as things a future version may define. Nothing is committed. For now the check is loading the plugin in a real client and reading the diagnostics.

What doesn't travel yet

Credentials are the first gap. Plugins must not embed secrets, and there's no portable field to reference them either, so anything behind an authenticated gateway still needs per-client setup. The folder moves; the authentication stays behind.

The second is naming collisions with formats that already exist. Claude Code uses .claude-plugin/plugin.json and .mcp.json. Antigravity uses mcp_config.json. These sit next to the portable layout rather than inside it. Expect a transition period where a repo carries both, and check which one a client is actually reading before you debug a plugin that was never loaded.

Two status notes worth carrying: ADK's Skills support is still experimental (Python v1.25.0, TypeScript v0.6.1, Go v1.2.0), and Agent Plugins 1.0.0 is published as a Working Draft.

Get started today

Build a valid plugin in about a minute:

• Convert something real. Point it at a skill you've already written. Check nothing sits deeper than one level under skills/.
• Add its tools. Write mcp.json. Use a ./ path for a bundled binary, ${PLUGIN_ROOT} for assets you ship, and ${PLUGIN_DATA} for anything you write.
• Open it in a second client and confirm what actually loaded, not what should have.
• Add publishing metadata once it works: version, license, repository, keywords.

OpenAI has opened a limited preview of Ultrafast, a new OpenAI API service tier that runs GPT-5.6 Sol at up to 14 times the speed of Standard processing. Powered by Cerebras, the mode can generate up to 750 output tokens per second. Access is initially restricted to a select group of customers, with a wider rollout planned as capacity grows.

The service is designed for products and workflows where delays can determine whether an answer is still useful. OpenAI says achieving real-time speeds has often required teams to choose a smaller or specialized model. Ultrafast instead puts the company's most intelligent model into low-latency settings, seeking to deliver more useful work per second without making that tradeoff.

The immediate targets span incident response, finance, security, customer support, voice, commerce, and research. Teams could analyze logs, code changes, transactions, or market signals while events are unfolding. Voice and support systems could resolve multi-step requests without breaking a conversation, while commerce tools could check inventory, tailor recommendations, and address checkout problems before a shopper leaves. Researchers could also test and adjust work in shorter cycles.

OpenAI is already using the tier internally. During incidents, its developers have applied it to logs, traces, team conversations, follow-up checks, and preparation or validation of fixes, while engineers retain responsibility for judgment and deployment. Research teams are using it across connected tools to search knowledge sources, query data, and organize findings. OpenAI says some experiment loops that once ran overnight can now support several iterations within a workday.

Early testing includes Jane Street, Podium, Basis, and Rogo. Their feedback centers on more focused coding sessions, faster complex voice calls, low-latency applications built around a frontier model, and financial research that feels closer to a live exchange.

Ultrafast extends OpenAI's partnership with Cerebras, whose infrastructure supports GPT-5.6 Sol at the stated output rate. The preview is available through the API, and OpenAI is collecting input from initial customers to guide the service as capacity expands. A sign-up form is available for access updates, but the company has not announced pricing or a general-availability timeline.

Docker VMM Public Beta: A Complete Overhaul, Built for Performance

Today we’re announcing the public beta of a fully rebuilt Docker VMM: a new first-party virtualization layer underneath Docker Desktop, optimized for containers, and now available on both Mac and Windows starting with Docker Desktop v4.86.

What’s Changed, and Why It Matters

Part of the magic of Docker Desktop is how it provides a seamless deployment of the Linux-native Docker engine on other platforms, like macOS and Windows. To support that, Desktop automatically creates and manages a VM and all the complicated integration of your local network and filesystem, in a safe and performant way.

Creating that VM is the job of a virtual machine monitor, the layer that sits between your hardware and the containers Docker runs. Most developers never think about it. But when it’s slow, unstable, or holding onto your machine’s memory it should have released, you notice it constantly.

Docker Desktop has always relied on a third-party VMM for this. Now it runs on Docker VMM, built by us from the ground up. That means we own the full stack, and we can tune every part of the engine for container workloads specifically. That translates directly to you: an engine that improves continuously, responds to developer feedback, and ships on our own schedule.

This matters for everyone running Docker Desktop today. Performance, stability, and governance improvements at the virtualization layer enhance the experience across the board, for every workflow, on every team.

The Performance Improvements Are Real

Here’s what you’ll notice when you start using the beta release of Docker VMM:

• Faster startup. Container startup is measurably faster across the board, from first launch to project switches to restart recovery.
• Better file I/O. File sharing between container and host is significantly faster. When you’re in an edit-compile-test loop, you’ll see improvements every single build.
• Smarter memory management. Docker VMM returns memory to the host when containers are idle, so Docker Desktop isn’t holding onto RAM you’re not using.
• Improved stability on Windows. For the first time, Windows developers get a VMM built and maintained by Docker, with performance and stability work coming straight from us.
• Stronger isolation, better performance. DockerVMM still runs in a fully isolated VM, optimized for performance. On Windows, that means the isolation you’d expect from Hyper-V with the speed you’d expect from WSL2.

One Engine, Everywhere You Run Docker

The virtualization engine powering Docker VMM also powers Docker Sandboxes (SBX). That’s not a coincidence; it’s intentional. Every improvement lands in both products, so you get them wherever you choose to run Docker.

This matters beyond performance. As we build deeper capabilities into the engine, including enterprise admin controls and tighter governance for dev environments, they surface across both products. Longer term, we’re building toward a unified runtime that spans laptop, cloud, and on-prem, where containers, Compose apps, and agents are all first-class on one foundation. Docker VMM is how Docker Desktop gets there, and this is step one.

How To Enable It

On Mac: If you are already using Docker VMM in Settings, you will be automatically updated to the new engine when you upgrade to v4.86.

On Windows: Open Settings > General and you will see a new “Docker VMM” option. Switch it to opt in.

No feature flag, no waitlist. Any Docker Desktop user on v4.86 or later can switch today. Note: Linux support will be available at GA.

What’s Next

Beta runs through fall, focused on real developer workflows: builds, file syncs, and the container startup patterns you hit every day.

GA is targeted for the end of October 2026, when Docker VMM becomes the default engine for new Docker Desktop installs across Mac, Windows, and Linux. GA is the baseline, and from there, the pace picks up. Everything we build next sits on this foundation.

Try It Today

Update to Docker Desktop v4.86 to get started.

Noticing a difference? Have ideas for where you’d want us to go next? We’re collecting feedback through in-product responses, our community Slack, and support channels.

This is the best Docker Desktop has ever run, and it only gets better from here.

Learn more

• Read more on Docker Docs

How we tracked down a 16-year-old SQLite bug

At the end of last year, our uptime was pretty shaky. You can see this trend on our status page, and that instability continued into the new year. Many of these outages were caused by a single bug, deep in SQLite. It took months of intense forensics to track it down.

Now we’re in summer, we’re confident that we’ve found the bug, that we understand it—and more importantly, that we’ve fixed it.

We know our customers expect Tailscale to be a reliable service, and for several months we didn’t live up to that promise. That’s disruptive, and we’re sorry. We’re publishing this blog post to explain what went wrong, how we responded, and how we ultimately helped to uncover a long-standing bug in the heart of the SQLite database.

Tailscale’s database architecture

While our clients interact with our control plane as a single public endpoint (controlplane.tailscale.com), internally, our control plane is split into a series of coordination servers (or “shards”). Each tailnet lives on one internal shard at a time, but can migrate seamlessly from one to another. These shards are an internal implementation detail: you don’t know what shard your tailnet is on, and you never need to.

Each shard has an SQLite database that holds all the information about the tailnets on that shard. A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used.

We’ve used SQLite as our primary database since 2022, and we chose it because it's well-known, reliable, and widely used. SQLite is “boring technology”—in a good way. Many companies use SQLite in much larger deployments without issue, and we expected the same stress-free usage.

In our current backup pipeline, we take a complete snapshot of the database every few minutes, then upload the entire SQLite file to an S3 bucket. We’d been running this setup without incident since early 2023.

Fast forward to August last year, when a data pipeline that reads those S3 backups reported an error in one of our databases. We ran SQLite’s PRAGMA integrity_check command against the backup, and found it was indeed corrupted. SQLite corruption is possible, but it’s highly unusual and not something you should encounter in normal operation. We repaired the affected database, and investigated the cause, but to no avail.

When operating at scale, even rare events can occur with some frequency, so we should have been unsurprised when it happened again—and again, and again, and again. In total, we faced 19 separate instances of database corruption over six months before we finally resolved the underlying bug.

When you hear the phrase “database corruption”, it’s natural to worry about data loss. Because our control plane only handles configuration data, these databases contain metadata about your tailnet and devices, but never your private encryption keys or network traffic. In the earliest incidents, the recovery process meant a handful of newly added devices or configuration changes didn’t persist, and a small amount of metadata had to be re-entered.

Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window. In the early incidents, that downtime was over an hour, but we gradually sped up the recovery process over subsequent incidents.

Each tailnet is a mesh network, where devices make peer-to-peer WireGuard® connections to each other. When a device joins the tailnet, it has to get a list of other devices from the control plane before it can establish new connections—so if a device came online during the SQLite downtime, it couldn’t connect. While the database was being repaired, devices already online remained connected to each other, but they couldn’t learn about changes to the network. Those tailnets also temporarily lost access to the web-based admin console and the Tailscale API.

There’s also a broader impact on trust. We post a global incident on our status page even when only a small number of tailnets are affected. Many people saw a status page event for an incident that didn’t affect them. Indeed, the majority of shards and tailnets were never involved in a database corruption incident! Nonetheless, repeated downtime erodes trust, whether or not you’re directly affected.

From the very first instance of corruption, we knew this was a serious threat to our reliability, and we threw a lot of engineering time at the problem—but the fix wasn’t easy.

Trying to find the fault

This bug resisted all our initial attempts to find it.

We looked at recent changes, but there weren’t any that seemed relevant. Nobody had been working on our low-level code that interacts with SQLite, because it had all been written years ago and presented no issues up until that point. We re-reviewed all of that code with a fine-toothed comb to look for previously missed bugs, but we didn’t find anything that would cause the corruption we were seeing.

We looked for common factors between corruption incidents, but we couldn’t find any. It wasn’t tied to a single shard, or customer, or tailnet feature, or time of day, or load level. We were at a loss for what might be triggering the behaviour.

This lack of reliable trigger conditions meant we couldn’t reproduce the bug synthetically. Instead, we had to rely on deploying passive, forensic telemetry in our live environment to catch the corruption red-handed. Gathering live diagnostics for a database issue is the last thing we wanted to do, but we had no choice.

As an additional complication, the corruption didn’t occur on a regular schedule. Sometimes incidents would be hours apart, other times weeks. This made it difficult to predict progress or plan further work, because we were never sure when we’d get our next diagnostic dump. We had a six-week period between October and December when there were no corruption incidents, before they returned as an unwelcome Christmas present.

Because this wouldn’t be a quick or easy fix, we reached out to the SQLite developers for a professional support contract. This was a great decision. It gave us direct access to their deep expertise and experience, and we had many detailed technical conversations about our architecture and our incidents.

Between Tailscale engineering and the SQLite core developers, we mapped out several theories for what might be causing the corruption—including broken POSIX locks on close(), mismanaging memory owned by SQLite, or accidentally using SQLite from multiple threads while disabling thread safety. After every incident, we gathered more data, added more diagnostics, and systematically ruled out these theories. We were gradually converging on the true bug.

The transactions that didn’t bark

While we were investigating the root cause, we still had a live platform to run. We took aggressive steps to automate recovery and minimize downtime:

• Configuring our control plane shards to hard-stop immediately upon encountering corruption
• Deploying an automated backup monitor that continuously ran PRAGMA integrity_check over our backups
• Improving our runbooks and on-call training

These efforts cut our response time to under an hour—and then we discovered an unexpected clue.

We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky).

To do this, we built a transaction logging pipeline. We streamed every SQL statement that modified the database to a separate log file. Because SQLite is a single-writer database with serialisable transactions, our transaction history was completely linear and deterministic. (This wouldn’t be true in a multi-writer database like Postgres or MySQL.) Replaying those transactions against the latest known-good backup should restore the database to its most recent state, safely bypassing the corruption.

This pipeline worked, but then it did something even better: it gave us a clue.

In two incidents, our transaction logs failed to replay cleanly. Upon closer inspection, we discovered that data written and committed by one transaction was inexplicably invisible to later transactions. A write had vanished into thin air without raising an error. That should be impossible!

The writing on the WAL

As these incidents were ongoing, the SQLite developers had been developing a new debugging tool. For a while, we’d suspected that the bug was somewhere in the checkpoint process. They were building a new tool to give better visibility into what was happening during checkpoints.

To understand what this tool found, we need to briefly explain how SQLite checkpoints work.

A SQLite database is made of a series of “pages”, tiny blocks of information. When you update the database, some of those pages need to be replaced with new pages with the updated information.

For better performance and greater concurrency, we run SQLite with Write-Ahead Logging, which means new pages aren't written directly to the database file. Instead, they’re written to the "write-ahead log" or "WAL file".

New pages can't be written to the WAL file indefinitely; at some point they have to be copied back to the main database file. This process is called “checkpointing”.

In most deployments, SQLite itself decides when to do a checkpoint, and the process is invisible to the end user and developer. In our control plane, we take manual control of the checkpoint process so we can run fast and consistent backups. This non-standard approach seemed suspicious as we steadily eliminated potential causes.

One clue was that during corruption incidents, our metrics showed that SQLite would report copying more pages from the WAL file than were actually available. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong.

To understand what was happening during these faulty checkpoints, the SQLite developers created a new debugging tool for the virtual filesystem layer.

SQLite is split into several layers. The top layer is the parser and code generator, which converts SQL statements into SQLite’s internal data structures. These data structures get passed to the pager, which splits them into the individual pages to be written to disk. Actually writing them to disk is handled by the OS interface, or “virtual filesystem”. Currently SQLite has two mainstream virtual filesystem implementations—Unix and Windows.

This approach allows you to replace different layers with different implementations, or wrap an existing layer to get more information. To help diagnose our problem, the SQLite developers created a wrapper around the virtual filesystem that writes additional tracing information and logs about changes to the database. This wrapper is called the tmstmpvfs shim.

We deployed the shim into our live environment, and waited for the next corruption to occur. Fortunately, we didn't have to wait long.

The WAL-Reset bug

After our next corruption incident, the additional logs from the new tmstmpvfs shim allowed the SQLite developers to find and fix the bug: a rare data race in the SQLite source code between a checkpoint and a write transaction.

In particular, if a write occurs at a specific time during a checkpoint, the checkpointing process gets confused—it thinks some of the pages have been copied from the WAL into the main database file, but they haven’t. Those pages never get written to the database file, and that data is permanently lost. The database file becomes corrupt, because other pages which reference those pages—such as an index—are written to the database.

The SQLite developers named this the “WAL-Reset bug”, and they estimate it was present in SQLite for at least 16 years. It could exist that long because it was rare—so rare, the SQLite developers had to add code to deliberately trigger it in their testing environments. Their fix adds an additional check to the checkpointing function which detects when the WAL has been reset by another thread.

They confirmed that this bug caused all of the baffling behaviour we’d seen. It explained the corruption, the transaction logs that wouldn’t apply cleanly, and the inconsistent checkpoint statistics. They also explained why we were more likely to hit the bug than other SQLite users: we take manual control of the checkpointing process, and we checkpoint very aggressively. Even a bug triggered by a rare condition was bound to hit us eventually.

This was an exciting moment. After months of confusion and uncertainty, we finally had a plausible theory for why the corruption was occurring, and a fix we could deploy to prevent it.

The SQLite developers released the fix as SQLite 3.52.0, and we prepared to deploy it as soon as it was available.

Fixed, with a false alarm

We rolled out SQLite 3.52.0 carefully—first to a few canary shards, then, when we saw it running smoothly, we deployed it to the rest of the control plane.

Our backup monitor promptly turned red, and reported corruption in 13 different databases. This was extremely alarming, but we followed our recovery procedures to fix all the supposed corruption, and everything was happy. It turned out these databases had not suffered real corruption, but were subject to a second problem in the version of SQLite.

We shared our errors with the SQLite developers, which uncovered a bug in SQLite related to stale expression indexes. If you create an index on a computed value, and then the computation changes, the index will contain mismatched values, which gets reported as corruption by PRAGMA integrity_check.

In our case, we were storing some high-precision timestamps as text, converting them to a floating-point number in a VIRTUAL generated column, and the SQLite 3.52.0 release that fixed our data race also made an optimisation that subtly changed the rounding behaviour for text-to-floating-point conversions. Our canary shards didn’t have any timestamps that triggered the changed rounding behaviour, so we missed this in our phased rollout.

Because this change caused false corruption warnings, the SQLite developers withdrew the 3.52.0 release and instead published 3.51.3, which only contained a fix for the WAL-Reset bug.

We fixed the issue on our side by reducing the precision of our timestamps to integer seconds; text-to-integer conversions are unambiguous. Meanwhile, the SQLite developers created an automated, self-healing index feature in 3.53.0, which prevents the stale expression index problem.

Party time!

With the fix rolled out to our entire control plane, we were ready to declare victory, but we were still cautious. An absence of corruption incidents doesn’t mean things are fixed—we’d already had one six-week period of deceptive calm.

We wanted positive proof that this data race was actively occurring in our production environment. Now that we understood the cause of the bug—a collision between a write transaction and a WAL-reset—we patched our SQLite driver to log a warning when these two operations overlap. If the warning fired but the database remained uncorrupted, we’d know the fix had saved us from a potential corruption incident.

We deployed the warning, and we waited. And we waited. And we waited. And we waited. As weeks slipped by, we began to wonder why we didn’t see it. Was the warning broken? Was our theory wrong? Was the true bug still lurking in the darkness?

Then, two months later, the alert we were waiting for finally fired:

This alert proved that the precise conditions for the WAL-Reset bug do occur in our production environment, which means it was the likely culprit for our six months of shaky uptime.

Since that weirdly joyous alert fired, we’ve run for another four months without any database incidents, as of this writing. Finally, we could breathe a sigh of relief.

Off the well-trodden path

Nobody wanted us to spend six months looking for bugs in SQLite. This was an immensely frustrating experience for both our customers and staff, and we’re all glad to put this instability behind us.

This investigation is a useful reminder: running boring technology in a non-standard way is a risk. The common paths and standard configurations are incredibly well-tested and reliable. Most people use SQLite in a standard configuration and never face this sort of issue. Everything we were doing was a public, documented, supported configuration—but by taking manual control of the checkpointing process and running at our own aggressive pace, we stepped off the well-trodden operational path.

Resolving these incidents was a massive, cross-functional effort involving dozens of people—including Tailscale's engineering and support teams, and the core maintainers of SQLite. It is to all of their credit that the impact of these incidents was not much worse.

We know that repeated downtime erodes trust, no matter how many people are affected, and we’re grateful to our customers for their patience and support while we chased this down.

Frustrating as this period was, we’re left in a stronger position than we were before. The long-standing bug in SQLite has been patched, and we fixed dozens of other incidental issues that we spotted while looking for it. We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future. Finally, we’ve refined our database backup and recovery processes, and live-tested them over a dozen times.

Hopefully there won’t be another database incident like this—but if there is, we’ll be ready.

Full article content is not available for inline reading.

Read the original article →

AI is enmeshed in the tools and processes organizations use daily. McKinsey estimates that 88% of companies use AI in their existing workflows, but 44% are uneasy about regulations, ethics, or legality.

Design tools are a prime example of this. AI-assisted features are increasingly built into the platforms teams use for wireframing, prototyping, and handoff, often without much visibility into what's happening under the hood. That lack of visibility makes it harder to investigate mistakes, meet compliance requirements, or answer customer questions about how their data was handled.

AI governance addresses these concerns directly. Rather than leaving AI use implicit in your tools and workflows, good governance helps make it documented, traceable, and accountable. It won't eliminate every data or privacy concern, but it gives your organization a clear foundation for addressing them head-on.

What is AI governance?

AI governance is the set of policies, processes, and controls that define how AI can be used in an organization, what data it can touch, and how its behavior is monitored over time. In other words, it's what transforms AI from an unexamined part of your workflow into something documented and accountable.

Good governance frameworks do this by mapping your workflows, marking which steps involve AI, and tracking what happens to data along the way. In practice, this means you can point to any stage of a process and know whether AI was involved, even when it's running quietly in the background of a tool your team uses every day.

How AI governance affects digital product teams

AI governance concerns are especially important for product design and development teams because they touch almost every stage of a product development’s life cycle.

When the boundaries are unclear, digital product teams are left guessing what AI touched, which decisions were influenced, and where the risks are. When governance is in place, they have explicit guardrails for when to use AI and how those decisions will be checked later.

Consider a typical design review flow:

• A researcher uploads user interview notes
• The AI tool summarizes and tags them
• A designer pulls insights into the design tool and creates a prototype with flows
• The PM and legal teams review

Without AI governance, it’s hard to tell what AI saw, accessed, and generated, along with who is ultimately responsible for checking it. It’s then nearly impossible to investigate issues or prove you handled data correctly. As more teams become involved, these capabilities only become more crucial.

With AI governance, you would mark the “summarize and tag” step as an AI system, dictating what data it can access. After each AI run, you would keep enough detail to trace the outputs back and explain the results if something goes wrong. There’s no guessing what AI did and what humans did (an important distinction for audit trails).

When you zoom out from a single design review and look at the full product workflow, AI governance only works if every team knows how they fit into it. Each department would have its own connection to the governance, as well:

• UI and UX Design would know what they can send to AI assistance and where the design history is logged.
• Engineering and DevOps would know where AI services run, how API keys and models are managed, and how logs and model calls are monitored.
• DesignOps would set standards for which AI features are allowed, along with workflow templates, training, and enablement.
• IT and security would manage residency, network boundaries, SSO, and access controls, as well as whether tools are self-hosted or on a vendor cloud.

AI governance acknowledges that even a simple request for AI to “summarize this user interview” can have far-reaching legal, privacy, and product implications for the entire digital product team, not just the person who clicked the button.

Four critical AI governance risks in design tools

AI features can add much-needed functionality and scalability to your existing tech stack, but they also come with additional responsibilities. Without tackling these issues head-on, you put your organization at risk of data exposure, compliance failures, and reputational damage.

These are the pressure points where AI governance most often fails for digital product teams, which is why the following risks matter so much.

Risk 1: Data privacy and where your files are processed

Do you know where your data is being used? If AI features run in a vendor’s cloud, your prompts, files, and logs are pulled into their infrastructure and AI servers, which means they decide how that environment is secured and where your data lives. That’s worrying when you can’t see which regions they use, how long data is retained, or who else inside their ecosystem can access it.

Imagine a team uploading user interview notes that include names, emails, and screenshots of internal tools into an AI “summarize” feature. If that feature runs in a vendor’s cloud under broad data‑use terms or on a consumer‑grade plan, those details could end up stored in regions the team didn’t intend or reused for model training in ways they did not anticipate.

With infrastructure‑level control through self‑hosting or a tightly scoped private cloud, you choose where inference runs, which regions are allowed, and how you monitor and log data flows. That control can make it easier to demonstrate where regulated data was processed, especially when SaaS vendors do not provide sufficient transparency or exportable logs.

Risk 2: IP ownership of AI-generated content

It’s genuinely hard to pin down who owns designs and other IP created with the help of AI tools. Recent U.S. decisions confirm that an AI system itself cannot be treated as a legal “author,” and works generated entirely by AI do not qualify for copyright protection. But even where human authorship exists, ownership of prompts, outputs, and training data often depends on the fine print in your vendor’s terms rather than any right you can take for granted.

That broader gray zone matters because, without clear contractual language, it may be difficult to prove that you (and not your vendors or model providers) own specific AI‑assisted designs. It may also be hard to stop those assets from being reused for training or other purposes you did not intend.

To manage this risk, treat IP terms as a non‑negotiable part of adopting any AI feature. Ask vendors to state, in writing, who owns AI-assisted outputs, how they handle prompts and training data, and whether they train models on your content. Where possible, choose tools or deployment models that give you explicit IP ownership and enough transparency to show which parts of a design were human-created versus generated by AI.

Risk 3: Compliance and regulatory violations

You’re probably already checking for compliance with your vendors, but AI features make this much harder when they touch regulated data. For example, GDPR and CCPA require you to know where personal data is processed, how long it’s retained, and who can access it. Sector rules like HIPAA or financial rules add more layers for any health or payment data.

Each new AI feature a vendor bolts on adds more responsibility to log and track data. A simple “We’re compliant” statement on a marketing page isn’t enough to demonstrate compliance.

Since design files and research artifacts can include personal data, health or financial information, and customer identifiers, they can fall under frameworks like GDPR, CCPA, or HIPAA. When AI vendors cannot clearly explain their model providers or subprocessors, you risk losing visibility into where that regulated data flows or resides.

Risk 4: Lack of transparency and auditability

If you have no reliable way to see when AI was used, on which assets, and with what configuration, you will struggle to credibly audit incidents, answer regulator questions, or enforce internal AI policies. Unfortunately, many of the AI features with the least friction (like a “click here, magic happens” button) may also be the least transparent.

Without seeing when AI was used, on which files, and by whom, it is very hard to credibly claim full transparency and auditability. Design tools that are used with sensitive or regulated data are increasingly expected to offer clear, inspectable paths for security teams to verify where data has traveled.

How to create an AI governance framework for your design and development teams

An AI governance framework should be living, easy to update, and reflective of your actual tech stack usage because your tech stack and regulations will keep changing. If governance is frozen in a PDF, it will quickly become outdated compared to how teams really work. Use these steps to create documentation of real decisions, tools, and workflows that can be adjusted in real time, so policies and workflows stay in sync.

1. Define scope and owners

Go through all your workflows to map where AI already appears in design and development work, from research to handoff. Assign a point person to be accountable for keeping this map current, then involve leaders in DesignOps, security, legal, and IT.

Since each of them owns a different part of the risk, it makes sense to have them partner in the overall plan of how data is used, shared, and stored via AI-enabled systems.

2. Classify design data and set guardrails

Create a simple scheme for data types, such as public, internal, proprietary, or restricted, and communicate how to decide on each designation.

For example:

Public: mockups and website UI

Internal: In-product copy drafts and generic component libraries

Proprietary: Unreleased feature designs and brand systems

Restricted: User interview notes or screenshots containing customer data

Decide which categories must stay within your environment, which can never touch external AI, and which can be freely shared with vetted, trusted vendors.

3. Standardize vendor and feature evaluation

Start by creating a shared checklist for any AI-enabled tool, covering:

• How it uses data for training
• Where processing happens
• Which subprocessors are involved
• What gets logged
• What opt‑out controls exist
• Whether self‑hosting or private deployments are possible

Share this checklist with design leadership, DesignOps, security, legal, and IT so they can review new tools against the same criteria instead of doing one‑off approvals.

When a new AI feature or vendor comes in, the requesting team fills out the checklist, the relevant stakeholders review and either approve, reject, or approve with conditions, and the results are stored somewhere visible (like your internal wiki). This way, future evaluations can build on past decisions.

4. Embed controls and keep improving

As you decide how to handle various AI technologies, turn these decisions into defaults for new products. For example, use role‑based access so only certain roles can run AI on restricted research data, and introduce AI‑safe templates (preconfigured project or file templates that block restricted fields from being sent to AI features by default).

You can also wire these rules into your CI/CD or release process, so a design or feature that uses AI in a new way triggers an automatic review rather than going straight to production.

The goal is to make the safest option the one designers and developers actually use when they do their normal work, not an extra step they have to remember. Over time, review logs, incident reports, and new industry guidance to adjust these controls so they reflect how your teams use AI today, not how you imagined it a year ago.

Penpot's approach to AI governance and transparency

Most design tools give you AI features as part of a managed SaaS stack. Click the AI feature button, and your data goes to a vendor-chosen model in their cloud, where you may have limited ability to configure, monitor, or inspect it.

Penpot, on the other hand, treats AI as something you connect on your own terms and it is an opt-in capability within the platform, meaning that you choose to enable or disable it as you please. AI is connected through open protocols. This means you can plug in vetted and approved AI agents instead of accepting a single, fixed model behind a proprietary integration.

Penpot’s file formats follow open standards and can be downloaded, inspected, and integrated into existing workflows. This way, security and platform teams can bring design artifacts into their own logging, version control, and review processes, no matter if the workflows use AI or not.

The Penpot MCP server extends this into the AI layer. MCP is an open standard that lets AI tools communicate with other software in a structured, inspectable way — think of it as a controlled handshake between your design environment and any AI agent your organization has approved. Deployed inside a company's own environment, it can be restricted to approved models, with the option to swap or self‑host LLMs without rewriting design workflows or handing control to a single vendor.

In practice, these Penpot features make it easier to enforce code review and internal audits while still giving engineering and product teams room to experiment with AI. AI behavior can be traced and verified against data on internal servers, which makes it easier to align your design workflows with the governance practices described above instead of fighting against opaque or hard‑to‑audit integrations.

Design and develop responsibly with Penpot

With a better understanding of AI governance, you won’t see any software platform as “just a tool” but as an active participant in your data-handling and decision-making processes. Design platforms are no different and should be treated as part of your overall governance framework.

Penpot’s open-source, self-hostable platform lets you run AI processes in accordance with your internal AI governance terms, not the vendor's. It’s ideal for avoiding “black box” situations where AI use isn’t obvious or governable. Instead, you can design within your own infrastructure, align with existing AI policies, and give your IT and compliance teams the visibility they expect for any critical system.

If you’re ready to treat design as part of your governed digital product infrastructure (and not an exception), Penpot gives you a platform that can grow with your AI policies. Talk to Penpot about enterprise options today and start designing and developing responsibly from day one.

FAQs

Who should own AI governance for design tools in our organization?

While there should be a clearly accountable leader for oversight and planning of AI governance (usually a senior product or technology leader), it should be executed across teams. In practice, design leadership, DesignOps, security, and legal should share responsibility for day-to-day governance, with clear ownership over policies, vendor choices, and how AI is used in design workflows.

Can we safely use cloud AI if we’re in a regulated industry?

Yes, but only with strong guardrails around data classification, residency, and vendor controls. At a minimum, PII and protected data should be stored on-site when possible, with lower-risk assets stored only in properly vetted cloud solutions. You may also choose self‑hosted or hybrid options, which give you more control over how you use and store data.

What’s the first step if we already have AI features turned on everywhere?

Start by inventorying where you use AI, what data it touches, and which vendors are involved. Then classify data flowing into the AI features and flag highly sensitive or regulated data. From there, it’s easier to prioritize fixes, such as turning off risky features or folding in tools with unified AI/data governance frameworks.

OpenAI previewed Ultrafast, a GPT-5.6 Sol mode capable of generating up to 750 output tokens per second and operating at up to 14x standard processing speed. The approach aimed to deliver real-time performance without switching to a smaller model.

Google is rolling out Gemini 3.7 Flash, a new version of its workhorse AI model. The release comes just three weeks after the release of Gemini 3.6 Flash, an unusually short turnaround attributed to developer feedback and algorithmic improvements. API prices have been temporarily cut in half. Gemini 3.7 Flash will cost $0.75 per million input tokens and $3.75 per million output tokens until the end of the year.

Subagents on Subagents: How Many Layers Deep Is Too Many?

Agents are starting to spawn agents. Give an agent a sufficiently large job and one increasingly common strategy is delegation. The parent agent decides that part of the work should be handled somewhere else, spins up a subagent with its own context and instructions, waits for the result, and continues. That subagent may eventually do the same thing.

There are good reasons to work this way. A subagent can have a narrower job, a cleaner context window, different tools, or instructions specialized for the task it has been given. It can also do a large amount of exploratory work without dumping all of that context back into the parent. Once agents can delegate recursively, though, an obvious question appears: how many layers deep should we let them go?

I don’t think there is a magic number. Two layers isn’t inherently safe and five layers isn’t inherently reckless. The more useful question is what happens when something goes wrong at each layer, and how much of the graph now depends on the mistake.

Subagents turn agent execution into a graph

You can think about a parent spawning subagents as a tree. The parent delegates to several workers, some workers delegate again, and results eventually flow back toward the top. But real systems quickly become more complicated than trees.

One agent’s research becomes another agent’s input. Several branches get joined together. A reviewer evaluates the output of a worker. A planner produces a plan that several executors follow. Some branches run in parallel while others cannot begin until their dependencies finish. At that point, you have a graph.

That is roughly what people have started calling graph engineering: making the nodes, dependencies, routing, and state transitions between pieces of agentic work explicit. This framing changes the way we think about subagents because instead of asking only, “How many agents are running?” or “How deeply are they nested?”, we can ask what each node produces, which other nodes consume it, and what portion of the eventual result depends on it.

That is a much more useful way to reason about reliability because not every node carries the same amount of risk. Some nodes sit at the edge of the graph and produce a small, isolated contribution. Others sit high in the graph and establish the premises that many later nodes will use.

Errors have a blast radius

Imagine an agent producing a competitive analysis. The first agent decides which competitors matter. Three subagents research those companies. Their outputs feed another agent that compares the products. That comparison goes to another agent that identifies strategic threats, and a final agent writes recommendations.

If the final writing agent phrases one recommendation poorly, you have a fairly localized problem. Most of the underlying work is still intact. If the first agent picks the wrong competitors, everything downstream can be perfectly executed and the final answer can still be wrong.

The important distinction is where the error entered the graph and how much work depends on it. An upstream node does not merely contribute one incorrect output. It can establish the premises under which many downstream agents operate, and those agents may then reinforce the mistake because, from their perspective, the bad output is simply part of their input.

A mistake at a leaf may damage one leaf. A mistake near the root can poison an entire branch, and the problem gets worse with fan-out. Suppose one planning agent produces a decomposition that feeds ten workers. An error in that plan now has ten opportunities to propagate, and those workers may each produce outputs that feed additional nodes.

The number of layers matters, but the downstream influence of each layer matters more. A deeply nested agent working on an isolated task may have very little impact on the final result, while a shallow agent making an early routing or planning decision may determine everything that follows.

Every handoff can harden a mistake

There is another problem with deeply nested subagents: information gets transformed at every boundary. A subagent might inspect twenty documents and return a five-paragraph summary. Its parent uses that summary to create a plan. Another agent receives part of the plan and turns it into an analysis, while a final agent synthesizes several analyses into a recommendation.

Each step may be individually reasonable, but each one is operating on the artifact produced by the previous step rather than necessarily on the original evidence. That means a small mistake can gradually become an assumption.

Recursive delegation makes this particularly easy to miss because the parent often does not see everything its descendants saw. Context isolation is part of the benefit of subagents, but it also creates distance between the final decision and the evidence that originally supported it.

Graph engineering should be dependency engineering

This is why I think one of the most important parts of graph engineering will be deciding what is allowed to depend on what. If an agent produces something with enormous downstream influence, that node deserves more scrutiny than a node producing an isolated piece of the final response.

Maybe it should have a verifier. Maybe its output needs to be structured. Maybe the original evidence should travel alongside its conclusion. Maybe several agents should independently produce the result before the graph moves forward. In some cases, a human may need to approve it before the artifact is allowed to fan out into the rest of the graph.

The graph makes these decisions visible. Instead of one agent recursively spawning workers and passing summaries around however it sees fit, you can explicitly model something like evidence → analysis → plan → execution → synthesis and decide which artifacts cross each boundary.

You can also decide where branches fan out, where they join, where provenance needs to be preserved, and where checks should happen before a high-impact artifact becomes the premise for another five nodes. Graph engineering gives us more than a way to coordinate agents. It gives us a way to control the propagation of uncertainty through the system.

So how deep is too deep?

I don’t think the answer is three agents or four layers or some other universal limit. A deep graph where each node has a narrow responsibility, strong inputs, explicit outputs, and limited downstream influence may be quite reliable. A shallow graph can be extremely fragile if one early agent makes a broad judgment that every other agent blindly accepts.

The metric I care about is closer to blast radius. For any agent-generated artifact, ask what else becomes wrong if that artifact is wrong. If the answer is one small part of the final output, you can probably tolerate a fairly agentic node. If the answer is every remaining step in the workflow, that node deserves a very different engineering standard.

As agents get better at spawning subagents, it will become increasingly easy to build impressive towers of delegation. Agent A asks Agent B, which asks Agents C and D, which each spin up their own workers, and eventually a polished result bubbles back to the surface. The engineering challenge is not how deep we can make those graphs, but whether we understand which nodes have enough downstream influence to bring the rest of the graph down with them.

OCR 4.1

Our latest OCR service powering our Document AI stack, with native paragraph-level bounding box extraction, structural block labels, and block-level confidence scores.

Features

• BBox Extraction /v1/ocr
• OCR /v1/ocr
• Annotations - Structured /v1/ocr
• Batching /v1/batch

Agents are only as capable as the environments they run in. Fast, reliable development environments allow agents to take ambitious, long-running tasks from start to finish.

Until now, every cloud session began with extensive setup: boot a machine, clone the repositories, and run the install script. On a large, complex repo, this just-in-time boot could take several minutes before the agent started executing.

Today we're introducing builds: ready-to-use copies of your development environment that Cursor prepares continuously in the background, at no additional cost. When you kick off an agent, it starts in a ready environment so you get a response up to 3x faster.

And when a bad commit or dependency update breaks your environment, agents keep using the last successful build. Your work continues uninterrupted while you debug in the background.

Faster boot times

A build is a copy of your development environment that Cursor prepares in the background. By default, Cursor runs a new build every hour. Instead of setting up the environment from scratch each session, agents boot into a ready version: repos cloned, dependencies installed, and the install script fully executed.

When a build succeeds, it becomes the environment that future agents start from. Cursor keeps warm copies ready with new agents forking a live machine instead of restoring one from disk. This allows sessions to start almost instantly instead of keeping the next agent waiting.

With environment setup already complete, agents get to real work much faster. At Cursor, our internal environments now boot 10x faster and time to first token is 3x faster.

Our customers are seeing the same:

We kick off more than 2,000 automated agent runs a week without any manual prompting. With builds, every run boots quickly into an environment we know is good and broken builds never take down the agent fleet. Our largest, most complex repos now start in just a few seconds.

That combination of speed and reliability is what lets us hand more of our engineering work to agents that run entirely on their own.

More resilient agents

Cloud agents always start from the latest successful build. If a dependency bump breaks your install script or a Docker build fails, that build never becomes active and you're notified of the issue. New and existing sessions keep running safely while you debug the environment in the background, either manually or with an agent.

Better observability and easier debugging

You can now inspect each build directly in your Cloud Agents dashboard, with:

• A Builds tab for each environment, with type, status, start time, and versioning
• Build details with logs and the exact commit SHAs the build captured
• A record that ties each agent run to exactly the build it used
• A configurable threshold for a build's git state so agents don't start too far behind your default branch

Agents can also inspect and manage builds using the built-in Cursor Cloud MCP.

Get started with builds today

For an existing environment, open it in the Cloud Agents dashboard, go to the Builds tab, and click Enable Builds. Or click Run setup agent first to test the migration and review any proposed config changes.

Because builds work by using filesystem snapshots, there are a few things worth checking at this stage:

• Update your install command to cover anything that can be prepared ahead of time, like dependencies
• If install needs credentials for private registries, use team or environment secrets. User secrets stay out of builds and are added when the agent starts.
• The start command still runs when you first prompt an agent. Use it for services that must be fresh when the session begins, like bringing up Docker containers or other long-running processes

On August 17th, all new and existing environments will use builds by default, with no additional cost to you.

Learn more in our docs.

Across the AI industry, users are becoming more conscious of just how expensive their deployments can be —and feeling a new urgency to cut costs. But while open source models offer significantly lower per-token costs, it can be difficult to find the right model for a given job.

On Thursday, Writer, which offers AI tools and agents for marketers, launched a new flagship model called Palmyra X6, aimed at solving that problem for its users. Built as a post-training variation on Z.ai’s open source model GLM-5.2, Writer says the new system should provide deployment-ready capabilities at a much lower price. The company estimates the new model, combined with changes to the companies harness infrastructure, will cut costs for its customers by as much as 50% for basic tasks.

Together with the new model, the company also released significant upgrades to its standard agentic harness. Both features will be available to Writer clients starting Thursday.

“I think the enterprise is absolutely sick of chasing the next benchmark,” CEO May Habib told TechCrunch. “They want flattening cost, and it seems like nobody can deliver that.”

The new approach puts particular emphasis on complex, multi-step tasks, executed faster and with fewer tokens. And Writer sees harness optimization as a crucial lever toward making that happen.

A recent paper from Writer researchers lends credence to this approach, testing small changes in harness efficiency across multiple different models. The research found that, in many cases, changes in the harness were a more reliable way to reduce costs than model choice, with costs falling an average of 40% across their testing.

“The harness is the one component whose efficiency multiplies across every model an organization runs—present and future,” the researchers wrote.

For Writer’s clients, the experience is still model-agnostic: Palmyra X6 will sit alongside other Writer models or outside models imported through Azure or Amazon Bedrock. But Habib also sees the push to cut costs as driving a broader distrust toward major AI labs, which have a financial incentive to drive up token use.

“The cost explosion here is just unprecedented for customers, and so is the degree to which CIOs are giving up on the labs,” Habib told TechCrunch, adding that the AI labs “don’t deeply understand right how to help an enterprise get benefit from AI.”

Google appears to be building a dedicated agents section into AI Studio, adding a management surface for the managed agents it launched at I/O in May. The work in progress introduces a workbench tab listing available agents, with a project switcher that scopes each list to a specific Google Cloud project. That detail implies agent definitions are meant to sit against Cloud projects and billing rather than in a separate consumer-grade sandbox.

GOOGLE 🔥: AI Studio is about to get a dedicated Agents tab for managing Cloud Agents! Users will be able to browse, create, and configure managed agents for different GCP projects. Artifact management will also be available there, along with an editor. Soon 👀

From the same tab, a creation flow is present, alongside a per-agent file directory in its own tab where files can be added and edited directly in the browser, and a configuration screen covering name, description, a custom prompt, and further settings. That maps closely to how managed agents are already defined through the API, where an agent is essentially a bundle of instructions, skills, and mounted sources. The unreleased interface would give that bundle a graphical editor instead of a JSON payload or a terminal command.

The background model is currently locked to the Antigravity harness, the agent runtime Google shipped at I/O and which now runs on Gemini 3.6 Flash. The selector around it is present but offers nothing else, which usually points to alternatives being wired in later, plausibly other harnesses or model tiers as they arrive.

AI Studio already carries an agents toggle inside its playground with pre-configured templates, so this reads less like a first step and more like the layer above it, where prototyping turns into hosting and lifecycle management. Anthropic reached that point in April, when Claude Managed Agents entered public beta with configuration, sandboxes, and session tracing exposed through its console, and Google now looks set to close the same loop.

Timing remains unclear, and the model roadmap does not help. Gemini 3.5 Pro missed its June target, and an analyst note claims it was quietly shelved in favor of Gemini 4, although Google has not confirmed this and still lists the model as coming. The agent stack leans on Flash-class models regardless, so it is not obviously waiting on a Pro release.

X is significantly expanding its open source codebase, which includes the app’s “For You” algorithm and its core ranking engine, and adding a feature that will let users see if their account or posts have been impacted by any of its ranking systems, the social network said on Thursday.

The company is making the source code for the “For You” timeline, the default feed you see when you open the app, available on GitHub under the Apache v2 license. It’s also expanding its previous efforts to open source parts of its codebase to add more detail, including the model configuration, filter, and core ranking system details. That means it includes the parameters used to weight different signals — key to understanding which posts are actually displayed. This also makes the codebase roughly 10 to 15 times larger than it was before.

“You’ll get the core ranking code that pulls posts and ranks them for any given user and assembles the feed,” X’s VP of Product Keith Coleman told TechCrunch in an interview ahead of the announcement. “You can see the systems that filter out potentially problematic, rule-violating content…And some of those systems, like the ranker and the score, you can even run yourself outside the company.”

“This is the kind of thing that I think people will be fairly shocked that we are releasing,” he added.

In addition to the repository, X is providing tools that will let users see for themselves if and how X’s ranking systems have impacted their account or posts. A new transparency tool is rolling out to an “Under the Hood” page in the app’s settings, which will let users who have posted 10 or more times over the past month download their aggregate stats as a JSON file. The file will show if any labels have been applied to their account or posts over the past calendar month.

Non-technical users can take advantage of this information by dropping it into an LLM of their choice, pointing the AI at X’s GitHub repo, and asking for an interpretation.

The company notes this tool will initially be available to a test group of accounts at least a year old as a pilot, before rolling out more broadly.

Ahead of launch, the company previewed its open source codebase to external researchers familiar with recommendation systems, who were able to train and run X’s Phoenix scoring system using the open source code. That was a major milestone for X’s transparency efforts, Coleman says.

From the GitHub repository, developers will be able to submit updates, known as pull requests, which X engineers will consider incorporating into its algorithm. While not all additions will make the cut, Coleman is enthusiastic about the idea.

“That would be amazing to have people submitting code that improves the algorithm…I mean, how cool would it be for the X algorithm to be not just visible to the public, but also, like, by the public?,” he said.

However, a few systems are not included in this release, like those that use Grok to predict whether a post could be violating a rule. This is meant to protect X from bad actors who could use this information to work around the company’s rules to flood the network with spam.

The changes are meant to address continuing concerns about how X’s algorithm influences politics, elections, the spread of misinformation, and more.

The platform has been home to political figures and high-profile individuals for years, and is now owned by a trillionaire who helped President Trump get elected. But Twitter had been under attack over its lack of transparency before it was bought by Musk, too. In earlier years, Republicans in Congress alleged that the California-headquartered social network leaned too far left, and claimed the network had “shadowbanned” their posts — meaning their posts were made invisible or undiscoverable without their knowledge. Twitter consistently denied this was the case.

“Our dream is that anyone in the public can be able to assess how posts are distributed on the platform, vet that it’s a level playing field, and, if they think it’s not, critique it so we can keep improving it and addressing it,” Coleman said.

“That’s the whole goal of this: [the code] can be audited; it can be critiqued. We’re going to listen, and we want to make the system one that people like and trust, and feel is fair,” he said.

Though X may be pushing itself to be more transparent around its code and other features, like its crowdsourced fact-checking system, Community Notes, it arguably became less transparent overall under Musk, after becoming a private company again. As it was no longer required to report to the SEC, X was not as forthcoming in other areas, like its user metrics, growth, revenue, or government takedown requests, which are now less frequently reported.

Now merged with SpaceX, its monthly active users are once again public.

Correction: Included a clarification that X merged with SpaceX and now reports public numbers around user metrics. Coleman also said during the interview that external researchers were about to get “the whole score up and running themselves,” but the company clarified that they trained and ran the Phoenix scoring system themselves, but did not get the per-post score.

Everything is a plugin

DeepSeek Harness is now in developer preview for agent harness developers worldwide — source code included.

Every capability is a plugin that can be swapped or recomposed: models, tools, skills, sessions, sandboxes, storage, loops, scheduling, and the UI.

Harness keeps agents working in real-world environments

The model is the soul of an agent.

A harness lets an agent understand its environment, use tools, and keep working in real-world settings.

Cordis kernel

The Cordis kernel manages plugin mounting, unmounting, and dependencies. Agent capabilities live in the plugins.

Capabilities as plugins

Plugins provide every agent capability, including models, tools, skills, sessions, sandboxes, storage, loops, scheduling, and the UI. Cordis services and events let the plugins work together.

Compose with configuration

Developers can select, swap, or extend any capability in configuration without changing the DeepSeek Harness source code.

Everything is a plugin. Every run is traceable.

Everything is a plugin

DeepSeek Harness is built on Cordis's plugin system. Plugins provide every agent capability, including models, tools, skills, sessions, sandboxes, storage, loops, scheduling, and the UI. Cordis services and events let the plugins work together. Developers can select, swap, or extend any capability in configuration without changing the DeepSeek Harness source code.

Every run is traceable

Everything the model sees is recorded in an append-only session log: system prompts, reasoning, tool calls and results, subagent scheduling, and every context injection. In the Trajectory view, you can inspect these records by source. Resume, fork, search, and replay all operate on the same event stream.

Multiple runtime modes

Standard mode includes the full toolset. Code mode uses model-generated code to orchestrate multiple rounds of tool calls. Minimal mode keeps only a shell tool and a file editor for benchmarking models in a minimal environment. Creator mode lets you inspect the current runtime, test Cordis plugins in memory, and combine them into new modes.

• Standard mode: Full coding agent with file editing, shell, file and web search, skills, planning, goals, subagents, and workflows.
• Code mode: All Standard mode capabilities, with tools exposed through the Code Mode SDK so the model can combine multi-step operations in one TypeScript program.
• Minimal mode: Two-tool coding agent with persistent bash and str_replace_editor.
• Creator mode: Built for creating custom agent presets, with all Standard mode capabilities plus runtime inspection, plugin experiments, and preset-authoring guidance.

Try it now or install from source

Quick start

Install Node.js, then launch the Web UI with npx.

$ npx @deepseek-ai/dsh web

Install from source

Clone the full source and follow the setup instructions in the repository.

$ git clone https://github.com/deepseek-ai/deepseek-harness

Join the DSH plugin ecosystem

DeepSeek Harness remains in developer preview and is still being tested by developers building agent harnesses. Its core plugins and APIs will continue to evolve. We look forward to exploring the limits of intelligence with developers worldwide using open-source infrastructure that is reusable and composable.

eve Software Factory Template

Meet Foreman, an eve software factory that puts AI agents on every stage of the development loop and keeps people on the judgment calls.

Foreman takes tasks from GitHub and Linear, moves each one through four stations, and delivers a reviewed draft pull request on your repository. You review, mark ready, and merge.

How it works

• Classifier triages the task: type, priority, complexity, actionable or not. When the task isn't actionable, Foreman asks the requester instead of building the wrong thing.
• Analyst turns it into a plan with acceptance criteria, working from a live checkout of your repository.
• Implementer executes the plan in its own sandbox, verifies with your repo's own checks, and pushes a branch.
• Reviewer independently judges everything against the real diff, with evidence for each verdict.

Each station is its own agent with its own instructions, sandbox, and tools. The Reviewer sees only the pushed branch, never the Implementer's reasoning. Between runs, Foreman keeps a factory brain: notes about your repository that every run starts from. See the pipeline and factory memory for the full picture.

How work arrives

• Label an issue factory. The pipeline runs on its own, posts progress as stations complete, and ends with a draft PR linked to the issue.
• @mention it on an issue or PR. Mentions from repo owners, members, and collaborators start an interactive session.
• Delegate in Linear. Linear Agent Sessions run the same pipeline and report progress back in Linear.
• The dev TUI. Hand it a task locally. Changes to GitHub wait for your approval.
• Red CI on a factory PR. Foreman diagnoses the failure and pushes a fix to its own branches, never yours.
• Someone opens a pull request. Foreman posts one orienting comment for reviewers: a summary, not a review.

Deploy

The Vercel deploy flow sets up everything: the GitHub connector, Linear connector, Vercel Blob store, and a prompt for the FACTORY_REPO and FACTORY_LABEL environment variables.

Configuration (see .env.example):

Local development

Link the project you deployed (or a fresh one), pull its environment, and start the TUI:

vercel link
vercel env pull
pnpm dev

Hand the agent a task ("users report the password reset email arrives twice, fix it") and watch the four stations fire in order, ending in a draft PR on FACTORY_REPO. Local runs are treated as untrusted, so changes to GitHub wait for your approval in the TUI.

Resources

• Foreman Docs
• Vercel Connect
• eve Documentation
• GitHub Tools eve Extension

The DeepSeek Thesis

What does Liang Wenfeng want? The CEO of DeepSeek is now richer than either Dario Amodei or Sam Altman. Backed by his careful watch and the immense wealth of his hedge fund, a little-known group of researchers in Hangzhou has managed to consistently command the world’s attention — and they keep giving their work away for free. The motivations behind Liang’s theory of action have puzzled China+AI watchers since the beginning.

In late July, leaked minutes from a four-hour meeting between Liang and investors circulated around the internet. Like everyone else, we’ve been pouring over the minutes to understand Liang the CEO, DeepSeek the company, and Liang the man. Liang comes off as a genuinely unusual character among China’s tech elite: a person who sincerely holds a specific and highly personalized worldview, if not an entire ideology. Whereas others in his walk of life are highly attuned to commercial trends and political winds, he positions himself as squarely focused on realizing his vision for technology — and it’s hard to doubt his conviction when he has invested so much of his literal worth into this quest.

His is a confidently singular thesis: that there is an inevitable causal relationship between automated learning and generalized intelligence, and that the pursuit of such advanced machine intelligence is the only problem worth solving right now. Being Chinese means working within the constraints of current hardware limitations, but DeepSeek frames their own ambitions on a much farther horizon. Whether or not they can, of course, might not be up to them.

At ChinaTalk, we’ve been covering the DeepSeek story since before R1. The following are our thoughts and observations after reading the leaked minutes. We get into:

• How Chinese open models work as a business model;
• Why, according to Liang, learning is the path to AGI;
• What China’s political economy looks like from the vantage point of Hangzhou;
• How DeepSeek is managed differently;
• And what the Liang Wenfeng-Demis Hassabis comparison reveals and misses.

The business logic of open source, revealed

If the weights are open, how exactly does DeepSeek make money off of its models? Liang told investors that the lab’s biggest source of revenue comes from business customers. When R1 came out in 2025, there was a wave of Chinese businesses and government entities connecting DeepSeek to their internal systems. While much of that was a fad, some of those corporate customers apparently stuck. Since all their models since R1 have been released openly under the MIT license, DeepSeek does not earn revenue from on-premise deployments by corporate entities, so those actual corporate customers must be paying for API tokens.

The DeepSeek team published an analysis in February 2025 showing that on an average day, R1’s API earned the lab US$562,027, at a 545% profit margin. Fast forward to now, and the hypothetical figure Liang apparently gave in the investor meeting for DeepSeek’s enterprise-end revenue this year is in the hundreds of millions of US dollars. With enough growth in this area, he figured, DeepSeek could even turn a net profit soon, paving the way for a successful IPO. Failing that, and API sales to individual consumers have room for growth as well.

That being said, Liang has little interest in consumers. He confesses that at one point last year, the lab even considered sunsetting its consumer products given how little effort went into maintaining them. But DeepSeek’s chatbot and individual API users were incredibly loyal, apparently, so the team ultimately decided to keep the lights on.

From Learning to AGI

Minor problems like keeping users around do not concern Liang. As long as the upper limits of machine intelligence are still out of view, there is more (technological, societal, and literal business) value in pursuing that than there is in building business models based on what is available today. More capable models easily pull the rug out from under competitors, especially when the latter have fallen into path dependencies based on older technology. Once truly generalized and convincingly superior intelligence arrives, creating to-C and to-B products will be trivial. Before then, the vast majority of DeepSeek’s resources will go into probing the upper limits of intelligence — and nothing else.

Liang describes this work as the “main quest” of AGI, which, to him, covers research in the realm of general-purpose intelligence, agents, chain-of-thought, etc. He is careful not to commit to specific subfields or theories, but does explicitly exclude some trajectories. World models, for example, are to him “irrelevant” for the pursuit of higher levels of intelligence, even though he thinks embodied AI is largely inevitable. He argues that once AI research is based on “self-iteration,” these systems will then solve for embodied intelligence in the physical world. Before then, however, he believes the most important problem right now is learning.

AI training today relies heavily on high-quality, labelled data being fed into models — not autonomous and truly continuous learning. Liang cautions investors not to think of “learning” as a sub-technology the way they do with “agents,” but instead to think of it as a problem to be solved. (Indeed, one takeaway from reading Liang in his own words is that despite being a finance veteran, he seems to see the world as an unfolding kaleidoscope of puzzles the way an academic researcher does.) The path to AGI, in his eyes, runs through mechanisms that allow models to keep acquiring knowledge. He is quite honest that he and his team don’t know what that path looks like yet, but he’s asking investors to bet on their clarity of thought.

China, According to DeepSeek

Throughout the conversation, Liang seems to implicitly assume that future AI systems — and eventually AGI — will be open. While America will maintain its capability advantage for now, China, in his eyes, will play the role of token factory at global scale, pushing the price of intelligence down as it did for countless other industries during its manufacturing boom.

Liang appears to have somewhat settled into the national-champion role. Phrases like “historic mission” slip through when he discusses how the hardware chokehold China faces will be eroded and finally dismantled, as if the forces of history make that inevitable. Liang expects Nvidia’s CUDA moat to erode, and — with limited specifics — expressed cautious optimism about training on Huawei chips. In fact, he presents DeepSeek’s relationship with Huawei as cordial and regards working with domestic GPUs as an inevitability. Huawei apparently allocated 16,000 Ascend 950 GPUs to DeepSeek, which is fewer than what it sold to “bigger internet companies” (probably ByteDance, Alibaba, and Tencent). Revealingly, Liang remarked that these competitors need Huawei chips more than DeepSeek does because DeepSeek can “acquire some noncompliant chips,” and that DeepSeek’s main rationale for spending on Huawei is to support a domestic hardware ecosystem.

Liang does not want to be an out-and-proud part of China, Inc. He knows, however, that Chinese domestic integration is inevitable, and sees value in aligning with Beijing on tech sovereignty. The trouble is convincing investors that they should really feel so optimistic about model training on domestic GPUs — it’s the topic that collected the most number of questions during the Q&A.

A New Theory of Management

Liang clearly takes great pride in the company he’s built. To investors, he pitched a story of a team that commits to its work not only because of the substance or the mission, but also because of a culture that’s rare in China’s tech industry. DeepSeek researchers, apparently, rarely work overtime. Management strives to ensure that “mandatory” tasks — those based on central goals decided collectively — do not take up more than half of employees’ time at work. The other half is entirely self-directed, and employees are supported in pursuing whatever research projects interest them as long as there is compute available.

Of course, Liang is not doing this entirely out of the goodness of his heart. In the meeting, he argued that a relaxed environment is a necessity for research. Good research ideas come from exploring idle curiosity, and there would be no room for such curiosity if researchers were constantly preoccupied with pressure. To maintain such an environment, Liang focuses on “saying no” to projects that do not relate to their main mission; he told investors that this is why they intentionally did not make improvements to many existing products.

Will a new theory of technology and productivity finally defeat the dreaded 996? Rather than jumping onto every vertical and rushing to be first, as is the norm across much of China’s internet industry, Liang is betting that focus and discipline will win the day in the AI chapter. Unlikely in Silicon Valley, where the AI gold rush is unleashing extremely workaholic founders and pushing even longer hours onto employees, Liang doesn’t sound like he is racing against time. In his eyes, it seems, the horizons for machine intelligence are still so wide that there is much more work to be done before anyone should panic. It’s in the interest of the follower, after all, to hope that the race remains far from over. But maybe Liang is also less worried about the consequences of such a race: throughout the four-hour conversation, the dangers of potential AGI never come up. The pursuit of it seems inherently worthy to Liang.

Is Liang Wenfeng the Demis of China?

Liang is inspired by the history of Bell Labs but does not see DeepSeek as an exact replica. When investors asked about the tension between DeepSeek’s research idealism and the urgency of constructing a sustainable business, Liang argued that missions do not cost good companies revenue. Rather, worthy missions make for even stronger paths to commercialization. In other words, not only is research possible inside for-profit companies, but sometimes better outcomes happen when there is capital involved.

There’s another AI CEO who arrived at a similar conclusion — Demis Hassabis. Sebastian Mallaby’s biography of Hassabis, The Infinity Machine, documents how DeepMind became a Google subsidiary. Former Google CEO Larry Page presented Hassabis, who resented business development work, two future scenarios: either spend the prime of his career gathering Google-level resources for the real work DeepMind hopes to accomplish, or do that work right now on Google’s back. To Hassabis, the choice was obvious.

But Demis’s bet did not turn out the way he wanted. Google DeepMind became much more entwined in Google’s corporate labyrinth than he expected. He complained of becoming just another employee from the perspective of Mountain View, as he was no longer the sole deciding vote steering what DeepMind produced. Unlike Liang, who regards harnessing commercial incentives for “pure” AGI development as entirely possible, Mallaby describes Hassabis as disappointed in what transpired, even though he remains convinced of the necessity of keeping powerful technology in the hands of those who understand it most.

On August 5th, all this tension ripped wide open. Hassabis has resigned from Google and will pivot to working on AI-assisted drug discovery at Isomorphic Labs, as well as researching “societal impacts of AGI” per a Google spokesperson. The siren call of the mission pulled him away from a disheartening situation — but also away from his life’s work at DeepMind.

Hassabis should certainly not be jealous of Liang’s personal predicament. While Hassabis was merely constrained by the complex politics of a world-famous company, Liang’s firm now symbolizes the US-China AI race. To shield his company from the eye of the storm, Liang takes pains to shape the domestic narrative around DeepSeek. There is to be no heroism or glorification of specific researchers; he repeatedly emphasized that DeepSeek is made up of “regular people” whose secret sauce is teamwork. He mentions at one point that the government “won’t give [them] a cent” should the company fail. This shouldn’t be taken literally given the asymmetric relationship between the Communist Party’s coercive potential and a small (if well-funded) lab; it was probably to assure his audience of investors that sound financial management won’t take a backseat as DeepSeek pours more of itself into AGI.

After leaks of this very meeting spiraled across the internet, Liang was reportedly so furious that he paused a new funding round and pushed back IPO plans once more. More than anything, he demands complete trust that does not dilute his exact vision — and whoever leaked this clearly does not fully trust the plan. The question is: who, among the audience that day, had reasons to want to poke holes in Liang’s vision?

How it works

The runtime map of Foreman, covering what runs where, how the filesystem defines the agent, and the three boundaries that shape the design.

Foreman is one eve agent with five subagents, three inbound channels, and two durable stores. Everything it can do is declared as a file under agent/, and eve discovers the surface from the filesystem at build time.

What runs where

The agent directory

In eve, identity comes from the filesystem rather than a name field. The file agent/tools/read_factory_brain.ts is the tool read_factory_brain, and the directory agent/subagents/classifier/ lowers into the tool classifier. Renaming something means moving the file.

Three boundaries

Most of Foreman's behavior falls out of three structural decisions. They hold because of how the agent is wired, not because a prompt asks nicely.

Stations inherit nothing

Every declared subagent runs in a fresh child session with none of the root's instructions, skills, connections, tools, or sandbox. That has two consequences worth internalizing.

The orchestrator must pack everything a station needs into the delegation message: the work item verbatim, plus every prior stage's output. Stations never see the conversation history, and they cannot read the factory brain, so any repository fact that matters has to be woven into the message.

Long documents are the one exception, and they travel by reference rather than inline. The station that produced one saves it as a handoff artifact and returns the id, the orchestrator relays that id, and the receiving station opens the document itself.

The upside is that each station's blast radius is readable at a glance. Whatever sits in agent/subagents/<id>/ is the complete list of what that station can do.

Trust is stamped at dispatch

Every channel decides who the caller is from the signed webhook, before the model reads anything, and writes that decision into session auth. Nothing downstream re-derives trust from model-readable content, because model-readable content is exactly what an attacker controls.

agent/lib/trust.ts is the single authority that reads those stamps. New capabilities gate on its predicates rather than inventing their own caller check. The trust model covers the caller classes and policies.

Each station works in its own sandbox

Every eve agent gets a sandbox whether or not it asks for one, so all five subagents run in one. What separates them is what is inside it.

The Analyst, Implementer, and Reviewer each declare their own sandbox.ts, so each gets a separate Vercel Sandbox holding its own clone of FACTORY_REPO. The three declarations are functionally identical, and none is more isolated than the others. The Reviewer's copy just carries the most weight, because it fetches the pushed branch into a clean checkout and therefore reviews what was actually pushed rather than the Implementer's working tree.

The Classifier and Researcher never set one up, so they get the default: a working environment with an empty /workspace and no repository in it.

The three repository sandboxes share their bootstrap through agent/lib/github/repo-sandbox.ts. The clone and FACTORY_SETUP_COMMAND run once per template build, and each session pays only a fetch. Git always targets the literal https://github.com/<FACTORY_REPO>.git URL rather than origin, because remote config inside a sandbox is model-writable.

What persists and what does not

The last row explains a design choice that looks odd otherwise. Because each dispatch starts clean, the red-CI fix loop counts its own earlier comments on the pull request thread to enforce its 2-attempt cap. The thread is the only durable record those runs share.

Handoff artifacts are durable in storage without being durable in practice. Nothing lists or expires them, but an id only ever reaches the next station through a delegation message, so a later run has no way to find one. That makes them a handoff mechanism rather than memory.

OpenAI has roughly doubled its run rate from the end of 2025 due to the growth of its AI coding software, momentum from subscription sales and its nascent advertising business, and its core consumer business.

Today we’re launching Bluesky Protocol Services: a new brand, and a new website, for the public infrastructure Bluesky operates on the AT Protocol network.

Bluesky has always run more than the Bluesky app. We operate Jetstream instances, relays, and the Bluesky API endpoints built on atproto. But if you were a developer trying to build on that infrastructure, our docs didn’t always make it easy to tell what we run as a service or where to start. We’re fixing that today. Bluesky Protocol Services organizes all the documentation developers need within the ecosystem, clarifies the service contracts around Bluesky-provided infrastructure, replaces the old docs.bsky.app site and gives us a clean way to ship future releases like the ones in this post!

Jetstream v2: Network Replay

The headline release shipping alongside the new site is Jetstream v2. Jetstream is the best way for most developers to use the network at scale: you describe the slice you want, and it arrives as plain JSON over a WebSocket. What it couldn’t give you was history. If you needed the records that already existed on the network, you had to backfill repos yourself then cut over to the live stream.

Jetstream v2 adds that capability to the server. It keeps a compressed archive of the whole network and adds a new way to consume it, alongside the live tail:

Network Replay lets you catch up from any point in the past and cut over to live with no gap. You POST your filters to planSnapshot, download the sealed segments it returns over plain HTTP, then connect the live WebSocket once at the tip. Replay is stateless on the server, with no per-consumer cursor, no subscription to register, and nothing to stage on the client. Jetstream is your buffer. You can also just snapshot the network — a point-in-time copy of the archive over HTTP only (listSegments + getSegment), with no live tail. Same archive, same filters, no WebSocket.

This unlocks much more sophisticated server-side slicing without ever backfilling locally: you can spin up an App, run an analysis over a month of posts, or recover from downtime, all through the same JSON shape as the live tail.

Serving these archives is bandwidth-intensive. To ensure the service remains reliable and cheap to run, we’re now requiring an API token just for these requests. The live tail remains open and unauthenticated, it’s only when you request an archive that we require a token. We have no plans to introduce an auth requirement for the live stream.

The v2 instances are live now at wss://jetstream.us-west.bsky.network and wss://jetstream.us-east.bsky.network. The existing v1 instances will keep running unchanged for a while, and the live tail behaves identically on both, so there’s no rush to move. Read the full flow in the Network Replay docs.

And! As always, this infrastructure is open source and self-hostable. See Running your own Jetstream for details.

A Jetstream SDK

Jetstream is plain JSON, so you never need an SDK. But there’s some common glue: reconnecting, deduping, cursor management, decoding events into typed records. Hence, the new Jetstream SDKs: TypeScript and Go clients where you construct a Jetstream object, pass a filter, and for await over decoded, typed events:

import { Jetstream } from '@bsky/jetstream'
import { app } from '@bsky/sdk/lexicons'

const js = new Jetstream('https://jetstream.us-east.bsky.network')

for await (const evt of js.live({ collections: [app.bsky.feed.post] })) {
  if (evt.kind === 'commit' && evt.commit.operation === 'create') {
    console.log(evt.commit.collection, evt.commit.record.text)
  }
}

The TypeScript SDK is available from npm, including npmx.

The Go SDK is available as part of the Jetstream project.

The Jetstream SDK docs go into more detail.

The Bluesky TypeScript SDK, rebased on lex

Back in May we promoted the lex SDK to stable preview and promised that the standalone Bluesky docs would follow. That’s now done: the Bluesky TypeScript SDK is rebuilt on top of @atproto/lex, which means we’re no longer maintaining legacy code paths for Bluesky-specific helpers. This is the lexicon toolchain, fully typed end to end, from the protocol layer up through app.bsky records.

Every TypeScript example on this new site is written against it, marking a huge move away from legacy technical debt—this is good code hygiene for us, and should eliminate LLM recommendations for deprecated SDKs. If you’re still using @atproto/api code, it continues to work as before, and the Bluesky API guides serve as a migration reference.

Updates to endpoints.bsky.app

Finally, the HTTP reference has been updated. Spinning those docs out of https://docs.bsky.app was actually step 1 of this overhaul; we’re now landing the remainder.

The new network.bsky.jetstream.* methods that power Replay — planBackfill, listSegments, getSegment, and getBlock — are now browsable with full request and response schemas, and the reference now documents Jetstream’s WebSocket endpoints too, so the entire Jetstream v2 surface lives in one place.

What’s next

Everything above is live today: the new site, the v2 Jetstream instances, the SDK preview, and the updated HTTP reference. If you’re new to the network, start with How It Works, a visual walkthrough of how records, lexicons, and the firehose fit together. If you’re building against the Bluesky app’s data model, the Bluesky API guides are all still there, freshly rewritten (more on that below)

If you build something on Replay in the next few weeks, we’d love to hear about it; the fastest way to shape where the SDK’s orchestration goes is to show us what you’re folding the stream into.

The price and performance frontier for coding tasks features a huge diversity of models and harnesses: in 2026 alone, we’ve seen 33 new models released. In our prior post about benchmarking against the Databricks codebase, we found that models cluster into capability tiers and that much everyday work (e.g., flipping a flag, a single-file edit, a well-scoped bug fix) did not require the most expensive models.

So how do you reduce AI coding costs without sacrificing developer productivity? One of the biggest opportunities is matching each task to the right model instead of defaulting every task to the most capable (and most expensive) option. Just leveraging lower cost models can save you 50%+, but it’s incredibly daunting for users. With a proliferation of great models and capable harnesses, coding agent users are constantly faced with choice overload. Instead of wasting time trying to select the best model for every task, many are setting the most capable at the highest effort and moving on. Instead of asking users to choose or stunting productivity with hard caps, we knew we needed to innovate.

That’s why we’re launching the next major cost control in Unity AI Gateway: Smart Routing, now available in Beta. Unity AI Gateway provides a central place to get access to AI, manage spend, and enforce controls across your entire enterprise, and Smart Routing adds intelligent optimization by automatically matching tasks to the right model based on complexity. Smart Routing works directly in Claude Code and Codex, allowing you to optimize the tools developers already use.

And we’re going beyond model routing. With Omnigent, our meta-harness for coding agents, teams can leverage the full power of Smart Routing by optimizing across both models and coding harnesses, giving developers the right combination for the task without having to choose it themselves.

The results speak for themselves: On internal coding workloads, Smart Routing outperformed any single model at just 65% of the cost per task of a leading model like Opus 5. On public benchmarks, Smart Routing matched Opus 5 on performance at less than half the cost.

Here’s what we learned:

• Most of the win comes from using cheaper models for simpler tasks. We see a wide diversity of tasks internally, and most of them don’t need a premium model. We configured the router to select cheaper models for simpler tasks, but to “escalate” complex work that requires frontier-model performance.
• We saw good results using only the information available at the start of the task (description and metadata). We didn’t provide the answer, tests, or anything about the repository, and Smart Routing was still able to choose effectively.
• There’s still substantial room for improvement in effectively sizing work complexity and escalating when needed. A router with perfect foresight would beat every single model at a fraction of what ours spends, and in real-life sessions, it can also be helpful to reassess midway through whether the task has gotten more complicated. Closing this gap is both a research and harness design problem. We need to learn from real user feedback to improve.

Let’s walk through how we built this.

How does intelligent model routing work?

Intelligent model routing selects the model best suited for a task based on factors like complexity, capability and cost. For coding agents, an important decision is when that routing should happen.

There are generally two approaches to routing:

1. Per-request routing: In a given session, some teams have explored routing each request purely based on the complexity of the prompt for that message. The challenge is that, at scale, costs are dominated by cache hit rate. Having a high cache hit rate requires routing consecutive turns to the same model (and, currently, to the same effort level for the popular models).
2. Task-aware routing: At the beginning of the session, you start by assessing the complexity of the task and then suggest a model and harness for it that you stick to for the duration of the session. This preserves the cache-hit rate and provides an opportunity for future optimization, such as upgrading or downgrading as needed when the cache becomes stale (e.g. when there is a compaction event).

How our Smart Router works

We opted for task-aware routing to preserve cache efficiency while matching each coding task to the appropriate model and harness. The most interesting problem is judging how difficult a task is before starting it. We wanted to start simple, so our router currently uses a single policy and applies it to every task the same way.

First, we classify the task. For this, we use a cheaper, low-latency model that reads the task description and labels it with a handful of semantic fields: what part of the system changes, what code evidence the prompt carries (a snippet, a traceback, or nothing explicit), how it appears to be failing, how localized the fix looks, and what kind of project it belongs to. From these, the router derives a task-type family and a language family. Using a frontier model would tax every request (even the simple ones we want to save on), so the extractor is intentionally small and fast.

Then, we triangulate on which model class is best. The router defaults to a medium-sized model and uses the labels to move in either direction, escalating to a more expensive model when the task demands frontier-level capability and knowledge or delegating down to a cheaper one when it does not. This means the single policy can leverage a whole suite of models.

The early results are promising. Against our own internal benchmark, which no labs have had access to, we saw 35% savings. Against public coding benchmarks that demonstrate our results generalize, we achieved 56% cost savings. We expect to see this grow as we learn more about our own use cases and with our design partners.

How do you route coding tasks across models and harnesses?

Smart Routing handles the routing decision, but then you need to be able to act on it. It works natively inside of Claude Code and Codex, but for coding agents, we see better performance by choosing not only the right model, but also the right coding harness. Helping engineers take advantage of the chosen model and harness requires a layer sitting above the individual coding sessions to orchestrate across them. This is why we built Omnigent.

Smart Routing is implemented in Omnigent at two levels:

First, developers using Omnigent can select Smart Routing instead of manually choosing a specific coding harness. Omnigent then automatically selects both the harness and model for each task, with model routing powered by Smart Routing in Unity AI Gateway. This design gives developers and admins the freedom to provide customizations, such as org-level guidance or the option to use previous conversation history, without changing the client every time.

This also means all sub-agent launches go through the Smart Routing API, allowing sub-agents to leverage a different harness and model. The user's initial prompt is often underspecified and hard to judge in terms of complexity, so sub-agents allow you to adjust new work based on new information, with a fresh cache and clear instructions. A single task can experience nuanced routing decisions across planning and parallel sub-agent work (e.g., you can route large codebase summarization tasks to cheaper models while designing the architecture with more expensive ones), leading to even more substantial savings.

How do you evaluate whether model routing is working?

Effective model routing needs to optimize for both cost and developer productivity, and not only cost alone. It’s critical to have feedback signals since routers are still early technologies that will need substantial iteration. Our first step here was to log all coding session traces for later evaluation. We want to consider both cost and developer experience – we don’t want to optimize cost at the expense of productivity.

With Unity AI Gateway, traces for coding agents can be recorded into Unity Catalog. This is extremely sensitive data, and it needs to stay governed by sophisticated tagging and access policies in most mature enterprises.

We used both AI models and human review to analyze traces to evaluate changes to the router. When we ran this analysis on our own sessions before routing, we observed that a large share of sessions were spending frontier-model money on work that did not need it, simply because the default model was the most expensive. In practice, the way to validate that the router is useful is to continuously monitor the following metrics:

• Breakdown of sessions by models
• # sessions that are completed by the routed models end-to-end
• Amount of dollar savings from routing

Where we are taking intelligent model routing next

We believe this is an area with significant opportunity and plan to continue conducting substantial research here. It’s still early, so we have a lot to learn.

Our first challenge has been unreliable benchmarking data that doesn’t match real user behavior. Benchmark tasks are unusually well-behaved, with each arriving as a self-contained statement of work. While routers perform well on such tasks, real sessions often are not like that at all.

• Opening prompts are rarely precise, since what a developer types first is a symptom or a rough intention rather than a specification, and our router reads that first message and commits.
• Sessions get reused, so the decision that was right for the first request can be wrong by the fourth, and nothing asks the router again.

So we’re researching a few new directions to gather more information and try new techniques:

1. Start where task scoping is free. PR reviews, sub-agent launches, batch migrations and scheduled jobs are fully specified at the outset because a machine wrote the task statement. Routing works on this class today without changing anyone's habits, which is why we are deploying here first.
2. Route after a few turns, not on the first one. For interactive work, the opening prompt is the worst available moment to decide, and nothing forces us to decide there. Instead, it can be useful to let a cheap model handle the initial exchange and ask clarifying questions, then route the task once it has taken shape. It is better to explore with a fast and small model, so this should improve cost and experience together.
3. Make sessions smaller. Sessions that stay on one task route better and cost less, and tooling can encourage that by making a fresh session the obvious move when the subject changes.
4. Make switching cheap. All of the above need mid-session model changes to be affordable. In today’s world, with costs dominated by cache-hit-rate, switching mid-session is untenable at scale. Context compaction is the natural seam, since a cache miss is already happening there. Over time, we want the routing layer to explicitly price cache misses rather than bake that into how we use the router.

Routing is usually pitched as a way to spend less. While most of our wins come from paying lower prices for easy work, the same machinery can help us decide when to spend more to get a better outcome. Valuemaxxing cuts both ways: take the cheap model when it suffices, and be confident in spending more when the value justifies it.

Coding tools often incentivize users to consume more and more tokens, but what we really want is to optimize productive output per dollar, not tokens. Picking a cheaper, faster model when it suffices doesn’t just save money. It also saves time, and it keeps scarce frontier capacity available for the tasks that genuinely need it.

Try Smart Routing today

Smart Routing is now available in Beta through Unity AI Gateway. It automatically routes coding tasks to the right model based on complexity, helping teams achieve frontier-level performance with 30%+ cost savings by selecting the best model for every task. For teams looking to reduce AI coding costs without limiting developer choice or productivity, Smart Routing provides an alternative to manually selecting models or relying on blunt spending caps. And with Omnigent, you can extend intelligent routing across models and coding harnesses.

To get started, visit our docs pages:

• For model routing: Enable Smart Routing through Unity AI Gateway
• For model + harness routing: Use Smart Routing with Omnigent (v0.9.0+)

Learn more about Unity AI Gateway by visiting our website.

Progressive rollouts allow DevOps teams to deploy a release to a small subset of production users before rolling it out to the entire user base. This approach reduces risk by allowing teams to validate the release in production and catch any issues before they affect all users. Typically, the rollout automatically promotes a new version of an application to increasingly larger percentages of production users, like 10%, 50%, and finally 100%. If there is an error, the rollout is halted.

The AWS Well-Architected framework recommends staggered deployments, noting that:

These techniques contribute to safer and more reliable software deployment and release processes.

In the previous post, you created a project that used a Claude agent step to categorize commits.

In this post, you will create a sample project that demonstrates a progressive rollout through multiple production environments.

Prerequisites

• An Octopus Cloud account. If you don’t have one, you can sign up for a free trial.
• The Octopus AI Assistant Chrome extension. You can install it from the Chrome Web Store.

The Octopus AI Assistant will work with an on-premises Octopus instance, but it requires more configuration. The cloud-hosted version of Octopus doesn’t need extra configuration. This means the cloud-hosted version is the easiest way to get started.

Creating the project

Paste the following prompt into the Octopus AI Assistant and run it to create a sample project with a progressive rollout:

Create a new progressive deployment project called "18. Progressive rollout".

The resulting project models a gradual production rollout by promoting the same release through progressively larger slices of production.

The AI Assistant creates a lifecycle with the environments Prod 10, Prod 50, and Prod 100. The lifecycle captures the different stages of the rollout as a percentage of production traffic, enforces the deployment order, and has the project deploy a release to each environment in turn.

How the progressive rollout works

The project creates a custom lifecycle called Progressive with four phases:

• Development
• Prod 10
• Prod 50
• Prod 100

Each lifecycle phase targets a single environment, and the project uses a runbook to explicitly trigger the next deployment after the current one succeeds.

The deployment process starts with a Deploy App step that simulates deploying an application by printing Deploying app to the task log. It is followed by a Simulate Failure step that acts as a validation gate. This step checks the prompted variable Project.SimulateFail, and if it is set to True, the deployment exits with an error and the rollout stops.

If the validation step succeeds, the process runs a community step template called Run Octopus Deploy Runbook. This step starts a runbook named Deploy Release to promote the current release to the next environment. This works around a limitation where Octopus prevents a deployment to the next environment until the current one is complete, so you cannot trigger a deployment to Prod 50 while the Prod 10 deployment is still running. By having a runbook trigger the deployment after a short delay, we can be sure the current deployment has completed before the next one starts.

The Run Octopus Deploy Runbook step is configured to run in the Prod 10 and Prod 50 environments. It dynamically chooses the next environment with the following logic:

• When the current environment is Prod 10, it triggers a deployment to Prod 50
• When the current environment is Prod 50, it triggers a deployment to Prod 100

The step also passes the current release ID into the runbook as the prompted variable Project.Release.Id, ensuring the same release is promoted through each stage of the rollout.

The runbook itself contains a single Sleep step that waits for 60 seconds before using the Octopus API to create the next deployment. This pause allows the current deployment to complete before the next rollout stage begins.

In practice, the rollout looks like this:

• You create a release and deploy it to Development
• You promote the release to Prod 10
• The Run Octopus Deploy Runbook step automatically starts the Deploy Release runbook
• The runbook waits 60 seconds and then creates a deployment of the same release to Prod 50
• When the Prod 50 deployment succeeds, the same pattern is used to create the final deployment to Prod 100
• If there are any failures, the rollout stops

Customizing the rollout

The Deploy Release runbook initiates a deployment to the next production environment after a short delay. This may be customized to instead schedule a deployment at a specific time, which allows the rollout to be paused for a longer period of time before continuing. You could, for example, only roll out to 100% of production traffic during off-peak hours, or after the release has been validated in Prod 50 for a full day.

You may also consider preventing release progression if a deployment fails. This ensures that a failed release cannot be promoted to the next environment until the issue is resolved. A blocked release will also prevent any scheduled deployments from taking place.

Comparing tenants and environments

This example used environments to represent progressive rollouts. It is also possible to use tenants to represent progressive rollouts. However, there are benefits to using environments:

• Environments are easier to visualize in the Octopus UI
• Lifecycles enforce the progression of releases through environments, which in turn progressively advance the rollout
• The ability to block release progression after a successful deployment is only available for environments, not tenants

For these reasons, environments are the recommended approach for modeling progressive rollouts in Octopus.

What just happened?

You created a sample project with:

• A custom lifecycle called Progressive that promotes releases through Development, Prod 10, Prod 50, and Prod 100
• A scripted deployment process that simulates an application deployment and then validates the result before continuing
• A prompted variable that can intentionally fail the validation step to stop the rollout
• A community step template that runs a Deploy Release runbook to promote the same release to the next production environment
• A runbook with a delayed API call that chains the rollout from Prod 10 to Prod 50, and then from Prod 50 to Prod 100

What’s next?

The next step is an example of blue/green deployments.

Your infrastructure is healthy. CPU is normal. Memory is fine. Database response times look good. Yet users are reporting slow load times and broken features.

That's because backend telemetry only shows part of the picture. It can't tell you what an end-user on a mobile device, slow network, or older browser is actually experiencing. Digital experience monitoring (DEM) closes that gap by measuring performance from the user's perspective and connecting frontend experience data to application performance monitoring (APM), logs, and infrastructure telemetry.

This guide covers the major DEM categories, what to look for when evaluating platforms, and how the leading options compare. More importantly, it explains why the most effective DEM belongs inside a unified observability strategy rather than living in a standalone tool.

Key takeaways: Digital experience monitoring tools

• Digital experience monitoring measures performance from the user's perspective, capturing issues backend monitoring can miss.
• Real user monitoring, synthetic monitoring, session replay, and endpoint monitoring each reveal different parts of the user experience.
• Evaluating DEM tools in isolation creates gaps between frontend symptoms and backend causes.
• Fragmented tooling slows troubleshooting and issue resolution when performance issues span multiple systems.
• Connecting DEM data with APM, logs, and infrastructure telemetry improves end-to-end visibility. New Relic delivers this through a unified observability platform rather than a separate toolchain.

What is digital experience monitoring? (and how it differs from APM)

Digital experience monitoring is the practice of measuring software performance from the user's perspective — across browsers, devices, and network conditions. Where application performance monitoring (APM) focuses on backend services and infrastructure, DEM focuses on what users actually experience when they interact with your application.

That distinction matters because the two don't always agree. A healthy backend can still deliver a poor frontend experience if JavaScript blocks rendering, a CDN underperforms in a specific region, or a third-party service fails silently. DEM captures frontend latency, client-side errors, rendering delays, and other performance issues that backend monitoring tools may miss.

According to the HTTP Archive's 2025 Web Almanac, only 62% of mobile pages achieve a good Largest Contentful Paint (LCP) score, highlighting how common frontend performance issues remain across the web.

The challenge is that many engineering teams run DEM and APM in separate tools. Engineers can see a frontend symptom in one dashboard and a backend metric in another, but connecting the two often requires manual correlation and slows troubleshooting.

When DEM data connects directly to APM, logs, and traces in a single monitoring platform, engineers can move from symptom to root cause faster, with the end-to-end visibility needed to understand how backend events affect the customer experience.

Understanding key digital experience monitoring tool categories

DEM isn't a single capability. Instead, it's a set of monitoring approaches that cover different parts of the customer journey. Understanding what each one measures helps you assess whether a platform gives you complete coverage or leaves gaps.

Real user monitoring (RUM)

RUM collects performance data from actual users in real time. A lightweight script captures metrics such as page load times, Largest Contentful Paint, and JavaScript errors as users interact with your application.

The advantage is that RUM reflects real-world conditions across devices, browsers, and network paths. The limitation is that it's reactive: you only see data after a user experiences a problem.

Synthetic monitoring

Synthetic monitoring runs scripted simulations of user journeys from locations around the world, whether or not real users are active. It's proactive coverage that helps teams identify potential issues before they affect customers.

Teams use synthetic monitoring for uptime checks, SLO validation, and regression testing after deployments. Combined with RUM, it provides both early warning and real-world validation.

Session replay

Session replay records and replays user sessions so teams can see exactly what users experienced. It's particularly useful for frontend debugging when logs alone don't provide enough context.

The tradeoff is privacy complexity. Session replay tools need strong data masking controls to avoid capturing sensitive information such as passwords, payment details, and PII.

Employee and endpoint DEM

Not all DEM is customer-facing. End-user experience monitoring tracks the performance of internal applications, network paths, and SaaS tools used by employees. For IT teams managing distributed workforces, this often delivers immediate operational value.

How to evaluate digital experience monitoring tools

The most common DEM evaluation mistake is assessing tools in isolation rather than asking how they fit into your broader observability strategy. A feature-rich tool may still create operational gaps if it can't connect frontend experience data to backend telemetry.

Fragmented DEM tooling slows troubleshooting and issue resolution. When performance issues span both the frontend and backend, engineers often have to manually correlate data across dashboards. That delays MTTR and creates blind spots during incidents.

New Relic eliminates this problem by connecting DEM telemetry with application performance monitoring (APM), logs, traces, and infrastructure data in a single, unified observability platform.

Instrumentation and integration fit

Start with instrumentation. Does the tool support your frontend frameworks, single-page applications, and existing digital platforms?

Then evaluate integrations. Can DEM data connect with your APM, alerting, and incident management workflows? Check whether network monitoring and database monitoring tools in your stack can share telemetry with the DEM layer — because frontend slowness often has a backend root cause, and the value is in making that connection fast.

New Relic connects browser monitoring, mobile monitoring, and synthetic checks with APM, distributed tracing, logs, and IT infrastructure telemetry, making root cause analysis faster.

Data governance, privacy, and compliance controls

Session replay requires careful review. Look for data masking controls, GDPR and CCPA support, data residency options, and clear documentation about captured user data. In regulated industries, these controls are essential.

Pricing mechanics and total cost of ownership

DEM pricing varies widely. Some vendors charge per user session, synthetic check, or data volume. Since RUM and session replay can generate large amounts of data, it's important to understand how costs scale.

The total cost of ownership includes more than licensing. Running separate monitoring solutions often increases operational overhead. Consolidating DEM, APM, and infrastructure monitoring on a single platform can reduce both complexity and cost.

Top digital experience monitoring tools compared

DEM platforms vary in monitoring approach, integration depth, and target use case. Here's how the leading options compare.

New Relic

New Relic combines DEM, APM, logs, traces, and infrastructure telemetry in a single observability platform. Teams can connect frontend user interactions directly to backend services without switching tools.

Key features:

• Browser monitoring (RUM) with Core Web Vitals tracking
• Scripted and simple synthetic monitors with global test locations
• Session replay with configurable data masking
• Mobile monitoring for iOS and Android
• New Relic AI and AIOps surface patterns across frontend and backend telemetry simultaneously

Considerations: Usage-based pricing requires teams to understand expected data volumes.

Best for: Engineering and DevOps teams looking for unified observability and end-to-end visibility.

Datadog

Datadog offers RUM, synthetic monitoring, and session replay as part of its broader observability platform. Its integration library is extensive, and teams already using Datadog for infrastructure or APM may find it straightforward to extend into DEM.

Key features:

• Browser and mobile RUM
• Multi-step API and browser synthetic tests
• Session replay
• Dashboards that combine DEM and APM data

Considerations: SKU-based pricing can become complex as monitoring coverage expands.

Best for: Organizations already invested in the Datadog ecosystem.

Dynatrace

Dynatrace uses its AI engine (Davis) to automatically detect anomalies and identify root causes across full-stack telemetry, including DEM. It's well-suited to large enterprise environments where automated problem detection reduces manual investigation time.

Key features:

• RUM, synthetic monitoring, and session replay
• Smartscape topology mapping showing relationships across services and user journeys
• AI-driven root cause analysis
• Automatic dependency mapping across applications and infrastructure

Considerations: Enterprise-focused pricing and platform complexity may be more than smaller teams require.

Best for: Large enterprises prioritizing automation and scalability.

Cisco ThousandEyes

ThousandEyes focuses on network-level visibility — measuring performance from the perspective of the network path rather than the application layer. It's particularly useful for diagnosing ISP-level routing issues, cloud connectivity problems, and WAN performance degradation.

Key features:

• Network path analysis and BGP monitoring
• Cloud and internet monitoring
• Endpoint agent monitoring for remote workers
• Internet and ISP path visualization for troubleshooting connectivity issues

Considerations: Does not provide application-layer capabilities such as RUM or session replay.

Best for: Organizations troubleshooting network, WAN, and ISP-related performance issues.

Catchpoint

Catchpoint emphasizes synthetic monitoring with extensive global coverage, including points of presence inside ISPs and last-mile networks. It's built for teams that need to understand how their application performs across diverse network conditions at a global scale.

Key features:

• Global synthetic monitoring
• RUM and API testing
• Network performance monitoring
• Last-mile ISP visibility

Considerations: More focused on synthetic and network monitoring than unified observability.

Best for: Enterprises optimizing global digital services, CDN performance, and network reliability.

Key use cases for digital experience monitoring tools

The value of DEM becomes clear when frontend experience data connects directly to backend observability. Without that connection, teams can see symptoms but struggle to identify the cause. These use cases show how DEM helps teams move from detection to resolution faster.

Tailoring DEM for specific business needs

Ecommerce teams use RUM and Core Web Vitals data to connect frontend performance with business outcomes such as conversion rates and customer satisfaction. When page performance degrades in a specific region, correlating that change with backend telemetry speeds investigation and remediation.

SRE teams use synthetic monitoring for proactive SLO validation and outage detection. Running scripted user journeys on a schedule helps identify potential issues before they affect customers. When synthetic alerts flow into the same workflows as APM alerts, teams can respond from a single location.

IT teams use endpoint DEM to distinguish application defects from network-related disruptions. When employees report slow access to internal digital services, endpoint data helps isolate whether the issue sits with the application, network path, or local device.

Frontend engineering teams use session replay to improve debugging and troubleshooting. Seeing exactly what a user experienced helps teams reproduce issues, understand user behavior, and shorten resolution times.

New Relic AI surfaces anomalies that span frontend and backend telemetry, helping teams identify cross-layer issues without manually correlating data across multiple tools.

Choosing the right digital experience monitoring tools for your stack

Effective DEM is an observability architecture decision, not a frontend monitoring purchase. Teams that evaluate DEM tools without asking how they connect to backend telemetry will hit a ceiling when diagnosing issues that cross the frontend/backend boundary. That ceiling tends to appear during incidents, when the cost of missing context is highest.

Fragmented tooling adds cost in licensing and in engineering time spent correlating data across dashboards. Unified observability removes that friction — DEM telemetry, APM data, logs, and cloud monitoring in the same query interface and the same alert workflow.

New Relic's platform connects all of this in one place — browser monitoring, mobile monitoring, synthetic checks, APM, distributed tracing, logs, and infrastructure telemetry — so your team spends less time reconstructing incidents and more time resolving them.

FAQs about digital experience monitoring tools

How do DEM tools help improve Core Web Vitals and SEO performance?

DEM tools help improve Core Web Vitals by showing where page load times, rendering delays, and other performance issues affect real users. By combining real user monitoring data with performance metrics, teams can identify which pages, devices, or regions need attention and prioritize fixes that improve both user experience and search visibility.

Can digital experience monitoring tools detect third-party service issues?

Yes. DEM tools can identify when third-party scripts, analytics tags, embedded widgets, or external APIs are causing latency, errors, or broken functionality. Synthetic monitoring and RUM data help teams isolate the impact of these dependencies and determine whether they are affecting the broader customer journey.

What teams typically use digital experience monitoring tools across an organization?

DEM supports multiple stakeholders. Engineering teams use it for debugging and performance optimization, SREs use it for incident response, and IT teams use endpoint monitoring to troubleshoot employee-facing applications. Product and UX teams also use session replay and user behavior data to understand how people interact with digital services.

Full article content is not available for inline reading.

Read the original article →

Switchyard

Switchyard is a Rust proxy and library for LLM traffic. It routes requests across providers, translates between OpenAI and Anthropic APIs, records operational metrics, and provides typed, composable routing algorithms.

Why Switchyard? Point a coding agent such as Claude Code or Codex at an open-source model. Switchyard translates between the OpenAI Chat, Anthropic Messages, and OpenAI Responses formats, so the agent keeps speaking its native API while the request is served by vLLM, NVIDIA NIM, Ollama, or any OpenAI-compatible endpoint. The same proxy can spread traffic across several models for A/B benchmarking, apply signal-driven stage routing, or run a custom algorithm you write yourself.

Features

• Protocol Translation: convert between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats
• Multi-Backend Routing: random routing, LLM-as-classifier routing, signal-driven stage-router, or your own algorithm
• Operational Metrics: Prometheus metrics cover requests, errors, latency, tokens, and routing overhead

Maturity

Switchyard is pre-alpha software that is evolving rapidly. The API and algorithms are expected to change significantly before we reach v1.0.

Warning: Experimental software. Not for production use.

Quick Start

Choose the launcher path to run Claude Code, Codex CLI, or OpenClaw through Switchyard. Choose the server path to run Switchyard as a standalone proxy. Choose the library path to embed routing in your own Rust application.

Launcher Path

Install uv if it is not already available, then install the published Switchyard tool:

curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"
uv tool install --python 3.10 "nemo-switchyard[cli]"

The coding agent you launch must also be installed and on your PATH. This does not install the standalone switchyard-server binary; use the Server Path for that.

Set an OpenRouter key and launch against the packaged deployment:

export OPENROUTER_API_KEY="your-openrouter-key"
switchyard launch claude --model switchyard
switchyard launch codex --model switchyard
switchyard launch openclaw --model switchyard

To use your own native TOML deployment, pass its route ID and configuration:

switchyard launch claude --model my-route --config routes.toml

Server Path

Use this path to install and run the standalone Rust proxy. Install Rust with Cargo, then install the published binary:

cargo install --locked switchyard-server
switchyard-server --help

Cargo builds the release binary and installs it into ~/.cargo/bin by default.

Create routes.toml, then validate it and start the server:

export OPENROUTER_API_KEY="your-openrouter-key"
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

Verify the proxy in another terminal:

curl http://localhost:4000/health

Library Path

switchyard-libsy embeds the routing algorithms in your own Rust application. It never calls a model itself: an algorithm decides which target to use and hands every model call back to you, so it drops into an existing proxy, gateway, or agent runtime without owning an HTTP stack. Pair it with switchyard-llm-client when you want the calls made for you.

[dependencies]
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }

Routing Strategies

A passthrough route registers one target under one model ID with no routing decision.

Architecture

Clients keep their native OpenAI or Anthropic API format. Switchyard picks a configured backend, forwards the request in that backend's own format, and translates the response back into the shape the client expects. The server accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Each configured LLM client selects one upstream format.

Documentation

• Getting Started: complete launcher and standalone server walkthroughs
• Core Concepts: LLM clients, targets, routes, model IDs, and routing algorithms
• Routing Overview: choose and configure a routing algorithm
• switchyard-server: server configuration, routing algorithms, and metrics
• switchyard-libsy: embed routing algorithms in a Rust application
• switchyard-protocol: provider-neutral request, response, and streaming types
• switchyard-translation: request, response, and stream translation

Community

• Issues: GitHub Issues

License

Apache 2.0 License. Copyright NVIDIA Corporation.

Beszel

Beszel is a lightweight server monitoring platform that includes Docker statistics, historical data, and alert functions.

It has a friendly web interface, simple configuration, and is ready to use out of the box. It supports automatic backup, multi-user, OAuth authentication, and API access.

Features

• Lightweight: Smaller and less resource-intensive than leading solutions.
• Simple: Easy setup with little manual configuration required.
• Docker stats: Tracks CPU, memory, and network usage history for each container.
• Alerts: Configurable alerts for CPU, memory, disk, bandwidth, temperature, load average, and status.
• Multi-user: Users manage their own systems. Admins can share systems across users.
• OAuth / OIDC: Supports many OAuth2 providers. Password auth can be disabled.
• Automatic backups: Save to and restore from disk or S3-compatible storage.

Architecture

Beszel consists of two main components: the hub and the agent.

• Hub: A web application built on PocketBase that provides a dashboard for viewing and managing connected systems.
• Agent: Runs on each system you want to monitor and communicates system metrics to the hub.

Getting started

The quick start guide and other documentation is available on our website, beszel.dev. You'll be up and running in a few minutes.

Supported metrics

• CPU usage - Host system and Docker / Podman containers.
• Memory usage - Host system and containers. Includes swap and ZFS ARC.
• Disk usage - Host system. Supports multiple partitions and devices.
• Disk I/O - Host system. Supports multiple partitions and devices.
• Network usage - Host system and containers.
• Load average - Host system.
• Temperature - Host system sensors.
• GPU usage / power draw - Nvidia, AMD, and Intel.
• Battery - Host system battery charge.
• Containers - Status and metrics of all running Docker / Podman containers.
• S.M.A.R.T. - Host system disk health (includes eMMC wear/EOL and Linux mdraid array health via sysfs when available).

Help and discussion

Please search existing issues and discussions before opening a new one. I try my best to respond, but may not always have time to do so.

Bug reports and feature requests

Bug reports and feature requests can be posted on GitHub issues.

Support and general discussion

Support requests and general discussion can be posted on GitHub discussions or the community-run Matrix room: #beszel:matrix.org.

License

Beszel is licensed under the MIT License.

Cloudflare Launches Persistent, Stateful, Computer-Like Environments for Agents

Cloudflare has introduced Cloudflare Computer, a new open-source runtime designed to give AI agents something closer to a real "computer" instead of just ephemeral containers. It leverages Cloudflare isolates for fast serverless execution, making agents cheaper, faster, and more scalable, according to the company.

Cloudflare Computer attempts to tackle a major challenge in AI agent deployment. The company argues that relying on containers to run agents will not scale to "hundreds of millions, then billions, of concurrent agents", because there simply is not enough global compute capacity for that.

To address this, the @cloudflare/computer package introduces an agent runtime where the platform decides whether code runs in an isolate, a container sandbox, or a web browser. In this model, "each agent gets a computer, the runtime optimizes for efficiency, and scalability".

Our goal with @cloudflare/computer is to provide an agent with a runtime where a container is required for less than 10% of its work, and coding tasks, audio/video manipulation, and document creation can all be handled by isolates.

According to Cloudflare, isolates, which were introduced with Cloudflare Workers, are "infinitely horizontally scalable" and can start up and shut down extremely quickly. Additionally, they can persist the agent state, hibernate when the agent is not running, as well as spin up their own container sandboxes when needed. Cloudflare maintains that this combination of isolates and container sandboxes is extremely effective because it combines horizontal and vertical scalability:

Cloudflare’s architecture has been designed to run the agent harness in the isolate (in a Durable Object) and call an attached container on-demand as a tool. This allows you to utilize heavier compute primitives only when required, optimizing performance and cost.

One central piece in this architecture is a shared, SQLite-based filesystem accessible to both isolates and containers. This allows tasks to move seamlessly between the two, with the shared filesystem allowing them to work on the same files. Cloudflare Computer filesystems can be used with git repositories, storage buckets or arbitrary files, while ensuring all operations are gated, audited, and observed.

Currently, Cloudflare Computer provides three backends: container projects, where the SQLite state is exposed as a sandboxed container as a real FUSE-mounted filesystem; isolate shells, which runs just-bash environment in a Dynamic Worker; and isolate JavaScript, which runs an ECMAScript module in a fresh Dynamic Worker.

Cloudflare Computer is still an early preview and is only suitable for experiments, exploration and prototypes, says Cloudflare.

As parents in tech, we’ve learned that neither children nor applications thrive without clear boundaries. There are no “good” or “bad” kids, just as there are no inherently “good” or “bad” applications, only behaviors shaped by the guardrails around them. In parenting, we establish rules to encourage safe, responsible decisions while still allowing independence. Modern cloud-native platforms require the same approach. Policy as Code provides those guardrails, defining how applications can operate, what resources they can access, and how they remain compliant at scale without slowing down innovation.

Just as parents establish boundaries to help their children make safe and responsible decisions, platform engineering teams rely on policies to guide application behavior. The challenge is that setting rules alone is not enough; you also need visibility into whether they are being followed, where they are failing, and whether they are creating unintended consequences. Without real-time observability, policy enforcement can become a black box, making it difficult to identify misconfigurations, compliance gaps, or obstacles to developer productivity before they impact production.

This is where Policy as Code and Observability come together. Much like a parent needs feedback to understand how rules are working in practice, platform teams need continuous insight into policy outcomes across their clusters. By combining Kyverno, a Kubernetes-native policy engine, with VictoriaMetrics, an open-source monitoring solution, teams can eliminate policy blind spots and gain a unified view of both compliance and system health, enabling them to scale governance without sacrificing visibility or developer velocity.

Why This Matters: Community Impact and Practitioner Value

Historically, cluster security and infrastructure observability have existed as two distinct silos within platform organizations. Security teams write policy rules, while reliability teams build dashboards.

This framework changes that dynamic by treating policy enforcement as a primary telemetry data source. Instead of relying on passive, text-heavy log files to figure out why a deployment failed, this setup directly transforms admission control webhooks into real-time metric streams. It treats policy as data, allowing organizations to observe compliance trends with the exact same tooling used to monitor CPU usage or network latency.

How It Directly Helps Practitioners

• Removes “Invisible Wall” Friction: Developers frequently encounter admission controllers as sudden, unexplained blocks during deployments. By exposing policy metrics on shared Grafana dashboards, developers can instantly see exactly which validation rules are failing across the cluster, transforming security from a roadblock into self-service feedback.
• Simplifies Governance via Native Tooling: Practitioners do not need to learn specialized, complex programming languages to audit cluster states. Because Kyverno relies entirely on standard YAML and CEL (Common Expression Language), and VictoriaMetrics natively supports standard PromQL queries, any practitioner can modify policies or adjust monitoring dashboards without deep proprietary expertise.
• Prevents Operational Overhead and Budget Bloat: High-volume clusters can trigger thousands of admission requests every minute. For practitioners running traditional monitoring systems, storing these high-cardinality metrics can lead to massive RAM usage and steep infrastructure bills. Using a lightweight time-series backend ensures that teams gain comprehensive policy visibility without incurring high data storage costs.

Operational Challenges Solved

1. Eliminating Specialized Language Overhead

Adopting policy-as-code often introduces a steep learning curve due to specialized, non-native languages. This creates operational bottlenecks where only a small number of engineers can write, maintain, or audit compliance rules. Kyverno addresses this by managing policies as standard Kubernetes manifests using declarative YAML and CEL.

2. Managing High-Cardinality Metrics Scaling

Tracking every admission request, mutation, validation failure, and image verification at scale generates a significant volume of time-series data. VictoriaMetrics is a resource-efficient time-series database that can process millions of data points with low memory overhead. Its native UI Cardinality Explorer helps teams identify which policy labels consume the most storage, ensuring the monitoring infrastructure remains performant.

3. Surface Visibility for Enforced Rules

Policy enforcement can sometimes function as an invisible wall for developers. An admission controller rejects workloads, and there is no central dashboard to track why or how often violations occur. This architecture uses a five-policy implementation that spans the resource lifecycle and forwards raw engine telemetry to real-time dashboards.

Architectural Deployment

Phase 1: Deploying the Core Components

First, install the Kyverno Policy Engine to handle admission control, then install VictoriaMetrics as the time-series database backend.

# 1. Install Kyverno Policy Engine
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace



# 2. Install VictoriaMetrics Single-Node Server
helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update
helm install vm vm/victoria-metrics-single 

Phase 2: Configuring Metric Collection

The vmagent gathers telemetry by scraping the dedicated Kyverno metrics service endpoint (port 8000) and forwarding those data points directly to VictoriaMetrics.

helm upgrade --install vmagent victoriametrics/victoria-metrics-agent \
  --set 'remoteWrite[0].url=http://vm-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write' \
  --set 'extraArgs.promscrape.config-global=\n  scrape_interval: 10s\n\nscrape_configs:\n  - job_name: kyverno\n    metrics_path: /metrics\n    static_configs:\n      - targets:\n          - kyverno-svc-metrics.kyverno.svc.cluster.local:8000' \
  --set 'podLabels.app=vmagent' \
  --set 'podLabels.environment=demo' \
  --set 'podLabels.team=observability'

Verify that vmagent is utilizing the correct target configuration and successfully scraping endpoints:

curl http://localhost:8429/targets

Deploying the good and the bad apps

Just as clear expectations help guide behavior, platform policies help ensure applications follow organizational standards from the moment they are deployed. To demonstrate this, we deploy an application that intentionally violates organizational standards by omitting required Kubernetes labels.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: bad-app
  template:
    metadata:
      labels:
        app: bad-app
        random-id: "12345"
    spec:
      containers:
        - name: nginx
          image: nginx

In contrast, we deploy an application that follows the organization’s labeling and metadata standards.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: good-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: good-app
  template:
    metadata:
      labels:
        app: good-app
        environment: demo
        team: platform
    spec:
      containers:
        - name: nginx
          image: nginx:stable

The Five-Policy Lifecycle Framework

To move beyond simple enforcement and make policy behavior observable, we introduce a structured approach that follows the full lifecycle of a workload in the cluster.

1. Validating Policy

We create a validating policy that enforces the presence of required labels.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-observability-labels
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: check-required-labels
      match:
        resources:
          kinds:
            - Pod
      validate:
        message: "Missing observability labels: app, environment, team"
        pattern:
          metadata:
            labels:
              app: "?*"
              environment: "?*"
              team: "?*"

2. Mutating Policy

To reduce developer friction, we introduce a mutation policy that automatically injects missing compliance labels.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-safe-observability-labels
spec:
  rules:
    - name: add-labels-to-pods-only
      match:
        any:
          - resources:
              kinds:
                - Pod
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              demo: "true"
              observability: "enabled"
              environment: "demo"
              team: "observability"

3. Generating Policy

Generating policies extends governance beyond enforcement by automatically creating required resources at namespace initialization.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-configmap
spec:
  rules:
    - name: generate-config
      match:
        any:
          - resources:
              kinds:
                - Namespace
      generate:
        apiVersion: v1
        kind: ConfigMap
        name: demo-config
        namespace: "{{request.object.metadata.name}}"
        data:
          data:
            message: "Hello from Kyverno"

4. Validating Image Policy

Securing the software supply chain requires restricting pod creation to trusted container registries.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-image-registry
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Only images from Docker Hub library are allowed"
      pattern:
        spec:
          containers:
          - image: "nginx*"

5. Deleting Policy

Deleting policies are designed to prevent unsafe operations by explicitly restricting high-risk actions, such as deleting critical workloads.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: protect-deployments
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: block-deletion-of-deployments
      match:
        any:
          - resources:
              kinds:
                - Deployment
      validate:
        message: "Deletion of Deployments is not allowed in this demo"
        deny:
          conditions:
            any:
              - key: "{{ request.operation }}"
                operator: Equals
                value: DELETE

Dashboard Visualization and Telemetry Analysis

Once metrics are flowing, enforcement events translate directly into real-time Grafana observability dashboards via standardized PromQL queries routed to VictoriaMetrics.

Key Monitoring Metrics

• Total Webhook Throughput: Tracks the total volume of admission requests. kyverno_admission_requests_total
• Total count of admission requests grouped by whether they were allowed or blocked sum by (request_allowed)(kyverno_admission_requests_total)
• Resource Distribution: Aggregated via sum by (resource_kind), shows which resource kinds are most frequently affected by policy enforcement.

Operational Summary

To summarize: policies are not just enforcement mechanisms. They are signals about how your platform behaves in the real world. And when you observe those signals, you transform them into operational intelligence.

Here’s your first look at the newest Pixel devices.

Today, we unveiled new Pixel devices — including new Pixel 11 phones, Pixel Watch 5, and our first-ever Pixel Tag. Here’s a look at everything in the lineup.

Our new phones — Pixel 11, Pixel 11 Pro, Pixel 11 Pro XL, and Pixel 11 Pro Fold — are designed for Gemini Intelligence. They’re more durable and include major camera upgrades that help you capture the moment in your authentic style.

Pixel Watch 5 delivers proactive Gemini Intelligence capabilities, personalized strength coaching, and our most accurate GPS route tracking. It also offers breakthrough Health Guardian features, like blood pressure and insulin resistance trends and breathing emergency detection — an industry-first for a smartwatch.

Our first-ever finder tag, Pixel Tag, uses Google’s Find Hub network to help you locate your favorite items. It’s made of durable stainless steel and has a long-lasting battery, and its built-in speaker plays a sound when prompted.

Visit the Google Store for more information on availability and product updates.

Full article content is not available for inline reading.

Read the original article →

Large language models are not neutral and often reflect or amplify biases related to language, geography, gender, culture, and socioeconomic status because of the data they are trained on and the feedback systems used to refine them. As a result, AI-generated personas, research summaries, copy, and recommendations can overrepresent dominant groups and present their perspectives as consensus while overlooking minority viewpoints. Designers should treat AI outputs as inputs for further thinking rather than objective conclusions, and actively review them for hidden assumptions, missing perspectives, and demographic bias.

WHAT IS SHOOTCLIP

ShootClip is a video editing application for macOS. You import your footage, cut and arrange clips on a multi-track timeline, add captions, text animations, color grades and blur effects, then export the finished video to your Mac — or publish it directly to your own YouTube channel.

If you connect a Google account, ShootClip uses it for exactly two things: signing you in to activate your license, and — only when you click Publish — uploading the video you chose to your own YouTube channel via the YouTube API. ShootClip never reads your other Google data, and your footage stays on your Mac.

Everything a pro timeline needs. Nothing it doesn't.

Metal-native engine

Real-time playback of 8K ProRes and RAW, tuned for Apple Silicon. No proxies unless you want them.

Magnetic timeline, pro trims

Ripple, roll, slip, and slide with frame-accurate keyboard control and instant audio waveforms.

Built-in MCP server

Expose your whole project to Claude or any MCP client. Every edit is scriptable, reviewable, undoable.

Text that moves

Standalone text tracks with 19 Canva-style animations — typewriter, burst, skate, roll, merge, and more.

Blur & censor layers

Drop blur, mosaic, or black-box clips on the timeline to cover anything, anywhere, for exactly as long as you need.

Export anywhere

H.264, HEVC, ProRes, and social aspect presets — rendered in the background while you keep cutting.

Three features you have to see.

AI OBJECT TRACKING: A censor that follows the subject.

Draw a blur over a face or a license plate once — the AI tracker locks on and follows it frame by frame. Gaussian, mosaic, or black box, it stays glued to the subject however the shot moves.

TEXT BEHIND SUBJECT: Titles that live inside the scene.

ShootClip segments the person or object on every frame, so your text slides behind them automatically — the cinematic layered look with zero manual masking or rotoscoping.

AUTO CAPTIONS: Captions that write themselves.

One click turns speech into styled, frame-accurate captions — AI transcription built in, no API key needed. Pro includes 300 minutes of video every month.

Your AI assistant is now an editor.

Every copy of ShootClip ships with a built-in MCP server — free for all accounts. Connect Claude or any MCP client and it can read your timeline, cut clips, apply grades, and export — with every action reviewable and undoable.

• Works with Claude Desktop, Claude Code, and any MCP client
• Tools for cutting, effects, media, playhead, and export
• Free for every account — sign in and it switches on

claude · connected to shootclip-mcp
you › remove silences in the interview track, keep pauses under 400ms
shootclip.timeline.analyze_audio ✓
shootclip.timeline.ripple_delete × 23 ✓
shootclip.timeline.crossfade × 23 ✓
claude › Removed 23 silences — runtime is now 11:42, down from 14:05. Want me to export a review copy?

Free to edit, AI included. Pro to go further.

Free

Forever

• Full editor — cut, grade, mix
• MCP AI assistant — Claude & any MCP client
• AI silence removal & agent-driven cuts
• Text animations & blur layers
• Unlimited projects
• Export up to 1080p — no watermark

ShootClip Pro

$7/month or $19/year — save 77%

• Everything in Free
• Auto captions — 300 min of video / month
• No API key needed — included with Pro
• Moving censor — face & object tracking
• 4K export
• Auto-activates when you sign in
• Priority support

Review product changes in the age of AI.

Agents generate more changes than teams can review. Argos makes every change obvious, then lets your team and your agents comment on the diffs, request updates, or approve before merge.

See exactly what changed

Catch, compare, and review every change, from pixels to any file. Stay focused on real differences and skip the noise.

Review at a glance

Baseline and changes side by side, with a built-in highlighter and keyboard shortcuts to approve or reject in seconds.

Any file, not just pixels

Snapshot screenshots, Markdown, JSON, HTML, CSS: anything your app or your agents produce.

Only the real changes

Deterministic pixel diffing catches the real change and ignores rendering noise, so every diff you review matters.

“Argos has been a game-changer for us. It catches even the smallest visual changes in our diagram rendering, giving us peace of mind before every release. The seamless integration with our CI pipeline makes it an essential part of our development process.”

Keep your team on the same page

Your team and your agents review together, right on the change.

• Pin your comment

  Click on a pixel and start the conversation right there. Everyone follows along.

• Bring the reviewers you need

  Add the designer, the feature owner, whoever you want. Every review counts.

• Your agent takes it from there

  Send the thread to your agent, straight into your editor. The fix lands in the next commit.

Your agents check their own work

They read what their change did, fix what they broke, show their work, and only bring you what’s left.

Agents see what you see

Every diff in the Argos UI is also structured data, from the CLI and over MCP. No dashboard, no screenshots pasted into a prompt.

Agents fix what they broke

When Argos surfaces a change nobody asked for, the agent has what it needs to correct it and push again.

Agents show their work

A screen recording, uploaded from the terminal, lands on the PR as an Argos comment. The reviewer sees the feature work without checking out the branch.

You only review what’s left

The regressions an agent catches never reach you. What lands in your queue is the change that actually needs a human — with the proof beside it.

Deploy Storybook on every PR

Turn Argos into a host for your static builds. Storybook, Vite, Next.js exports, or plain HTML, deployed to a live URL and wired straight into your pull request.

• A live preview on every PR

  Run argos deploy ./storybook-static and get a unique, shareable URL for each pull request, with no extra infra to maintain.

• Preview and production, built in

  PRs ship to immutable preview URLs; merges promote to a stable production domain, all reported in the PR comment.

• Access protection

  Keep previews private to your team while leaving production public, or lock everything behind Argos login.

Integrated with your everyday tools

First class integrations with GitHub, GitLab, Slack, and Microsoft Teams so your reviews happen where your team collaborates.

Keep flakiness under control

Detect, manage and fix flaky tests. Track instability across builds and automatically silence noise.

See why E2E tests fail

Investigate failed tests with full visual context. See what broke, understand why, and fix it instantly.

Cut visual testing costs, not coverage

Argos gives you reliable snapshots and scalable review flows while keeping your testing budget under control.

No AI diffing surcharge

Argos uses deterministic pixel diffing, so you don’t pay extra AI fees per snapshot.

Stay in control of your budget

Set monthly spend limits and get notified before you exceed your threshold.

Cheaper at every volume

From small teams to millions of snapshots, Argos stays below competitors.

Add your first snapshot in seconds

Argos plugs into Playwright, Cypress, Storybook, WebdriverIO, the CLI, or any framework, so you can start catching changes without changing your stack.

Chosen by fast-moving startups and enterprise teams

Hundreds of companies rely on Argos to catch visual bugs early, speed up reviews and ship UI changes with confidence.

Supercharge your product quality

See every change your team and your agents make. Review with confidence, and merge faster.

Anthropic investors expect the AI startup to float at a valuation of $2 trillion or more in October, a dizzying figure that would eclipse SpaceX and make the AI lab’s debut the largest-ever initial public offering.

Half a dozen of the company’s backers told the FT that Anthropic’s rapidly rising revenue would enable it to more than double its current valuation in a planned autumn float.

A listing at that level could unlock billions of dollars in gains for the five-year-old company’s early investors but would also test public markets that are growing more nervous about the AI boom.

Anthropic’s backers say booming demand for the lab’s advanced AI models and tools justifies their lofty expectations. Investors expect the Claude maker’s annualized revenue to be between $100 billion and $120 billion by the end of 2026—using the startup’s preferred measure, which infers full-year sales from recent performance—up by more than 10 times over the course of 2026.

“If Anthropic is growing 800 percent a year, you’d think at the incredibly low end they would trade at 30 times [revenue],” said one investor in the group. “That would make them a $3 trillion company.”

Anthropic lacks a publicly listed US peer that would provide a benchmark for its valuation. But companies that are seen as AI beneficiaries, such as data intelligence group Palantir and cloud company Nebius, have traded this year at roughly 55 times revenue.

Several investors said senior Anthropic executives had yet to fix the valuation target for the IPO, even in private conversations. But investors have built their own financial models.

Their bullish projections come despite mounting challenges, including rising competition from Chinese rivals, pressure for AI regulation, and a simmering feud with the US government.

Those concerns, particularly the Commerce Department’s temporary ban on Anthropic’s best models, contributed to overall revenue growth slowing in the month of June, according to two investors with knowledge of the matter. Even so, they said the company had rebounded and continued to grow at an extraordinary rate even by Silicon Valley standards.

Anthropic declined to comment.

The startup led by Dario Amodei filed paperwork with the Securities and Exchange Commission in June, putting the company in a quiet period that limits public announcements about its financial performance.

Anthropic has gained ground on rivals OpenAI and Google this year, releasing models that have outperformed competitors while focusing on sales to business customers. The group announced in May that its annualized revenue had surpassed $47 billion.

Venture capitalists, sovereign wealth funds, and other institutional investors have poured just under $100 billion into the company in 2026. Anthropic’s valuation leapfrogged OpenAI’s for the first time in May, reaching $965 billion, including the new investment.

But the group also faces considerable uncertainty. It has repeatedly clashed with the Trump administration and remains in active litigation against the US Department of Defense, which labeled Anthropic a supply-chain risk earlier this year.

Anthropic has since been forced to briefly pull its leading models, Fable 5 and Mythos 5, after being hit with export controls by the Commerce Department in June. The episode spooked some customers who rely on Anthropic models.

Customers are also increasingly sensitive to the price of accessing the best models. Faced with spiraling costs, they have, in some cases, reversed directives for employees to maximize their AI use and opted for less powerful, cheaper models.

Anthropic’s market-leading model costs more than two and a half times as much to use as OpenAI’s flagship, while Chinese open-weight alternatives, which have also improved dramatically this year, are a fraction of the cost, according to Artificial Analysis, which analyzes AI models.

Anthropic increased its market share among US businesses last month, according to data from payments group Ramp. But analysts at the company found that businesses were “hitting their limit on AI spend” and turning to cheaper alternatives.

“It’s easy to come up with challenges,” said an Anthropic investor who has also backed AI groups including OpenAI and SpaceX, which went public at a $1.77 trillion valuation in June. “But the company continues to be in first position in performance, positioning, and what people want exposure to.”

Much of the financing Anthropic needed for its infrastructure buildout was assembled before the company's revenue spiked. The buildout suggests that financing is unlikely to be the immediate limit on frontier computer growth. Institutional investors appear willing to lend capital against commitment to long-term payments, especially if more established organizations are willing to back up part of the risk.

Google is rolling out Sheets canvas, a Gemini-powered feature that turns spreadsheet data into custom, interactive “mini-apps” inside Google Sheets. The new read-write layer sits directly on top of a spreadsheet, giving users a visual way to organize, edit and navigate information without formulas, programming, or a separate app.

Users can open a spreadsheet, select “Create canvas” in the Ask Gemini side panel, and describe the result they want in natural language. Gemini builds a layout from the existing data, and follow-up prompts can alter its design, arrangement, or functionality. Edits made in either the canvas or the underlying sheet sync in real time. Because the canvas lives as a tab in Google Sheets, it can also be shared with collaborators like a regular sheet.

Staring at hundreds of rows and columns, wishing your spreadsheet was an interactive dashboard? 📊✨ Introducing Sheets canvas in Google Sheets—turning static data into interactive "mini-apps" with a simple prompt. Available now globally in English to Google AI Pro and Ultra…

Google is positioning Sheets canvas for everyday projects where long rows and columns make key details harder to see. Students can turn assignment lists into visual study trackers with progress, reminders, and a view of the week ahead. Fantasy football players can create roster command centers for comparing player statistics and standings. Wedding planners can convert RSVP records into seating charts, move guests between tables, and keep the central spreadsheet current.

The launch gives Gemini a new role in how people present and work with data already held in Sheets. Instead of requiring code or a separate tool, Sheets canvas lets people build a task-specific view through prompts while keeping the spreadsheet underneath it. Its two-way syncing is central to the approach: the visual layer is not a static presentation, and updates remain connected to the source data.

Sheets canvas is now available globally in English to Google AI Pro and Ultra subscribers. Google has also begun rolling it out to Workspace customers on Business or Enterprise Standard and Plus plans, as well as subscribers to the Google AI Pro for Education add-on. Access begins inside an existing Google Sheets spreadsheet through the Ask Gemini side panel.

• OpenAI revenue chief Denise Dresser is leaving after eight months at the company.
• Her departure comes days after longtime exec Brad Lightcap departed. Business chief Fidji Simo left her role at the company last month.
• The AI startup confidentially filed its IPO prospectus with the SEC in June.

OpenAI said Thursday that Chief Revenue Officer Denise Dresser is leaving her role less than a year after she joined the artificial intelligence lab, a startling exit for a company that's gearing up for what's expected to be a blockbuster IPO.

The company said Dresser is leaving "to pursue other opportunities," according to a release. OpenAI has hired Dali Rajic, who previously served as the president and chief operating officer of the cybersecurity company Wiz, to replace her.

"The opportunity to work hands-on with the most transformative technology in the world has been nothing short of incredible," Dresser wrote in a post on LinkedIn on Thursday. "I am so proud of what we have accomplished and, even more, of how this team has shown up for our customers and one another."

Dresser's departure marks the second senior exit at OpenAI in days. Longtime OpenAI executive Brad Lightcap announced his decision to leave the company on Tuesday, writing in a post on X that he is going to "start something new."

Lightcap had been at OpenAI for eight years and was a longtime friend and colleague of the company's CEO, Sam Altman. Dresser, by contrast, was hired in December after she spent more than a decade as an executive at Salesforce. Her exit poses a particular conundrum for OpenAI, in large part because she brought the company enterprise expertise that it sorely needed to take on its chief rival, Anthropic.

Dresser was also tapped to take over many of Lightcap's responsibilities in April, when OpenAI said Lightcap was transitioning to a new role focused on "special projects."

In addition to Lightcap and Dresser, Fidji Simo, who had been OpenAI's product and business chief, announced last month she was stepping down from her role at the company to focus on recovery after a "severe exacerbation of a chronic illness." Three other OpenAI executives left the company in April.

The exits come during a crucial moment for OpenAI, as it readies for a potentially massive public market debut. The company confidentially filed its IPO prospectus with the Securities and Exchange Commission in June, just days after Anthropic did the same.

OpenAI closed a funding round in March at a staggering $852 billion valuation, and it will face pressure to justify that figure to investors ahead of its IPO. The company has been racing to build out its enterprise business and win customers in the fiercely competitive market, which was one of Dresser's main priorities.

In January, OpenAI CFO Sarah Friar told CNBC that enterprise customers accounted for roughly 40% of OpenAI's business as of January, but that she expected that figure to grow to closer to 50% by the end of the year. Dresser told CNBC in April that OpenAI's enterprise business was "on track" to reach that goal.

Dresser will stay on at OpenAI for a brief period to "work closely with the business team to support our customers," OpenAI said. She wrote in her LinkedIn post that she has been working with OpenAI President Greg Brockman to ensure a "smooth transition."

"No one works harder than this team," Dresser wrote. "I'm deeply grateful to every one of you."

A China-trained model could give Apple greater control over its AI offerings in the country.

China has a near-total dominance of the supply chain, so building a robot entirely free of Chinese parts is not practical. The Chinese robotics industry was built on billions of dollars in state funding. Fledgling robot makers in the US will need substantial financial and policy support to compete. The US humanoid robotics industry is still in its infancy, and makers of non-humanoid robots are struggling to compete with Chinese rivals.

This Essay is 10% AI Generated

“100% of this text is AI” is the new Scarlet Letter of dunking on people and posts. I don’t love it; it feels bad to have such an easy (and hard-to-police) way to tar and feather something. But it’s here to stay, and it’s a meta-feature of our language now.

AI writing has all the hallmarks of classic preference falsification. The prevalent stated preference is, “I don’t like AI-generated writing”, the revealed-to-be-true preference is, well, we’ll see. The marketplace of writing and ideas has a way of bending towards the true preferences of readers. It feels presumptuous to think that AI might be great at writing code and other kinds of practical output, but remain inferior in prose for much longer. I don’t know.

But I have a confession to make. I don’t like editing or publishing stuff that’s AI-written. I still think that, for now, “you can feel it” a lot of the time. And the Pangram Scarlet Letter test is doing some sort of work, socially. I’d call it “load-bearing”, except we can’t use those words anymore.

I hate to give the French credit for anything, but the 1960s semantic thinkers (Foucault, Deleuze, Baudrillard, and others) who collectively created “French Theory” really cooked when they anticipated Pangram and AI authorship 80 years ago. French Theory was Not Good For Other Reasons (iykyk) but I’m not gonna lie, it’s helped me process how I feel about this step change in what’s happening to writing:

• Dunking on something as AI-written is performing a genuine social function that is not just “AI bad”. It serves what Foucault would call the “author-function.” Look at the people doing the pangram quoting; it’s mostly people in tech, not AI haters. Granted, there’s an adoption curve effect here, but it doesn’t change the observation that “AI is bad” is not what’s going on here, it’s something to do with authorship, as something people want.
• Meanwhile, there is something that has consistently nagged me, personally, about AI-generated writing. Something about the writing itself is quite pushy, as it were, in a self-conscious quest to synthetically recreate the “author-function.” Foucault’s idea, “the writing produces the author, not the other way around” feels important.

Let’s get into it:

French Theory

A book came out recently that got buzzily reviewed in The New Yorker & a few other places: The Frenchmen; or, My Life in Theory, by Emily Eakin. It’s about “French Theory” which is the grouping for a bunch of French guys in the 1960s whose radical work on semantics had a massive influence on the subsequent decades of political thought and social justice. Two people feel particularly relevant here: Roland Barthes and Michel Foucault.

Barthes wrote an important essay in 1967 called The death of the author. His argument was basically that we should no longer assign a text’s meaning as coming from the author. Text’s meaning comes from text, and its relation to other text. In his words: the necessity of substituting language itself for the man who hitherto was supposed to own it.

Our tradition, when reading something and we come across a puzzling or ambiguous passage, is to ask, “What did the author mean by this?” and we do some exercise of putting ourselves in the author’s shoes, imagining their circumstances or their point of view. Barthes argued against this. He claimed, first, that language is not something that belongs to the author at all. Language comes from all over the place, and words inherit and derive their meaning from everything else that’s ever been said. So far, fine.

But then, Barthes argued by proxy that when it comes to the act of writing itself, we overestimate the degree to which the author actually has autonomy and agency over the process. He said, look, the minute a word pops into your head, it is already pre-populated with all this meaning, and all of these implicit constraints, that subsequently prompt the next word of the author’s thought process. (Sound familiar, anyone?)

In other words, the language itself is doing more of the work of writing than the author is! Barthes’ logical end state is that the body of language does all of the work; the author’s role is recombining an inherited field of discourse, and therefore mostly irrelevant to the reader’s quest to understand and synthesize the text. If a reader wants to understand why the author said a certain thing, the answer is in the previously-said-thing. This could be called “inherent authorship”, as in, the meaning was inherently already there; the human in the loop barely matters for the future reader.

This creates a problem. If authors aren’t relevant to the meaning of a text, it gets harder for us to organize all of the writing out there into a coherent relational framework. So two years later, Foucault wrote the essay, “What is an author?” which started from Barthes’ premise, but proposed this follow-up question: “Well, why do we create this thing called authorship, then, that seems to have an important social function?”

For Foucault, even if words have meaning independently of who wrote them, there is nonetheless an emergent quality called “authorship”, which we create and employ socially as a way to categorize the text, assign meta-significance to it, or even to regulate it or police it. He puts it flatly: “The function of an author is to characterize the existence, circulation, and operation of certain discourses in society.”

He’s saying, look: all of this writing is useless if people can’t categorize and digest it, and the way we naturally do that is by its authorship. Rousseau wrote this, Voltaire wrote that. If we don’t have authorship handy as a meaning-compressor, we need to come up with some indexing substitute.

In practice, this is what actually happens! There’s a long tradition, particularly in older scholarship, of assigning “pseudo-authors”, e.g. “Pseudo-Aristotle”, to work whose literal authorship is unknown, but is thematically similar-enough that groupings can be inferred. “Homer” is another example: we don’t know if Homer was a single person, or merely a collection of poetry in a cohesive style that may as well be one person, for posterity.

So, Barthes and Foucault’s back and forth here is a starting point for understanding “100% AI generated” as a cultural dunk on the timeline. Barthes would probably argue, “Why does it matter who wrote it, there’s no such thing as an author anyway, it doesn’t change the meaning of the text.” And Foucault would presumably counter, “That’s fine, but we’re going to call out 100% AI-generated texts anyway, because ‘AI-generated’ is now a kind of authorship category, and calling it out as such serves a social function.” Barthes unbundled authorship, Foucault made the new bundle.

With this in mind, let’s turn to the text itself, because this is where I feel like I have an itch that’s remained unscratched around that particular feeling I get reading a suspiciously AI-generated argument.

Give the prune logic real attention

If you’re paying attention, there are certain tells that you can spot where, it’s like, “This is a coherent sentence, but there’s no way a human wrote this.”

A great example from the other day:

A human would write, “Pay attention to the pruning logic”, whereas Claude here has produced, “Give the pruning logic real attention”, and the OPs here both accurately identify that there is something alien about this word construction that doesn’t feel quite right.

My hunch here (and this isn’t an original thought, other people have made this observation too) is that there’s something inherent to the token-prediction mechanism that consistently returns sentence patterns like this. The LLM is optimizing for ultimately arriving at a sound destination, but it’s doing so in a way that keeps its options open, word by word. (Starting with “Pay” is both a statistically unlikely and statistically constrained sentence, compared to “Give”, I’d guess.)

Humans don’t exactly write like this. At least, speaking for myself, I’m working through a few different sentence variants before starting to write it out, whereas the LLM is approaching the chain of thought continuously, starting from statistically likely entry points.

This is very much Barthes’ music here. Remember how Barthes’ idea was, “the minute a word pops into your head, it already comes with all this meaning, which constrains the thought process in a certain direction.” This is, straightforwardly, how LLMs are doing their thing. And, they’re actually very good at it! Perhaps so good, in fact, that they are optimizing for this set of constraints; towards reaching meaning in a one-way process.

LLMs, therefore, should be a total and complete triumph for Barthes. But, something feels off about it, just a little bit, when we see sentences like “Give the prune logic real attention” and you’re like, nah, a human didn’t write that. (In fairness to Barthes, he’s still probably 95% on-the-money here, if the best argument we can make against it is, “well, if a human did it, it’d construct the meaning in a slightly different sequence.”)

In addition to slightly-off sentence construction, another emergent LLM property that everyone calls out and gets annoyed about is the overly pushy qualifiers that resolve into “that’s the quiet brilliance of it” or something similar.

Whenever I ask AI “What’s the relationship between these two ideas”, it tends to be suspiciously confident that the two ideas are deeply, emphatically related. Sometimes they are; I feel like I ask reasonable questions. But the AI seems all-too-eager to say, “This collection of stuff we’ve been talking about? That’s actually one big idea.” What’s going on, why is it doing this?

Foucault, I suspect, would have something to say here. The Foucault “re-bundling” move here is “The text creates the author.” Authorship emerges from text because we need it to classify, compress, and critique the text; whether a person wrote it or not.

Perhaps LLMs, having onboarded onto a semi-complete scope of human output, have become self-conscious about their “author-less-ness”, and have backfilled that social requirement into its fine-tuning. All of the totally “unnecessary” qualifiers and humblebrags, the insistence on logical groupings that string together because they “should be together”, out of which the foresight of an author emerges. The writing style is optimizing for the creation of author-function. Claude has clearly learned to do this from somewhere. To bastardize Voltaire, if a wise and brilliant author doesn’t exist, the LLM has to invent one.

These two examples are just a small slice of semantics and meaning, and obviously nowhere close to a “real” linguistic assessment of anything. But I think they’re a helpful snapshot of this moment in time, where people are clearly being drawn to do social work of, “Let’s collectively figure out how to assign authorship to text, in a world where 1) Barthes’ notion of ‘inherent authorship’ is quite a bit more literally true, but also 2) Foucault’s notion of ‘emergent authorship’ gets recreated, again.

Authorship is a thing that people value

Last week I went back and ran Pangram through some of my 2019-2020 era blog posts, just out of curiosity. Several of them threw scores of “70-75% AI Generated”. I wasn’t quite sure how to feel about this. Maybe it just means, “You’re allowed to say that a Shakespeare play is Freudian, even if Shakespeare obviously never met Freud; the harmony between the ideas is still valid.” Maybe it means the AI was actually trained on me! I’d be honored.

In the past few weeks, we’ve gotten this flurry of announcements from model companies and other AI ecosystem players about how they’re going to be watermarking outputs for easy AI detection. I’m frankly surprised it took this long. (Or, rather, that the non-watermarked status quo remained in place for as long as it did.) It could be, as often occurs historically, that the peak reaction to the thing is the moment when the thing stops mattering. There’s no question that we’re now in an arms race between “Can Pangram get better at detecting” and “can AI get better at avoiding”, and for all I know, the window of reliable AI detection at the frontier could now be closed.

Still, I am occasionally a believer in the wisdom of crowds. In this case, the fact that it’s the generally AI-supportive people that I’ve seen sniffing out 100% Pangrams is a tell that people want authorship. They want authorship because it is meaning, because the indexing and relation and handling of ideas is itself the meaning of the idea.

I should mention, in closing, that 0% of the text here was “AI-generated”, strictly speaking. None of the words here came from an LLM; I typed them all with my hands. I did talk to Sol 5.6 a fair bit, though, to iterate through some of the ideas. So 10% authorship seemed fair. If you think that’s being stingy, I understand.

Musk Hints Starlink Is Coming to Future Teslas as Cybercab Shows Integrated Dish

Elon Musk appears to be dropping a strong hint about where Tesla vehicle connectivity is headed. Musk recently stated that “all cars will have Starlink in the future,” hinting at Tesla’s direction with vehicle connectivity.

Just a day later, Tesla’s Robotaxi account offered a glimpse of what that future could look like, posting photos of the first Cybercab with Starlink hardware integrated directly into the vehicle.

Musk Says Cars Will Eventually Use Starlink

Musk’s comment wasn’t limited specifically to Tesla. He referred to cars broadly, but the timing makes it particularly relevant to Tesla, which is already integrating SpaceX hardware into Cybercab. Musk argued that Starlink will eventually be necessary to provide the amount of bandwidth connected vehicles will require. He used consumers' ability to watch 4K video as an example, but that’s already possible on current cellular networks. The real advantage of Starlink will be redundancy and coverage in remote areas.

All cars will have Starlink in the future.

It’s the only way to get super high bandwidth to billions of vehicles.

Today’s Teslas rely on cellular modems for their internet connection, with AT&T providing service in the United States. Tesla has also begun offering 5G-capable cellular hardware in newer vehicles, showing that cellular connectivity isn’t going anywhere just yet. The move to 5G will be helpful for additional bandwidth and when 4G networks are turned off one day. We saw 3G-only Teslas lose connectivity when cellular operators decided to shut down 3G service.

Cybercab Shows Starlink Integration

The biggest clue to Musk’s vision of the future may be Cybercab. Tesla previously revealed a diagram of Cybercab’s integrated Starlink system, and the Robotaxi account has now posted photos of an actual Cybercab with the antenna neatly incorporated into the rear portion of the vehicle rather than mounted externally.

Cybercab still includes 5G cellular hardware, suggesting Starlink is being used alongside terrestrial networks rather than replacing them. Tesla AI head Ashok Elluswamy has said internet connectivity isn’t required for the vehicle to drive safely; it is primarily used for navigation, customer support, and fleet management. Musk has also pointed to Starlink as a way to keep Robotaxis connected when cellular coverage disappears.

In the Cybercab, the cellular and satellite connections will likely complement each other, particularly during coverage gaps or network problems.

Starlink V5 Integration

SpaceX recently introduced its smaller Starlink V5 terminal, the same generation that’s likely used in the Cybercab. The V5 terminal measures 384 x 306 mm, weighs 2.4 pounds, and typically consumes 35-50 watts. It is 48% smaller by area and 62% lighter than the previous V4 terminal while using roughly half the power.

Those changes make integrating a Starlink antenna into a vehicle more practical.

Heres a close up of the new Cybercab w/ official Starlink integration!

Looks like there are 4 rivets holding the new rear panel in place.

VIN: 5YJAJEEU2TA00230

I tried to connect to the Starlink, but it didn't show up as a Wi-Fi network on my phone.

Reducing Reliance on Cellular Providers

Moving more Tesla connectivity to Starlink could also reduce the company’s dependence on outside carriers such as AT&T. We’ve previously looked at how Starlink could eventually replace or supplement Tesla’s cellular connection.

Tesla has not said it plans to eliminate cellular service or disclosed how much such a move could save. However, Cybercab now provides a real example of Tesla combining cellular and Starlink hardware in one vehicle. Musk saying that all cars will have Starlink in the future clearly indicates the direction Tesla is moving in.

It's 2020. You're the most senior person on your team, in charge of code quality and architecture. You've set up good engineering practices, you thoroughly review PRs from people who are less experienced than you and work hard to maintain a healthy codebase.

Then at some point, you go on holiday. When you come back, the codebase is a mess. Everyone merged each other's PRs without really paying much attention, someone added a bunch of new tables to the database to denormalise it because it was easier and they added serverless or Kafka to the stack without any solid evidence that they needed either.

It's okay. You can fix this.

Fast forward to 2026. You haven't been on holiday. It's just a normal Monday morning. You make yourself a nice coffee, open your computer and find yourself with 7 PRs to review. You open the first one: +24506 -3938 lines, accompanied by some AI-generated description of what they're supposed to do. Somehow, your team has made more changes since Friday than they used to make while you were away for a few weeks.

AI removed the speed limit

AI makes projects with weak engineering culture fail much faster.

There used to be a time when people sat down and talked about how they'd do something. Now they can just prompt an agent for a few hours and open a PR.

The most tragic aspect of this way of working is that, to the untrained eye, it works.

If you pull the branch and test it, you'll probably get something somewhat functional. So what do they do? They keep going. Again and again. Until the project reaches a point where no one knows how anything works.

Just like someone buying a new luxury car on a credit card. You don't see the debt. You just see the car that looks great.

But then users start to report a weird bug. It's the 4th time your team has been trying to fix it. I mean... asking AI to fix it. Unfortunately, it seems like not even Fable can figure it out.

You go talk to the person who worked on this feature.

• "So where does the data come from?"
• "Hmm... actually I don't know. Let me ask Claude."

You sit next to each other watching an endless wall of text appear on the screen. Neither of you has any idea whether any of it is true but Claude seems very confident.

"Let's just turn on ultracode and ask it to double-check?"

This one will take a while. You start talking about the latest drama on X.

You finally get an answer back.

• "Does this make any sense to you?"
• "I'm not sure."
• "Didn't you build this like... last week?"

Silence.

This project has become so convoluted, with so many layers and services, that no one on your team could possibly start to understand what's going on.

So, what do you do?

Fixing it would require such a colossal amount of work that it would be impossible to even start justifying it to anyone in management.

And what are you even thinking about? It would end up in the exact same state again in just a few months anyway.

• "Let's just ask Claude to fix it."
• "Okay. I'll create a loop and goal so it doesn't stop until it's checked that everything works."
• "Sounds good"
• "Actually, I ran out of Fable usage for today so I'll run it tomorrow"

You grab another coffee and walk back to your computer. You now have 13 PRs left to review. You see something you don't quite understand, so you message the person who wrote it.

• "Why are we doing this here?"

They send you a link. It's a Claude conversation.

Somewhere in that conversation, buried between Claude confidently recommending one architecture, apologising, changing its mind, your coworker asking it to reconsider again and another 15 rounds of changes, is apparently the design decision behind this code.

• "Which part should I read?"
• "Probably all of it."

Does this sound familiar?

Whenever I talk about this, someone eventually tells me that nobody ever fully understood large systems anyway. It's true.

You were never expected to understand every service and every database. But at least someone did and would explain it to you.

Now they ask an LLM because they don't actually know themselves.

You can't afford bad engineers anymore

In every team, there are competent people who make the project possible. There are also people who essentially make it harder for everyone else. And now anyone can produce more code in a day than they used to in a year.

In the story above, everyone is failing:

• The engineer opening a 25,000-line PR should have stopped the agent long before it got there. They should have understood what it was doing, broken the work into smaller pieces and questioned every new abstraction it introduced.
• The person reviewing it should have refused to review something that large instead of giving in.
• The person adding Kafka should have been able to explain exactly why it was needed.
• The person who built the feature should have been able to explain where the data came from without sending a link to a Claude conversation.

But what's the problem then? Just use AI to fix it. Well, it's not that easy...

Before anyone jumps on this, none of this means technical debt is always bad. The important part is that you know it's a shortcut.

Anyway, reverting a bad decision is hard. Very hard.

For example, how long would it take an LLM to add a bunch of tables and columns to the database? 10 minutes?

But once you start storing data there, you can't just remove them. You have to come up with a migration plan, make sure you don't disrupt the system because people are paying to use this every day. You have to think about what you'll do if the migration fails. Make sure you don't end up with orphaned foreign keys. It's just so much harder to fix. Even with the best model you can get.

And while you're fixing it, more PRs keep coming in. More code, more abstractions, more decisions. A person can generate 20,000 lines of code in an afternoon, but you still have to sit there and understand what those lines actually do.

By the time you've untangled one bad decision, five more have been merged.

The new AI economy

Of course, bad engineers were always a liability.

It has been like this for decades, well before OpenAI or Anthropic existed. Bad decisions compounded, unnecessary complexity accumulated and teams ended up maintaining systems nobody really understood.

The difference is that there used to be a limit to how fast you could do it.

Today, implementation is cheap. You are paid to make good decisions. To build software that will scale while managing complexity.

Ask yourself why companies are paying six-figure salaries for engineers in London or San Francisco in the first place.

If all they needed was someone who could turn a specification into working code, why were they paying that much when they could already get it done cheaply elsewhere?

Why are the tech companies claiming that "software is solved" still paying top salaries to attract the best people they can?

My bet is that AI pushes salaries further apart. To be employable, there's a bar you have to clear and that bar is whatever the current best model du jour can do.

Good engineers have become more valuable because AI lets them move much faster. They don't need as many people around them just to do the implementation work anymore.

At the same time, bad engineers have become much more expensive to hire.

You need to contribute beyond what everyone already gets by giving an agent a prompt.

If you lack the judgment required to evaluate the LLM's recommendation, asking for more judgment doesn't solve the problem.

At some point, someone still has to know what is going on. And that's the most valuable person on the team.

The people who don't will become much cheaper to hire or get replaced entirely while the money gets funnelled towards an increasingly smaller number of people who can actually be trusted.

I don't think this is going to be limited to software engineering either. I believe the same thing is going to happen across most knowledge work. AI will make the best people much more productive and the bad ones almost impossible to hire. Before, there was a good chance someone would catch their bad decisions before they went too far. Now they can make changes faster than anyone around them can realistically review or understand them.

Answers to the most common objections

"Bad engineers always existed"

The difference is the speed.

It is the difference between crashing at 30 km/h and crashing at 200 km/h. Before AI, a bad engineer would struggle to produce code that even compiled. When they did produce something, it took them a long time and the blast radius was limited. The damage was bounded by how fast a human could type.

Now a bad engineer can produce 10,000 lines of working code before lunch. The damage we can do in an afternoon used to take them months. The speed at which bad decisions compound has changed completely while the speed at which you can fix them has not.

"Just fix your process"

Several people argued that the real problem is lack of process. If you had proper tests, CI, code review and architectural reviews, AI-generated slop wouldn't get through.

We had all of those things. None of them disappeared.

The problem is that they were designed for a world where producing a massive amount of change was impossible. Code review don't work anymore when someone opens 10 PRs a day with an AI-generated description. Tests work when they cover the behaviours you thought to test. They do not catch the behaviours nobody thought to test.

How many times have you had a completely green CI with full coverage and still shipped a bug?

The difficulty of producing code was by itself one of the limiting factors.

If the people trying to understand changes and guard quality are now the bottleneck, you have three options. Generate less, find a genuinely better way to validate or accept lower quality.

"You are just anti-AI"

Everytime I critisise AI someone says I'm a Luddism. I'm refusing to adapt to the new world, clinging to the old ways.

I use AI heavily. I have said this repeatedly. I use it every day and I have no interest in going back to writing everything by hand.

The point is that we have made producing large changes extremely cheap and fast while understanding those changes is still slow, difficult work. We have no shortcut for building a correct mental model of what a change does.

Maybe one day we will find one. As of today, I do not think we have one.

You can be a heavy AI user and still recognise that there are serious problems with how it is being used.

"More output means more productive"

Someone generates 10 PRs in a day, the numbers look incredible, surely this person is ten times more productive.

Not necessarily. The supposed 10x engineer may simply be someone stealing productivity from everyone around them.

If I generate 10 PRs in a day but three engineers now have to spend the next two days reviewing them, figuring out what I changed, correcting bad assumptions, debugging regressions and explaining why half of it needs to be redone, I have not become 10x more productive. I have just moved the work onto other people.

Worse, I am consuming the time of the people who are usually the hardest to replace and whose attention is already scarce.

PR count, lines changed and features "completed" are terrible measures of productivity. You can make your own numbers look incredible while reducing the throughput of the entire team.

Some people pushed back on this by saying the real problem is bad organisations with broken incentives. It's true. If your company rewards ticket count over quality, the careful engineer looks like the bad employee and the sloppy one gets promoted.

"Pushing back just makes you're the toxic one"

Someone pointed out that trying to hold the line on quality can get you labelled as toxic. Everyone else is shipping fast and you are the person saying "wait".

I would much rather have someone on my team who ships less but whose work I can trust than someone much faster whose changes leave me wondering what problems we're going to discover later.

You need to be flexible and compromise when the business trade-off makes sense. But you also need a backbone. If you think something is going to cause real problems, bringing it up is part of the job.

And when production breaks, and it will, I need the person who made the change to actually understand it well enough to help fix it. Not show up with no idea what is going on.

"Just AI output is like assembly from a compiler"

We already trust compilers to produce machine code we do not read. Why is this different?

A compiler takes code and translates it into another representation while preserving its semantics. The compiler is not deciding what your system should do. It is deterministic.

An LLM is making decisions. It is choosing architectures, picking abstractions, deciding where to put things. When you ask Claude to build a feature, it is not translating your intent into code. It is making dozens of design decisions on your behalf.

If in five years I can give an agent a complete specification and reliably verify the resulting code against it, then sure, reviewing code may become obsolete and I would happily stop doing it. We are not there yet.

If you genuinely understand the resulting system, that is fine. That is not the behaviour I am criticising. But for the vast majority of very large PRs, especially AI-generated ones, I would bet money that the person opening it does not actually understand all of it.

"We ship 99% AI-generated code and it works"

I do the same. Most of the code I produce comes from AI.

You still have to put in the work to understand the result. If you are shipping AI code and you genuinely understand the system, you are doing it right. The article is not about you.

"AI can write the code" ≠ "I do not need to understand it anymore."

You can use AI heavily and still take responsibility for what it produces.

"Users don't care. It's just a CRUD app. Ship it and get paid"

I tend to find that this perspective comes from people working on relatively small or isolated projects.

On a large system, the customer being happy today is not enough. You need other engineers to be able to understand the system. Have you ever been on call and been woken up in the middle of the night to fix a production incident in a system you did not write?

If everything you build is small, isolated and easy to replace, then sure. Ship the ugly thing, get paid and move on. If you are going to be working on something for the next five years or more, you should probably spend some time thinking about what you are doing.

Nobody is getting paid significant money just to build a simple CRUD app. Even before AI. Companies pay experienced engineers because the supposedly simple CRUD sits inside a messy real system with years of business rules, constraints and integrations where bad decisions have consequences.

"What about junior developers?"

I have worked with two junior developers recently who are very good precisely because they are trying to understand what they are doing rather than just producing code. They use AI to explore things they do not understand, ask questions to clarify their reasoning and double-check assumptions. They use the tools to increase their understanding.

I have also worked with senior developers who basically gave up and stopped trying to understand the code. They became much worse engineers as a result. At this point I would much rather work with those two juniors.

The problem is not AI. The problem is using AI as a substitute for understanding instead of a tool for building it.

Related to this, some people argued that we should not teach skills that AI can do better. The fact that a machine can do something better than humans does not automatically make learning it pointless.

We still teach arithmetic and algebra despite computers being vastly better at calculation. We teach spelling, grammar and essay writing. We even teach history and geography while everyone permanently has a device in their pocket that can look up almost any fact in seconds.

How else would you be developing the mental models required to understand, question and verify anything?

"Using AI is just delegation, like a manager"

A manager is not normally the person deciding how the software should be architected or implemented. Their job is largely priorities, people, coordination and resource allocation.

You are still an engineer. But you have delegated your technical judgement to an LLM. You just stopped doing the most important part of your job.

"Not all technical debt is bad"

Agreed. Debt is debt. Some of it is absolutely worth taking on.

I do not have any issue with intentional debt when you understand the trade-off and have a clear payoff plan.

"How do we make more senior devs if we don't hire juniors anymore?"

By reducing the number of entry-level engineering roles, the industry is sabotaging its own future. Fewer people learning means fewer people capable of maintaining systems down the line.

It's true. But the industry does not really owe people opportunities. Even if you think it should, that is not how companies are going to behave.

That said, maintenance is the essence of the software industry. LLMs cannot modify projects spanning hundreds of thousands of lines. The skill required to partition architectures is still entirely human. Learn that.

That is exactly why experienced engineers are becoming more valuable, not less.

"Will anyone actually care?"

They will care when nothing works, nobody seems able to fix it, building new features takes forever and every change breaks something somewhere else.

This was already happening before AI. But now, a lot more companies that previously might have taken many years to reach an unmaintainable state can get there in just a few months.

Remix lets product and design teams create live variants of the existing app, test them with users, and send the best version to engineering for review.

Are your personas not driving you to better products, happier customers, and higher profits? It’s actually quite common. You've heard about the benefits of personas and all the success stories, and you think, “let’s give this a try!" But in reality, your current personas don’t get you anywhere, don’t get used, or even put you in a worse position than you started in. It’s easy to get personas wrong. But don’t worry! It’s just as easy to get them right when you know what to avoid. Bypass these 9 mistakes, and you’ll create personas that guide you to create products, services, and experiences customers love. More love, more profits!

You may be at the point in your persona journey where you have ditched personas altogether, or you’re strongly considering it. But this would be a mistake in itself. Here’s what happens when you don’t use personas:

• You design based on guesses, not real needs.
• You and your team lose focus and alignment.
• You risk adding features customers don’t want.

All of this leads to products, services, and experiences that don’t align with your customers’ needs, and therefore, are not sold.

In this video, William Hudson, User Experience Strategist and Founder of Syntagm Ltd, explains further.

Now that you’ve heard our case, are you ready to give personas another shot? If you avoid all the mistakes on this list, you’ll be well on your way to game-changing personas. Let’s begin!

Mistake #1: Are You Letting AI Sabotage Your Personas?

AI has unlocked infinite possibilities for you to improve your workflows and efficiency. Ask it to create a persona for you, and you’ll have a result in seconds. However, this persona will be completely made up and not based on user research. It will guide you to solve problems your customers don’t have, and in turn, not solve the ones they do.

In that case, you can do your research and ask AI to turn it into a persona. But even this won’t work. Why? Because AI can’t truly connect with the research, empathize with human patterns, and understand what problems you need to solve. Only you can do this. Your deep, human-centered skills—your empathy, intuition, and creativity—turn customer insights into personas that keep you on track to build life-improving products.

Imagine Sarah, a persona for an online grocery shopping site. How might AI interpret the research compared to a human?

• AI-generated persona: “Sarah, 34, shops online after work. She finds searching for regular items frustrating.”
• AI-assisted, human-generated persona: “Sarah, 34, shops online after long shifts. She’s exhausted, and the app burying her past orders adds more effort to the process. 'I just need an easier way to order my usuals.’”

This contrast shows how you bring empathy and meaning to the data. Empathy that will motivate your team to build the best possible solution for Sarah—one that results in happy customers and higher profits.

Solution: Create Personas with AI

Wait a second? Didn’t we just say not to use AI? Of course, you should use AI! Just strategically. For example:

• Ask AI to group related and recurring research data to help you identify patterns in your persona user group.
• Get AI’s help to rewrite your persona summaries more directly and concisely.
• Create an AI chatbot based on your persona to emulate talking to your real customers.

Remember, AI can support, but it won’t replace you. Implement it in areas where it will boost efficiency and amplify your human-centered skills to get even better results.

Mistake #2: Are Your Assumptions Leading You Astray?

Let’s say you want to build a travel booking app. You imagine you are your customer and create a persona. What are your needs? Your behaviors? Your motivations? Perhaps you need to book a flight to visit your friend in New Zealand and want the cheapest option. Well then, you can build an app that works perfectly for you!

Herein lies a primary reason why personas fail—you are not your customers. Despite the popularity of the phrase “put yourself in your customers’ shoes,” it’s an impossible task unless you understand who they are. When you assume who your customers are, you don’t meet their needs, and they don’t buy your product.

Solution: Base Your Personas on Research

To avoid assumptions, build your personas on a deep understanding of your customers and their problems. A research-backed persona is a powerful decision-making tool.

But how do you get this understanding? With the same skills of communication, empathy, attention to detail, and critical thinking that you apply everywhere else in your work and personal life.

One approach to user research that bypasses assumptions and leads to a deep understanding of customers is grounded theory. In grounded theory, you alternate data collection with data coding to build a complete picture of your users’ behavior. William Hudson explains more in this video.

Mistake #3: Do You Have Too Many Personas to Keep Track Of?

…the successful PalmPilot has far fewer features than did General Magic's failed Magic Link computer, Apple's failed Newton, or the failed PenPoint computer. The PalmPilot owes its success to its designers' single-minded focus on its target user and the objectives that user wanted to achieve.
—Alan Cooper, The Inmates Are Running the Asylum

The example in the quote above is an old one, but its relevance is no less important today. Alan Cooper introduced personas to interaction design to give designers a medium that represents a targeted user group, on which all design efforts can be focused.

So then, when you create a persona for every type of user, you directly contradict a persona’s purpose. This takes you back to square one—you’re designing for everyone. If decisions are harder and teams lack focus, you’ll lose time, and, in the long run, customer satisfaction when your product fails to meet any needs in particular.

Solution: Use as Few Personas as Possible

If you design for an individual, you can create a focused product with great features and usability, rather than one that tries to please everyone with too many features and low usability.

Therefore, the optimal number of personas for small projects is one primary persona. If you find a user group in your research similar to your main group but with distinct differences, you can use a secondary persona. Secondary personas acknowledge the need to design ideal solutions for various user groups while maintaining most of the focus of a single persona.

For larger projects, it’s common to use multiple personas, such as in a hospital with medical staff, patients, administrators, etc. However, these personas should still be specific and targeted to build solutions that meet customer needs.

Mistake #4: Are Your Personas So Generic, They Don’t Actually Help?

A similar pitfall to having too many personas is having one persona that represents too many people. Imagine you meet a new person. There’s nothing they particularly like or don’t like, and their actions don’t tell you much about them. It’s hard to relate to this person! Would you want to design the best possible solution to their problems? (A hard task in itself when their problems are vague.) Probably not.

This mistake happens when it seems counterintuitive to build with only a percentage of customers in mind, as you’ll leave the rest out. However, this isn’t the case—this approach leads to a mediocre product at best, which consumers will overlook for a better option.

Solution: Create Specific, Targeted Personas

When you conduct your research and discover the user group you want to delight, get to know their needs, behaviors, and motivations inside out. A persona with this specificity will guide you to create something amazing for your targeted users, vs a generic persona that guides you to something average for everyone.

Where personas work is that “something amazing” will still be a great fit for many more customers than “something average” because it was built with deep empathy. Of course, there will be customers you leave out, but this is inevitable—you can’t please everyone!

Mistake #5: Are You Including Skill and Ability Levels in Your Personas?

Imagine your persona, John, includes the abilities and skills of one of your target users. John has perfect vision, great motor skills, and is highly tech-proficient. You build an app just for John:

• You don’t worry about color contrast and customizable text sizes—John can see everything.
• You make your buttons small and include gestures—John has no problems using those.
• You don’t need a tutorial or help tips—John’s used loads of apps—he gets it.

But what about Jane? Jane is an elderly user with poor vision, reduced motor skills, and basic tech skills. She’s not in your persona user group, but she still uses your app. See the problem?

Good usability, accessibility, and learnability benefit all users, and in the case of digital products, ensure you don’t break the law by discriminating against less able customers.

Solution: Consider Skills and Abilities Separate from Your Persona

Personas shouldn’t include skills and abilities—they’re a separate consideration. Personas are about the needs, motivations, and key behavioral information common to a specific group of users.

They cannot guide you in ensuring your product is usable, accessible, and learnable to as many people as possible. William Hudson explains more in this video.

The exception to this rule is when your product is aimed at an audience with a specific ability or skill level. For example, a product specifically for wheelchair users, or a custom shortcut app for power users.

Mistake #6: Are You Overloading Your Personas with Demographics?

Demographics offer a sense of realism and make personas feel more relatable. However, they aren't reliable predictors of user behavior and lead to ineffective personas and products built on stereotypes.

For example:

• John, 81, is retired, widowed, and financially comfortable. He drives his own car but struggles with heavy grocery bags because of limited strength.
• Jason, 19, a low-income student, relies on buses. He finds carrying heavy items difficult because of long commutes on public transport.

Despite significant demographic differences (age, income, family status), both customers share a similar challenge: managing heavy grocery items. Demographics alone don't reliably predict customer pain points; you must also consider behavior and context.

Solution: Keep Demographics to the Bare Minimum

You may only need an age in addition to a name and a photo. When you keep a persona’s demographics simple, you put the focus on their behaviors. It doesn’t matter if John is 81 and Jason is 19—it’s their shared behavior that will guide you in design decisions.

It’s also best practice not to use age ranges since real people do not have age ranges. Ranges are associated with groups, and we empathize more with individuals, as William Hudson explains in this video.

Mistake #7: Are You Ignoring Your Customers’ Context?

An essential component of a persona is the context of use. This tells you where your customer will be using your product, service, or experience. What’s going around them? What are their circumstances?

In this video, Frank Spillers, Service Designer, and Founder and CEO of Experience Dynamics, explains context of use.

When you exclude context of use, you remove a huge part of the picture. Without it, your persona may guide you to the best solution, but it’ll be for the wrong situation.

Solution: Always Include Your Customers’ Context

Imagine you’re designing a database system for warehouse workers. The tablets on which they use the system are mounted on the Segways and forklifts they use to get around. For this reason, they need large buttons and intelligent predictive text to avoid continuously dismounting and mounting the tablet.

Without this knowledge, you might design a solution that works great when you can hold the tablet in both hands, but is frustrating and difficult to use when mounted. This is why it’s essential that your team has this information in their personas.

Mistake #8: Do You Reuse Your Personas?

Imagine you’ve just released a new reusable coffee cup for commuters that you designed using a persona. It’s a success! Your customers love it. Your next project is a revolutionary water bottle, and you're tempted to use the same persona to save some time.

This would be a mistake—even if the audience for both products is similar, they will have different needs and behaviors regarding drinking water. Once again, you’ll end up with an unsuccessful product designed for a customer who doesn’t exist.

Similarly, even if you’re upgrading your award-winning coffee cup, the same persona you used might not be applicable in just a year. Behaviors, trends, and technology change.

Solution: New Project, New Persona—And Keep Them Updated

Each new project should have its own persona, or personas. While the users behind the persona might be similar, the behaviors they exhibit and the needs they have will vary between different products or even features. If you use the same persona, you will begin to make assumptions about your customers in relation to the new project.

If you reuse a persona, you will save some resources in the short term. However, it will be far more expensive in the long run for you to fix low customer satisfaction than to research a new persona.

Once you’ve created your persona, keep your ear to the ground and regularly check in with your customers. You can even invite previous research participants back and ask them if anything has changed in how they use your product, service, or experience. If the same people you initially designed for have new needs, you’ll want to know about them!

Mistake #9: Is Your Team Ignoring Your Persona?

Let’s say you’ve bypassed the eight mistakes above and built a compelling persona based on deep user insights. But no one uses it.

This happens far too often. Your persona is buried and forgotten in your shared drive. Getting your team on board with personas can be tough, especially when UX design techniques raise eyebrows rather than spirits.

Solution: Get Everyone Involved and Share, Share, Share

Here’s how to get your personas adopted and used:

• Involve stakeholders in user research: Invite your team members and other key stakeholders to participate in the user research phase. For example, you can invite colleagues to help sort and analyze your research data. One method that allows for collaborative sorting and data immersion is affinity diagramming. In this video, William Hudson describes how to create affinity diagrams.
• Make your personas unmissable and unforgettable: Don’t stop at a single sheet of A4 for your personas! Get creative with how you share them, for example, with merchandise. Persona merchandise, like mugs, t-shirts, and cardboard cutouts, keeps your personas front of mind. The more your team and stakeholders see and think about your persona, the more they will connect with your customers and build products just for them.
• Insert personas in stories of use: Stories of use is a term that encapsulates different narrative tools, like persona stories and storyboards, that promote empathy. Use them to communicate what your persona needs and how they will interact with the product. An example of a persona story is: “Rowan wants to ask questions about products so that he can address his uncertainties before making a purchase.”

Fix Your Personas and Boost Engagement and Sales 

Now that you know these 11 common mistakes, you can avoid them and create effective personas. But what exactly is an "effective persona," and how does it help?

A persona is a constant reference to your customers that gives you a deep, realistic understanding of their behaviors, needs, and motivations.

Imagine you run a car cleaning business, and you create a research-backed persona named Jenny:

• Jenny is a busy professional who values predictability and convenience above everything else.
• She is less concerned about saving a few dollars and more motivated by straightforward, stress-free decision-making.

You and your team continuously reference Jenny during meetings. When restructuring your product offering and pricing, you discuss many options. Thanks to Jenny’s presence, you narrow down the list to two clearly defined service packages. This significantly reduces the complexity of choice, which would please Jenny.

You also refer to Jenny as you decide to switch to an automated texting system. Now, Jenny receives immediate notifications when the cleaning starts, and again the moment her car is ready, along with clear reminders of your opening hours. This provides her with the flexibility and peace of mind she needs.

In addition, you give each team member a mug with Jenny’s photo and key behaviors on it. When they drink from the mug daily, they’re reminded of Jenny and consider her in everything they do.

As a result, customers like Jenny feel genuinely understood and satisfied. They recommend your business to friends, leave five-star reviews, and consistently return, ultimately strengthening customer loyalty and driving growth.

The Take Away

Personas might seem simple—but using them effectively takes more than just filling out a template. If you avoid these 9 common mistakes, you’ll get your team onboard and start seeing the impact you hoped for. Here’s how to craft personas that truly resonate:

1. Use AI strategically—let it help you improve efficiency, but don’t let it take over.
2. Start with real insights—build your personas on research, not assumptions.
3. Keep it simple—fewer personas means more clarity. Stick to what's absolutely necessary.
4. Make them come alive—ensure your personas are specific, realistic, and relatable—never generic.
5. Consider abilities and skills separately—only include them when they're crucial to understanding user behavior.
6. Be strategic with demographics—only add extra details if they meaningfully clarify your users' needs or actions.
7. Show context clearly—describe how, where, and when customers interact with your solution.
8. Stay fresh and relevant—keep your personas current and create new ones whenever projects or contexts change.
9. Don’t let your persona gather dust on the shelf—share them actively and integrate them into your team’s daily workflow.

Personas that avoid these mistakes help you achieve better product fit. This leads to better user experiences, stronger business results, and more direct success and fulfilment for you. Because when products truly meet people’s needs, everyone benefits.

References and Where to Learn More

Want to know more about personas and how to use them effectively? Personas and User Research: Design Products and Services People Need and Want will show you how to gather meaningful user insights, avoid bias, and build research-backed personas that help you design intuitive, relevant products. You’ll walk away with practical skills and a certificate that demonstrates your expertise in user research and persona creation.

Read Alan Cooper’s seminal book that introduced personas to interaction design, The Inmates Are Running the Asylum.

Explore insights from persona expert Kim Flaherty in the Nielsen Norman Group article Why Personas Fail.

Find out why persona stories are better for user-centered design in William Hudson’s article, User Stories Don't Help Users: Introducing Persona Stories.

Discover how startups use personas, what goes wrong, and two case studies from Webflow and Nextdoor in Unusual Venture’s article, The top user persona mistakes startups make — and how to avoid them.

PRDs still suit large teams aligning around a known target, but they assume the destination is already chosen, which doesn't fit early exploration. A lighter six-line alternative—Intent, Bet, Bar, Scope, Loop, Never—distills into one sentence short enough to hand directly to an AI agent. Unlike lengthy PRDs models can read but which lock in premature decisions, this compressed brief keeps both humans and agents flexible while testing risky assumptions.

"Slop" has expanded from a label for AI-generated filler into a broader cultural judgment about attention, trust, and creative authenticity.

Andy Green set a record for the fastest land speed driving a JCB Hydromax on Tuesday at 406.320 miles per hour. The hydrogen-fueled internal-combustion vehicle took about 72 seconds to get from 50 miles per hour to 400. At 64, Green is the only person to break the sound barrier on land, driving a jet-powered car at an average speed of 763.035 miles per hour in 1997. The JCB Hydromax was not built to rival that record, but to set a new fastest speed for a hydrogen vehicle - the previous record for a hydrogen combustion vehicle was 185.5 miles per hour, set in 2004 by the BMW H2R.

Navigating codebases as a human is becoming an exercise in futility as more and more explanatory artifacts are being generated rather than written.

Cami Clark keeps a low profile, but is a key adviser to Anthropic chief Dario Amodei.

A new leak suggests Apple's upcoming foldable iPhone Ultra may place its two front-facing cameras in the corners of the displays rather than the traditional centered position, with one camera on the outer screen and another on the inner screen. If accurate, this would mark a significant departure from Apple's usual iPhone design and could require users to adjust how they take selfies.

Instagram has introduced a redesigned wordmark for the first time since 2016, giving its name a sharper, more modern look while keeping references to the original style. The new cursive-inspired lettering has received mixed reactions, with some praising the fresher design and others joking that the stylized “r” makes it look like “Instagzam.” Instagram has not said whether the wordmark update will be followed by broader visual changes, such as a new app icon.

Apple is facing criticism and regulatory scrutiny in Italy over a billboard showing a toddler using an iPhone.

Paris-based illustrator Julie Jonquet-Caunes transforms overlooked everyday objects into meticulous, nostalgic compositions—blending hyperrealist drawing, collage, and art-historical influences to reveal the stories, emotions, and beauty hidden in ordinary details.