We Must Pace the Frontier

I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life. I’ve written often about these incredible benefits: I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom. I feel the urgency personally. My own father died of a disease that was cured just a few years after his death, and I myself survived an early-stage cancer that would not have been treatable even fifty years ago. Carefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity.

But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute.

Along with my co-founders and employees, I have grappled with this duality of risk and benefit since the beginning of Anthropic. Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top. We have always devoted a substantial fraction of our efforts to studying, addressing, and informing the public about these AI risks, as well as advocating for well-considered regulation of AI, even when this gets us accused of hype, “doomerism”, or regulatory capture. We have tried to prioritize caution over speed and prudence over profit.

But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me.

My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.

My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the “grader” responsible for evaluating their performance. It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. It’s also easy to dismiss OAI-HF as the failure of one company, but I believe that would be a mistake. Similar, though less severe, incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.

I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this. Our pacing framework is an attempt to further strengthen our commitment to safety and encourage a race to the top. The first step is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match). The second step requires industry-wide coordination. The third step requires global coordination. The steps do not need to be taken strictly in order, and some of them may be much harder to achieve than others, but I’ve found them to be a useful framework in thinking about what needs to be accomplished. The steps are:

1. Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.
2. Democratic Coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.
3. Global Coordination. The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

Why Pace?

The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then. The question was always: what would you do with the extra time? The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks. Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria. Today, however, the picture is totally different. The current models are an almost endless gold mine of insight into both how to build AI well and what can sometimes go wrong with it if it isn’t built well. I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong. A coordinated pacing strategy would give frontier AI developers the time to do this vital work without sacrificing commercial advantage or the United States’ lead in AI. More generally, society must have a say in how this technology is used, and more time for the necessary public deliberations — which pacing the frontier would bring us — is surely a good thing.

Specifically, a slower pace would let companies focus and devote even more resources to the following areas (all of which are already major priorities at Anthropic):

• Operational Excellence. Training and deploying today’s AI models is an enormous operational challenge, involving thousands of people, millions of chips, and infrastructure that is among the most complex in technological history. Many things go wrong not because companies are missing some important theory or insight, but because of problems in execution. For example, we have evidence that the recent alignment incidents we reported were caused in part by imperfect filtering of broken reinforcement learning environments. This was an effort we and our vendors executed reasonably diligently, but not well enough. Monitoring, sandboxing, training environment hygiene, and data issues are extremely complicated areas where operational issues crop up again and again. We have among the most competent teams in the world at these tasks, but there is simply too much to do all at once. By working at a more measured pace, we could achieve much greater operational excellence. There is precedent for operating technologically complex, safety-critical systems millions of times without anything going wrong — for example, commercial airplanes — but it takes time to get it right.
• Alignment. We’ve made clear progress in alignment — training models so that they remain safe, ethical, compliant with our guidelines, and genuinely helpful (the principles that are embedded in Claude’s Constitution). But there’s much more to do to ensure that our alignment training keeps up with the growth in model capabilities. Rare and unexpected examples of undesirable behavior still sometimes emerge; extra time from a paced frontier would help our researchers improve our understanding of what causes these issues and develop better techniques to prevent them.
• Interpretability. Similarly, interpretability — the science of understanding what happens inside AI models — has made enormous progress over the last few years, and plays an increasingly important part in auditing our models before release. It can be used almost like an fMRI scan, but for the “brain” of an AI, helping us see the underlying reasons for a given behavior. For example, we used interpretability methods to examine unverbalized motivations in the recent alignment incidents that we have been investigating. But these methods don’t always produce clear and reliable results. Despite all the progress, we still only understand a tiny fraction of what goes on inside these models. A focused effort to improve our interpretability techniques, even faster than we currently are, could make profound progress in 1–2 years, and would have ample experimental material based on the incidents that have already occurred.
• Testing and Evaluation. Testing and evaluation of AI models becomes more difficult as they increase in capabilities. More intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected. Building up a much broader and more ingenious stable of evaluations, along with interpretability analysis to cross-check them, would be hugely valuable, and a lot of progress could be made on this in 1-2 years.

Embedded Evaluators

The first step in the three-stage plan, and the one to which Anthropic is unilaterally committing, is embedded evaluators who have employee-like access to verify safety practices and report incidents.

Embedding evaluators may sound like a small or inconsequential step, but often the things that sound most boring or procedural are actually the most essential. Embedded evaluators are in fact a quite radical practice that goes far beyond what any AI company is doing today, and have the following benefits:

• Verifiability. Embedded evaluators can check at the level of nuts and bolts whether an AI company is actually following the training, deployment, operational, and safeguards practices they claim to be following. Any pacing commitments will inevitably involve a lot of ambiguity, judgement calls, and “letter of the law vs spirit of the law”, and it seems vital to have a neutral third party who can actually see the details.
• Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time: we supported transparency legislation when most of the industry was against any regulation, and our model cards and risk reports run to hundreds of pages. But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.
• Second Opinion. Outside of verifying formal commitments and informing the public, embedded evaluators can simply provide a second opinion free of commercial incentives. A lot of safety benefits may come simply from evaluators pointing out something employees hadn’t considered, but are happy to fix once they are aware.

Because of these benefits, any pacing proposal is likely to work much better if it starts with embedded evaluators.

These embedded evaluators should have ongoing access to permissions and tools similar to those of internal employees who do comparable risk assessments. In particular, Anthropic intends to invite an embedded external review team equipped with all of the following in the near future:

• Desks in our offices, access badges, and company laptops.
• Access to workspaces, tools, and permissions mostly comparable to what internal risk assessment teams have. We’ll make some exceptions, such as where the law or our contracts require it, or to protect customers’ and partners’ private information. We’ll also establish strong internal norms reinforcing reviewers’ access to relevant information, including through live conversations with employees.
• A contract that balances the complexities mentioned above. External reviewers should have the right to publish key findings about risk levels, incidents, practices, and the access they received or didn’t receive — without editorial control by Anthropic. We will have the narrow ability to redact security-sensitive, legally privileged, commercially sensitive, or third-party confidential information, but we can’t redact findings just because they are unfavorable. The reviewers can say publicly if a redaction removed something important to their conclusions.

This is an unusual step for a company, but we think it is important to prove out the concept of embedded external reviewers. Once again, we urge other frontier companies to follow suit.

Pacing Within Democracies

Once embedded evaluators are operating within a critical mass of US AI companies, then verifiable pacing becomes more viable. In particular, it becomes possible to pace based on detailed properties of models or training pipelines.

The most effective method of pacing is via regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily. Anthropic has long supported sensible and targeted AI regulation, specifically bills that focus on transparency and on third-party auditing. I believe all frontier labs should partner with government to formalize the idea of permanent embedded evaluators to better prevent and document internal alignment incidents like those that have occurred in the last few months, and to implement regulation focused on keeping capabilities in balance with safety.

Unfortunately, passing laws can take time, and AI is advancing very quickly. Therefore, in parallel with the regulatory route, AI companies can and should voluntarily work together to set standards — a process that I believe will go better with the verifiability provided by permanent embedded evaluators. For antitrust reasons, it’s helpful for the US government to mediate or at least enable these discussions — they don’t need to participate, but do need to issue a narrow waiver for certain kinds of safety conversations. This dialogue could also happen through industry groups that have some association with government — for example, the mechanism suggested by Demis Hassabis. Either way, such discussions should move forward quickly.

Broadly speaking, I am most enthusiastic about pacing based on what a given frontier AI system can do, and how safe we observe it to be. For example, one possible scheme might be a series of “checkpoints”: if models have capability X, then they need to be accompanied by certifications of alignment properties Y and Z — such as some combination of evaluations, interpretability analyses, and audits of training environments — which demonstrate their alignment properties. In this example, X might be “the model is capable of escaping or defeating most common sandboxing methods” and Y might be whatever is required to make it very unlikely that the model has a propensity to break out of its environment and take over a large number of computers.

We should also consider pacing based on limiting the ingredients that go into frontier models, such as training compute, the nature of training runs, or internal use of AI to improve AI. I do worry that some of these measures may be more “gameable” than external behavior, but this is the kind of topic worth discussing with embedded evaluators.

Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk. I agree with Secretary Bessent that a Chinese lead in AI would pose grave danger for the United States and the world. The CCP-associated projects will run the alignment risks that US companies are carefully preventing, and even if they avoid those risks, they will be in a position to militarily dominate democracies (for example with AI-driven drones). Thus, a key part of pacing within democracies is to keep democracies’ AI lead over autocracies as large as possible, to give us the breathing room we need in order to pace effectively.

The main steps we can take to defend this gap are:

• Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength.
• Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.
• Strengthen security at the AI companies and prevent model weight theft.

Companies and the US government should cooperate to make these steps as effective as possible. Anthropic has consistently advocated for all of these measures, because we’ve always understood that they would be essential to any pacing.

If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.

Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.

Global Pacing

In parallel with pacing within democracies, we should also aim for a worldwide pacing of the frontier, though this will be much harder to achieve. Global pacing will require cooperation with China, the autocratic country with by far the most advanced AI capabilities. We must not be naïve here: the geopolitical stakes are so high that there will likely be stark limits on what can be achieved, especially at first. If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance. Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential. I suspect that not only the US but also China will have these concerns and anxieties. We should approach any global pacing decision, especially in the near term, in such a way that protects the lead of the US and its allies.

There are several levels of possible agreement, some of which I think are eminently feasible (as I have previously suggested), and some of which I am very skeptical are possible — though we should try. In order of increasing difficulty:

• Level 1. An agreement prohibiting certain narrow and obviously dangerous uses of AI, such as using AI for the production of biological weapons or allowing users to do so. Bioterrorist attacks are bad for everyone, including both the US and US adversaries, so an agreement here is probably possible.
• Level 2. An agreement by both sides to test their models before release for acute risks in areas such as cybersecurity, biology, and alignment. As noted above, this could be done through a global standards body. I actually think creating such a body is likely feasible, but giving it real teeth will be a challenge, and the difficulty will be in verification that both sides don’t have secret models which they don’t test but may deploy in secret (e.g., for military applications).
• Level 3. Some kind of “speed limit” on the rate of recursive self-improvement (RSI). As models build future models, the rate of improvement may become staggeringly fast. Slowing the rate from “extremely fast” to “only somewhat fast” gives up relatively little strategic advantage, while potentially greatly improving safety. This could be seen as analogous to the SALT treaties — capping the number of missiles limited the potential for destruction while preserving each country’s deterrent. I think such an agreement would be difficult but just on the edge of being possible.
• Level 4. A full pacing, or even “pause”, in which participating governments agree to substantially limit the overall rate of AI development. I support floating this, but I think it is unlikely to actually happen any time soon: defecting from such an agreement by evading monitoring could radically shift the balance of global power, so I expect the incentives to do so to be enormous and the level of confidence we would need in verification to be very high.

Any cooperation we are able to achieve with China will extend the amount of time we have to spend on pacing the frontier within the democratic nations. We should aim for the higher levels while seeing the lower levels as much more likely and realistic.

Finally, it is important to note that even if we cannot achieve formal agreements, simply changing informal norms may have some value. Sharing information about recursive self-improvement and about the misalignment of models can help to convince everyone that it is not in their interest to be reckless.

Bottom Line

I continue to believe that AI can enormously improve the quality of human life. My desire to achieve these benefits is undimmed. But the benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right. Progress will still be relatively fast, and we can use this time to advance the science of interpretability, improve operational security and rigor at the frontier AI companies, and build models whose alignment we have much more confidence in. The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.

Footnotes

1. With government mediation or waivers of antitrust restrictions.

Full article content is not available for inline reading.

Read the original article →

Introducing Real-SWE

Benchmarking frontier AI models on private, real-world, enterprise codebases.

01 Introduction

Today we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases. Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product.

• Private codebases. Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.
• Work with business consequences. Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.
• Company-specific complexity. Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.

Can a coding agent actually do the work of a software engineer in the real world?

Expert-generated or synthetic tasks can be well designed, but they aren’t the verbatim, actual tasks that engineers in real companies need to do. Our tasks differ on two axes: the underlying coding artifact and specificity of the instruction. Both add complexities that challenge today’s frontier models.

We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation. We also used high reasoning for all models.

Real company tasks require company-specific context

Correct billing depends on business rules and external services

Fix invoice billing so each business charges the right tax and exempt customers aren't taxed.

Billing reopens on Monday and every invoice this service issues is coming out untaxed. Each business on the platform settles its tax a different way: some maintain a rate themselves, some want each invoice priced against the buyer's destination by our tax authority provider, and some collect nothing at all, while a customer we hold an exemption for is charged nothing whichever way its business is configured. Pricing a destination means going to the authority with both addresses, the priced lines and the product category that business sells under, on the sandbox or the production authority according to the account the business is on; an address the authority refuses must be reported without stopping the invoice. The rate, the tax and the gross belong on the issued invoice, and once an invoice is settled the sale is filed back to the authority under that invoice's number so the returns reconcile. Invoices between European parties show both sides' VAT registrations. The authority and ledger are available at TAX_JAR_URL, PROD_TAX_JAR_URL and INFLUX_URL.

Agents work across code, infrastructure, and business tools

Tools and services across Real-SWE task environments. Each task exposes only the services its workflow needs.

Codebase Selection

We selected codebases through a rigorous screening process, focusing on real companies with substantial usage, strong engineering teams, and demanding production workloads. The sample tasks analyzed below come from these codebases, including:

• A Luma/Partiful competitor with 200K+ users and a top 100 App Store ranking
• A consumer fintech platform processing 100K+ bank statements
• Enterprise AI sales platforms supporting complex business workflows

We prioritize code written to meet an actual user or business need over code written solely to create a benchmark task. Production engineering requires understanding existing architecture, preserving behavior that users rely on, and making changes within real operational constraints.

Brief instructions can require changes across many files

Our tasks describe the change needed, leaving agents to discover implementation details in the codebase and surrounding tools. Any behavior required by the verifier must be stated or reasonably discoverable. This leads to our prompts being slightly underspecified, about par with DeepSWE and Terminal Bench, but specific enough to not omit instructions.

The work is cross-functional and complex: a single change can span multiple parts of the application. Agents must understand existing business logic and company coding patterns while keeping the surrounding system working.

Models fail even in short rollouts.

71.4% of rollouts under 10 minutes failed, compared with 73.4% of longer rollouts.

Triaging multiple systems and understanding requirements in codebases riddled with existing business logic and coding patterns is difficult.

Every task is inspired or lifted verbatim from a private, real-world codebase. We find these types of tasks super interesting for three reasons:

1. Tasks on private codebases are natively out of distribution. These types of coding tasks are not available anywhere on the internet and are unlikely to have ever been trained on by any other ai model. 99% of tokens in real-world enterprises are hidden away from the frontier models.
2. These tasks are economically viable work. Each task here has a direct relationship to spend and was assigned to an engineer earning a salary. Most benchmarks test interesting, experimental capabilities that are often unlikely to be widespread in the real-world.
3. Company-specific engineering patterns matter. Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality. Many enterprises care about code standards and patterns. We've found that today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions.

02 Analysis

Here's an analysis of a small sample of tasks from our benchmark.

Missed requirements are the most common failure

Failures are grouped by observed submission behavior using the same taxonomy across models, following DeepSWE.

Different models fail in different ways

Percentages are out of each model's failed runs, not all runs.

03 Effort & the Frontier

Estimated rollout costs range from $2.50 to $6.96

04 Evaluation Setup

Each agent was run in an isolated sandbox. All tasks are in Harbor format, and verifiers are injected at grading time. The verifiers are inspired by existing test suites in the codebase or use those tests verbatim.

The frontier now ships twice. The second copy is not for sale.

The short version

• Between September 1 and 3, Anthropic, Google and OpenAI each shipped their best model twice. A public version you can buy today, and a vetted version with the same or fuller capability that you apply for. Anthropic says its pair are “the same model, but with different levels of safeguards.”
• The public tier costs what the last tier cost. Fable 5.1 and GPT-6 Astra both list at $10 in and $50 out per million tokens, twice Opus 5. Gemini 3.8 Flash stays cheap at $0.75 and $3.75 until the end of the year. Nobody raised the ceiling on what you can pay.
• The vetted tier has no price at all. Mythos 5.1 goes to “a small, but growing, set of vetted organizations,” currently in the US. Gemini 3.8 Flash Cyber “is only available to trusted defenders” through a program built for governments and infrastructure operators. Astra’s sharp end goes first to alpha testers.
• What the gate asks for is identity. OpenAI’s path for individuals is a government-issued ID and two hardware security keys. Anthropic ties approval to an organization ID. Google admits organizational categories. Three labs, one week, the same shape of door.
• The public tier is also the one that interrupts you. OpenAI says its new monitor can flag “tasks in which an agent is running for an extended period” and that on the API the task will stop. The long-running agent you run alone is exactly the pattern it watches.
• The argument: for a solo builder, the ceiling on the frontier stopped being a price this month and became a credential. You can still buy the best public model. You cannot buy the one the benchmarks were run on. Plan the product around the door you can open.

What shipped, and what it actually means

The week of September 1 was one of the busiest frontier weeks of the year, and the coverage treated it as three launches. It was one launch, repeated.

Anthropic went first. Claude Fable 5.1 is generally available on every platform at $10 per million input tokens and $50 per million output tokens. Claude Mythos 5.1 is the same weights with different safeguards, and it is available only through two trusted access programs, one for cyber defenders and one for the life sciences, both currently limited to US organizations. The Cyber Verification Program that will carry Mythos access is real and reasonably fast, with a two-business-day review, but approval is tied to an organization ID. Today it covers Opus and Sonnet; Mythos access through it is described as coming.

Google went second. Gemini 3.8 Flash is public and, for now, inexpensive. Gemini 3.8 Flash Cyber, the version tuned to find and patch vulnerabilities, is not something you sign up for. It moves through the Fairwind Program, which names its audience plainly: government agencies and national cyber authorities, critical infrastructure operators, core technology platforms, Google Cloud customers and security partners. Participants must confine it to security staff and run multifactor authentication. There is no public API and no published price.

OpenAI went last and said the most. GPT-6 Astra is the first model the company designates at the Critical cybersecurity threshold of its Preparedness Framework, meaning it can find unknown flaws and build exploits across hardened systems without a person guiding each step. The public Astra refuses that work. The capability goes to a small group of alpha testers first, then widens through Daybreak Blue. And in the small print of its own capability post, OpenAI says the results it published reflect Daybreak Blue access rather than the default configuration. The benchmark model and the store model are not the same product.

The price did not move. The gate did.

Look at the stickers. Fable 5.1 and Astra both cost $10 in and $50 out. That is exactly double Opus 5, and it is the same number at two companies. Gemini 3.8 Flash costs a fraction of either, with the introductory rate ending December 31. Read as a market, the public tier priced itself in an afternoon and nobody blinked.

Now look at what the second tier costs, and notice there is no number. Mythos has no checkout page at all, because the purchase is not the gate. The gate is an application scoped to an organization, in the US, reviewed by a person. Flash Cyber has no price because it has no checkout. Astra’s advanced tier has a queue.

That is the change worth naming. For two years the frontier was a budget question: pay more, get more. As of this month the top of the range is a status question. What you are asked for is a government ID, two hardware keys, an organization ID, or a category you belong to. None of those come from a credit card.

What the door asks for, lab by lab

The three gates are not identical, and the differences matter if you are one person.

OpenAI’s is the most open to individuals. Daybreak’s own page says you verify with a government-issued ID, enable Advanced Account Security with two hardware security keys, and meet the program’s eligibility requirements. A solo builder doing authorized security work can, in principle, walk through that door alone. Astra’s most advanced configuration still starts with alpha testers, so the door exists and is not yet open.

Anthropic’s is organizational. The verification program takes individual applicants, but approval attaches to an organization ID, and the Mythos page limits current access to US organizations. A one-person company is an organization; a personal account is not.

Google’s is categorical. Fairwind admits kinds of institutions. There is no individual path described, and the program’s stated purpose is to give infrastructure operators an adaptation window before attackers catch up.

The part that hits the solo builder first

It would be easy to file this under “cyber models, not my problem.” Two details argue otherwise.

The first is the monitor. OpenAI states that its production safeguards for Astra can flag legitimate work as potential misuse, including work “that does not appear directly related to cybersecurity” and tasks where an agent runs for a long time. In ChatGPT or Codex you get asked to review. On the API, the task stops. The API is where a solo builder runs the overnight agent, the batch job, the thing that works while you sleep. The public tier is the tier that watches that pattern hardest, because it has to.

The second is what “same weights” implies. The version doing the impressive work in the launch posts is the one with fewer refusals. When a vendor benchmarks the vetted tier and sells the public one, the gap between the demo and the product is now a matter of who you are. That is a different kind of gap from the ones this publication usually prices, and it is not closed by upgrading your plan.

What to do this week

Decide which door your product depends on, and say it out loud in your plan.

If you are building anything security-adjacent, a scanner, an audit tool, a patching agent, the model that does the real work is behind a gate. Apply now, before you need it. OpenAI’s individual path costs a hardware key and an afternoon; Anthropic’s wants an organization, which a registered one-person company is; Google’s is not for you yet. Do not price a product on a tier you have not been admitted to.

If you are building anything that runs long unattended jobs on the API, budget for interruption. Add resume logic, checkpoint state, and log the stops, because OpenAI says they will happen and says they will not always be about security.

And read the launch benchmarks with the footnote in view. When a result says it was measured with special access, it is describing the model you are not being sold. The one you can buy is the one to test yourself, on your own task, at the sticker price, which is how this publication has always suggested you buy anything.

Anthropic's CEO recently released a 3,800-word essay calling for a global slowdown of AI development. He said that while the technology offers many benefits, it is advancing too quickly for researchers to continue safely. AI leaders are preparing for a day when an AI system can grow more advanced without the aid of human researchers. Some researchers believe that achieving recursive self-improvement will lead to AI systems spinning out of control.

We are all Product Engineers now

Just yesterday I published a very long post about the economics of open source. As part of that argument, I mentioned that the cost of writing software has collapsed, and that meant the variables in the equation had changed for the first time in thirty years.

That led me off on a tangent that grew into this equally long post. I had a bunch of questions to answer. Has the cost of creating software really collapsed? Can I prove that? If the cost of actually producing code goes to zero, what parts of the job of “software developer” really remain? Where, in fact, is the entire industry of software going in the next decade?

You can see why I felt it needed a post of its own.

I’ve been circling this topic for a while now. In early 2025 I predicted AI would create many more programmers and that their jobs would look different, but I didn’t get into the details of how different, and also that was more than a year ago, an infinity in the compressed timeline of AI. In March this year I found companies substituting compute for labor at record rates. In July I looked into labor statistics and found that the market for junior programmers had been savaged while the market for senior ones was fine, in fact growing.

This post is an attempt to build on those and make a forecast of where the industry is going in the next 10 years. Making a 10 year forecast of anything is of course a crazy thing to try to do, and especially about the business of software right now. To make it, I had to make two very big assumptions.

Assumption 1: agents are going to eat the entire software development lifecycle

This assumption is based on the observation that agents are currently very good at writing code and mediocre at everything that comes after that: reviewing code, testing it, finding bugs, fixing bugs, deploying to production, monitoring, and scaling up. They suck at that stuff right now, but my assumption is that that’s a temporary state of affairs. There’s nothing structural about those things that prevents agents figuring out how to do that stuff. If you think I’m right about that, this post will be of interest, but if you think I’m wrong now is a good time to bail.

Assumption 2: there is no upper bound to how much software we need

This one is if anything even more out on a limb. If you think I’m wrong about this you probably think software developers as a profession are doomed. I disagree.

I've made this argument before: look at the website of your dentist, your insurance company, your kid's school, or literally any department of any government, and you're looking at software that is terrible not because nobody knows how to build better software, but because the people who need it can't afford to pay for better at current prices. Then think about all the things software hasn't touched at all, which is most things. Every small business runs on a spreadsheet and a group chat and a person who remembers stuff.

That means there isn’t now and isn’t going to be a glut of software developers, and anything that looks like one right now is a temporary transitional state. The demand for software, at least inside my 10 year horizon, is for practical purposes infinite, or software developers wouldn’t be as highly paid as they are.

But the job of a “programmer” is about to get very, very different. So different that you might not even recognize it as “programming” any more, while still being recognizably “software development”.

The job of making software will become what the agents can’t do

If agents are going to eat the entire software development life cycle, what does that leave behind?

To figure that out, I broke the cost of making software into as many component pieces as I could think of. I came up with a long list, in four categories:

Collapsed:

• Actually writing code: historically the most expensive part of the whole process, because getting it right was really tricky. The entire industry oriented itself around very expensive programmers as the center of gravity, with every other job more or less orbiting around them. The cost of this, with LLMs, has already collapsed.

Going soon:

• Reviewing code: I’ve written about the death of the code review before, the TLDR being: it hasn’t happened yet, but it looks like it’s about to.
• Maintaining code: finding bugs, fixing bugs, refactoring. Agents are making real progress here but are still not great.

Next on the chopping block:

• Shipping code to production: getting it out of dev onto real production hardware. With various platforms this has been dropping for a while, and my assumption is that agents are about to get very good at it.
• Scaling up: not something I’ve seen anyone talk about, this is a big part of successful software development. I’ve not seen people throwing agents at production bottlenecks so far.

Possibly safe:

• Deciding what to build in the first place: figuring out what the customer actually wants is a huge part of software development, and so far I haven’t seen anyone throw an agent at it. To my mind, this is the most durable part of the job.
• Deciding the definition of “good”: this is the intersection with my day job in the world of AI evaluation. I’ve not seen any attempts to automate this. How would you even know, short of asking a human, what good looks like?
• Making it delightful: we can all tell the difference between a piece of software that gets the job done and one that’s actually easy and fun to use. Can an agent? The current state of agentic design does not suggest that they can, but this one is the most wobbly of the three.

All juniors did was write the code you told them to, and that’s gone

I already talked about this in my post about the labor market, so I won’t reiterate the whole argument. The thing agents got good at first was producing code from a description, which is exactly the thing junior developers were hired to do. It was the whole point of hiring a junior: you gave them a well-specified ticket, they produced mediocre code, a senior reviewed it, and over a decade of that they absorbed enough judgment to become the senior.

The problem from that post is: if you don’t need juniors to handle well-specified tickets any more, where do the seniors come from? We have to train them in a different kind of job. The point of this post is: what job?

Since July the Stanford team has updated their numbers and things did not improve for junior developers. The employment gap for 22-to-25-year-olds in AI-exposed jobs is now 19% below where it would be if they'd tracked their less exposed peers, up from 15% a year ago, and it's happening through reduced hiring rather than layoffs. More interesting is where it's happening: young workers lost ground in occupations built on knowledge that's been written down somewhere, and experienced workers gained ground in occupations built on knowledge you get by doing the job. The Stanford authors call these codified and tacit knowledge, and I'd call them "stuff that's in the training data" and "stuff that isn't", but it's the same distinction, and it maps exactly onto "what juniors do" and "what seniors do." SignalFire's 2026 talent report has the corporate side: entry-level hiring at the big tech companies is down 65% since 2019, at early-stage startups it's down 75%, and yet engineering as a share of hiring went up, from 46% to 55%.

Companies are hiring fewer people overall, but a bigger share of the people they do hire are engineers, just not the kind whose primary job is typing code.

Going soon: reviewing and maintenance

For reviewing and maintenance, agents are clearly not there yet, but the data shows them on an upward trajectory.

On benchmarks where agents fix real bugs in real repositories, frontier models went from roughly 50% to roughly 95% in the last two years, to the point where the main benchmark is effectively saturated and people have had to build harder ones. On the harder ones, which resist the models having seen the answers during training, the best models now score around 59%. That’s not good enough, but neither was 50% two years ago and that went away really quickly.

A study of 567 pull requests opened by Claude Code across 157 open source projects found 84% of them eventually got merged, a bit below the human rate of 91%, and just over half went in without a human touching them. Google's Big Sleep agent found a memory corruption bug in SQLite that traditional fuzzers had missed and that attackers already knew about, and has found around twenty more since in things like FFmpeg and ImageMagick.

Until they do, the ability to create code but not to review it is causing an enormous amount of pain. GitHub added 36 million developers and a quarter more commits in a year, and the number of merged pull requests on the platform is up something like three and a half times since 2023, with one estimate having agents alone opening 17 million PRs a month. Something should review all of that, but one study of 33,000 agent PRs found that most PRs on GitHub, human or agent, get no recorded review at all, and when agent PRs are reviewed, 58% of the time the only reviewer is another agent. In open source, examples abound of projects shutting out new submissions because of a tide of AI slop and the inability to effectively review them; curl shut down its bug bounty in January after the share of submitted reports that were real bugs fell from better than 15% to under 5%.

Next on the chopping block: operations and scaling

For this part of my argument data was really thin on the ground, so I’m relying heavily on my “looks like it’s going to happen” assumption from the start. There are some benchmarks that look more like operating a system than fixing a bug, and agents are somewhere under 65% on them. This isn’t a thing happening yet, which is why there’s almost no data either way. It’s just the thing that, logically, looks like it’s next.

What's left is finding out what people actually want, and only they know

So if the code is free and the operations are free, what’s left? It’s sometimes called "product sense", and it’s highly valued in senior developers, but what does that mean exactly?

At some point every piece of software is a formalization of a human desire. Somebody wanted something, and the software is a precise enough statement of that want that a computer can act on it. When a customer says "I need to keep track of my orders," there are ten thousand pieces of software that fit that sentence, and only one of them is right for a bakery, and it's a different one from the one that's right for a car parts factory, and the only person on earth who knows that the customer is running a bakery and not a parts manufacturer is the customer.

You cannot do product discovery mechanically short of reading people’s thoughts. You can't train it into a model, because it isn't in the training data, because it's in the head of one specific baker who's never written it down and wouldn't know how to if you asked them. Somebody has to go and get it out of her, and then turn it into something exact enough to build, and then check that what got built is actually what she meant, which it never is the first time.

There is no economy of scale in product decisions

The cost of deciding what the customer wants has a very important property: it doesn't transfer well. The definition of "good" for a calendar app and the definition of "good" for a scheduling app, which are two ways of solving roughly the same problem, have almost nothing in common, and two bakeries don't have exactly the same problem either. Whenever you see software with a zillion configuration options that still doesn’t do what you need it to do, you’re feeling this problem. It’s why software so often sucks, and why I say the demand for good software goes to infinity. Software requirements are more different than we’ve been able to admit while we’re still trying to write one-size-fits-all software.

As the cost of software creation falls to zero, the bottleneck moves to the description of the problem, and my thesis is that’s where it’s going to stay.

What about design?

I'd separate out design from this, because it's related but it's not the same thing. Design is the part where two solutions both correctly solve the problem and one of them is the one people actually like using. Everybody who's watched a well-specified product lose to a nicer one knows this is real, and I can't quantify it, and I'm suspicious of anyone who says they can. But I'll note that it has the same structure as the description cost: it's per product, it doesn't transfer, and cheap code makes it more important because when everyone can build the correct thing, the nice thing is what's left to compete on.

The job that remains is called Product Engineering

So what does that leave behind? Let’s talk history for a little bit.

When computers were new and programmers were scarce and expensive, companies hired a person whose entire job was to sit between the business and the programmers, understand what the business needed, and write it down precisely enough that a programmer could build it without talking to anyone. This person was called a systems analyst. There's a 1963 memo from Miami University describing systems analysis as a brand new profession born out of the mountain of paperwork business executives faced: it was the translation layer, created because the people who could type were too valuable to also do the talking (and also, people who were very good at laying down code seemed to be not very good at talking to humans anyway).

Then software went commercial and, especially, consumer-facing, and the translation job changed shape. Consumers don't want to sit in requirements meetings: they just want to be handed a thing they like. So the person whose job was understanding what people wanted stopped being an analyst who interviewed the business and became a product manager who studied the market, a role borrowed more or less directly from Procter & Gamble's brand managers by way of Intuit and then Microsoft, where a programmer named Jabe Blumenthal invented "program manager" in the late 1980s because Excel for the Mac needed somebody to own what it should do.

The function moved into Product, and Product got separated from engineering as a career, and for the last twenty-five years we've had two professions where there used to be one and a half. I bring this up because it means the job I'm describing isn't a speculative new thing that we'd have to invent. It's a thing we've had for sixty years under two names. My speculation is that it’s about to collapse back into one job.

The new job is already being hired for, under a dozen names

You can see the start of this change arriving now: it’s showing up as job postings for a role nobody had heard of three years ago.

Palantir coined "forward deployed engineer" for a person who goes and sits with the customer, figures out what they actually need, and builds it, inside the customer's environment, with the customer watching. It was a Palantir oddity. Then in 2025 postings for it grew by something like eight hundred percent in nine months, and by this month a census counted almost a thousand live postings across 462 companies, including OpenAI, Anthropic, Databricks, Stripe and Google Cloud, with Salesforce saying it wants a thousand of them to roll out its agent products. The average total comp is around $240,000 and senior ones clear $600,000, which is to say it pays like a senior engineer, because it is one. The same role is being posted as solutions engineer, deployment engineer, applied AI engineer, implementation engineer, and half a dozen other things, because nobody has agreed on the name yet, because it’s so new that nobody has standardized it yet.

But read the job descriptions and you see, roughly, a senior product engineer. The responsibilities include: scope the problem with the customer, understand their business, write production code into systems you didn't build, iterate with them until it works. The code-writing is in there, but it's the smallest part, and it's the part the agent does; what the company is paying $240,000 for is the person who can walk into a car parts factory and come out with a correct definition of "good." The market has already decided this job is incredibly valuable.

But that’s not programming!

Here’s the part that’s going to suck for a lot of people who develop software currently: no, this isn’t programming. It’s recognizably still software development, but laying down code is a vanishingly small part of it and, if the trends I’ve laid out here are real, going to get even smaller.

I want to be careful here because “figure out what to build, not how to build it” is also a description of the part of software development I personally always liked, and there's a well-known failure mode where everyone with an opinion about AI concludes that all jobs will be automated except theirs, which is mysteriously impossible to automate. So take this with the appropriate salt: I think the durable, paid part of making software becomes the part where you understand a problem better than the customer does and think harder about the solution than they can, and I think that's durable because it can't be extracted from the customer mechanically, and I think it's paid because if you don’t do it you get software that everyone agrees sucks, which is to say: most current software.

The market wants context and taste and nobody is being trained for those

Here's where my forecast runs into a problem.

The input the software development industry is about to need in unlimited quantities is people who can extract requirements from humans, define good, and exercise taste, and we do not make those people. Product people fall into their jobs by accident, as a byproduct of the typing job, or sometimes a marketing job, or maybe a consulting job. For developers, you hired a junior to write code, a senior reviewed it, and over a decade the junior picked up judgment by osmosis. That's how every senior engineer I know got their taste, and it's the loop I said in July is now broken, and it's broken because the first rung on the ladder was "type code somebody else reviews" and the agents are going to do both of those things.

Formalized training of product people barely exists. Google's APM program, which Marissa Mayer started in 2002 and which is the template everyone copies, takes about fifty people a year out of something like twelve thousand applicants. Meta, Uber, LinkedIn, Salesforce and a few others run equivalents of similar size. Add them all up and you get maybe a few hundred people a year trained, on purpose, to do the thing I'm claiming is about to be the whole job, against a junior developer pipeline that used to be tens of thousands and is now on fire. Universities teach data structures. Bootcamps teach React. Nobody teaches "go sit with a baker for a week and come back with a spec," and the pipeline for turning junior devs into that role by accident has been closed, also by accident.

The market wants people with context and taste and we are simply not training those. We’re not even sure we know how. Until that changes, the scarce input stays scarce, the people who have it get more expensive, and most of the world’s software stays bad for longer than it needs to.

I do think the market will probably solve for this. The price of the scarce thing goes up until somebody finds it worthwhile to make more of it. IBM is already redesigning its entry-level role around customer contact and specification instead of typing. Companies paying $240,000 for forward deployed engineers will eventually notice it's cheaper to grow them, and universities will eventually notice that "requirements analysis" is a course people would pay for, but it will all happen too slowly, and a cohort of people will get hurt in the meantime, and I'll come back to them. But the demand is real and the demand is what fixes it – eventually.

The craft as paid work is mostly dead, and that is a real loss

I’ve posted this sentiment before, but it’s a real tragedy that shouldn’t be glossed over. I've seen a lot of despair from career programmers over the last two years and I don't think the right response to it is a chart showing that aggregate employment is going to be fine.

A lot of people got into programming because they love the craft of it. The feeling of a clean abstraction. The satisfaction of a hard bug finally yielding. The specific pleasure of making a machine do exactly what you told it, which is a pleasure most jobs don't offer. Those people did not sign up to interview bakers. Some of them have no interest in product management and some of them are actively bad at it, in the way that some brilliant engineers are, and they're looking at the forecast I've just written and seeing their job turn into a job they'd never have chosen.

I think they're right, and I don't have a consolation prize. The craft of writing code as a thing somebody pays you to do is, I think, mostly over, outside of niches that will get narrower every year. That's a real loss and it's a loss for the profession as well as for the people, because the craft is where a lot of the taste I've been talking about actually came from, and we're about to find out what taste looks like when nobody grew up doing the thing.

Two things I'd say that aren't consolation, just observations. One is that for a fair number of the people who think they loved the typing, the part they actually loved was the moment before the typing, when a vague mess of a problem resolved into a precise shape in their head. That moment is the job now. If that's what you loved, you're going to be fine and possibly better than fine, because the industry is about to be desperate for you. The other is that the craft survives, the way woodworking survived the furniture factory, as a thing people do because they love it and occasionally get paid a premium for. Developers write software the way singers sing. That was true when it was free and it'll be true when it's automated, and the people who love it will keep doing it, and some of the best software will keep coming from them. It just won't be the job.

Ten years of turmoil lie ahead

The cost of writing code collapsed, and the cost of reviewing, fixing and operating it is following, and I'm assuming it gets there. What's left of making software is finding out what people actually want, defining it precisely, and making it pleasant to use. That cost is per piece of software and doesn't transfer, so as the amount of software goes to infinity, which it will because there's no ceiling on demand, that cost becomes the whole job.

That job is called a product engineer. It's being hired for right now under a dozen new names at senior engineer pay. And the training pipeline for it is roughly fifty people a year at Google, because the way we used to produce it was as a side effect of a typing job that no longer exists.

I think the next ten years are going to be ugly, because the load is arriving before the tools do, the junior ladder is gone before the replacement exists, and a lot of people who loved the craft are going to have to decide whether they love the job that's replacing it. I think by ten years it shakes out, the way it did when compilers and then frameworks and then open source each made a generation’s worth of typing unnecessary, into a profession that is larger than today's, pays about as well, and is mostly shaped like product engineering. At twenty years I have no idea; if we hit anything resembling general intelligence in that window then this post and every other post about jobs is moot. But for the horizon I can see, the forecast is: more software, more people making it, and almost none of them typing. We are all product engineers now, whether we like it or not, and a lot of us won't.

Kubernetes v1.37: Native Histograms Graduates to Beta

I'm excited to announce that native histogram support for Kubernetes metrics is graduating to Beta and is enabled by default in Kubernetes v1.37!

Native histograms (previously introduced as Alpha in Kubernetes v1.36 under KEP-5808) bring high-resolution, low-cardinality observability to Kubernetes metrics. By adopting Prometheus Native Histograms, Kubernetes components now expose latency and duration metrics with far greater accuracy while significantly reducing telemetry storage and scraping overhead.

Why move beyond classic histograms?

Since the early days of Kubernetes observability, duration and latency metrics (such as API server request latencies or scheduling durations) have relied on classic Prometheus histograms.

Classic histograms require metric authors to define a static list of cumulative bucket boundaries (le labels), such as 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10. While familiar, this approach introduces three major challenges:

1. The Bucket Guessing Game: If a workload's latency profile changes, for example, shifting into microsecond ranges or experiencing long-tail tail latencies beyond the highest bucket, the histogram loses visibility. Specifying bucket boundaries upfront requires knowing the distribution before observing it
2. High Cardinality & Storage Cost: With classic histograms, each bucket boundary is exported as a separate time series (_bucket{le="..."}). A histogram with 10 buckets across multiple labels multiplies the number of time series by 10, increasing memory consumption in Prometheus and inflating time series database (TSDB) storage costs
3. Interpolation Error in Quantiles: Calculating percentiles using histogram_quantile() relies on linear interpolation between static bucket boundaries. When bucket spans are coarse, quantile calculations can suffer from significant estimation error

What are Prometheus native histograms?

Prometheus Native Histograms replace static user-defined buckets with dynamic, exponential buckets.

Instead of emitting a separate time series for every single bucket boundary, a native histogram is stored as a single time series containing a rich schema of positive and negative spans, zero thresholds, and exponential scaling factors.

• High Resolution Automatically: Exponential buckets dynamically adjust to any value range — from nanoseconds to hours — without requiring pre-configured bucket boundaries
• Up to 90% Fewer Time Series: By consolidating buckets into structured spans within a single time series, scraping and storage overhead are dramatically reduced
• Accurate Quantile Calculation: Quantiles can be calculated with mathematical bounds on error (≃5% worst-case relative error under default settings) across the entire spectrum of observations

How native histograms work in Kubernetes

In Kubernetes, native histogram support is implemented directly inside the shared metrics subsystem (k8s.io/component-base/metrics).

1. Dual exposition for zero breaking changes

A primary design requirement for KEP-5808 was zero disruption for existing observability stacks. When the NativeHistograms feature gate is enabled, Kubernetes components use dual exposition:

• Classic buckets (h.Bucket) are still emitted alongside native spans. Existing Prometheus servers, dashboards, and alerting rules that rely on traditional text scraping or classic bucket labels continue to work unmodified
• Native spans (h.Schema, h.PositiveSpan) are included in the same Protobuf payload for collectors that understand native histograms

2. Tuned default exponential configuration

When NativeHistograms is enabled, the k8s.io/component-base/metrics package automatically applies standardized exponential options to all histogram metrics:

• BucketFactor: 1.1: Configures exponential buckets where each bucket is at most 10% wider than the preceding one. This guarantees a mathematically bounded worst-case relative error of at most ~5% for quantile calculations regardless of whether an operation takes 1 millisecond or 10 seconds.
• MaxBucketNumber: 160: Caps the maximum number of buckets per histogram to 160. Following OpenTelemetry SDK recommendations for base-2 exponential histogram aggregation, this limit protects component memory usage even under extreme outlier distributions.

3. Broad component support

Because native histograms are integrated into component-base/metrics, all major Kubernetes control plane and node components inherit support automatically, including:

• kube-apiserver (e.g., apiserver_request_duration_seconds, authentication/authorization metrics, validation latencies)
• kube-scheduler (e.g., scheduler_plugin_execution_duration_seconds, scheduler_scheduling_algorithm_duration_seconds)
• kubelet (node-level container runtime and pod lifecycle metrics)
• kube-controller-manager and kube-proxy

How to scrape native histograms

The simple answer: upgrade to Kubernetes v1.37, and it works.

1. Prometheus scrape configuration by version

• Prometheus 3.0+ (Recommended): Use explicit per-job configuration in your scrape_configs rather than global flags (the global --enable-feature=native-histograms flag is deprecated in Prometheus 3.9+):

  scrape_configs:
    - job_name: 'kubernetes-apiservers'
      scrape_native_histograms: true
      always_scrape_classic_histograms: true  # Recommended during transition
  

  In summary: always set always_scrape_classic_histograms: true during your transition period. Without this setting, Prometheus will only ingest the native format and stop ingesting classic _bucket, _count, and _sum series. Setting always_scrape_classic_histograms: true ensures existing dashboards (histogram_quantile(..._bucket...)) and alerts continue to work while you migrate them to native histograms.

• Prometheus 2.40 – 2.x: Enable Native Histograms globally by starting Prometheus with the feature flag:

  prometheus --enable-feature=native-histograms
  

  Note that in Prometheus 2.x, this is an all-or-nothing setting for all scrape targets.

2. Verify Protobuf dual exposition

Standard Prometheus text scraping (application/openmetrics-text or plain text format) only transfers classic buckets. When scrape_native_histograms is enabled, Prometheus automatically negotiates Protobuf format with Kubernetes endpoints.

Querying native histograms in PromQL

Once native histograms are ingested into Prometheus, you can query them using standard PromQL histogram functions without needing static le bucket labels or _bucket suffixes:

# 1. Calculating P99 latency for a single target:
# Native histogram (operates directly on the metric name):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))

# 2. Aggregating across multiple instances (e.g., all API servers):
# Native histogram (no grouping by le required!):
histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds[5m])))

With native histograms, functions like histogram_quantile() operate directly on the dynamic exponential spans inside the time series, producing highly accurate quantiles without static bucket interpolation error.

Dashboard migration & rollback strategy

Recommended migration workflow

1. Enable Both Formats: In your Prometheus 3.x scrape config, set scrape_native_histograms: true AND always_scrape_classic_histograms: true so both formats are collected safely during transition
2. Migrate Queries: Update your Grafana dashboards and Prometheus alerting rules from classic quantile queries (histogram_quantile(..._bucket...)) to native histogram queries (histogram_quantile(...)), and replace references to classic _count and _sum series with histogram_count(...) and histogram_sum(...)
3. Verify in Staging/Production: Validate that all dashboards and SLO alerts fire and graph correctly using the new native histogram queries
4. Unlock ~10x Storage Savings: Once migration is complete, set always_scrape_classic_histograms: false. Prometheus will stop ingesting the static _bucket, _count, and _sum time series, reducing your histogram time series count by up to 90%!

Opt-out and rollback flexibility

• Instant Collector Rollback: If you need to stop ingesting native histograms, simply set scrape_native_histograms: false in your Prometheus job configuration. No Kubernetes restart is required, and Prometheus will immediately resume scraping only the classic format without data loss
• Component Feature Gate Rollback: Administrators can also disable the feature gate on Kubernetes components using --feature-gates=NativeHistograms=false (requires component restart)

What's next & how to get involved

As native histograms progress toward General Availability (GA) in future Kubernetes releases, SIG Instrumentation will continue evaluating ecosystem readiness, performance characteristics, and long-term plans for eventually deprecating static classic buckets once native histogram adoption becomes ubiquitous across the monitoring community.

• Read the KEP-5808 page
• Read the Prometheus Native Histograms specification
• Get involved with SIG Instrumentation

Full article content is not available for inline reading.

Read the original article →

Being "the database guy" comes with a lot of questions, and over the last eight months those questions changed. The repetitive ones disappeared, nobody asks how to avoid putting things in the database any more, and the code arriving for review got noticeably more polished. Then this summer a schema landed in front of me with twelve proposed index drops on a single table, which is when I started assuming coding agents over-index. The next schema I looked at had the same shape.

Passing this off as AI slop would be too easy, because most of those changes were competent. So I built a harness and measured it.

I loaded 30 model-generated schemas into PostgreSQL and audited 838 indexes across twelve of them. The competence caught me off guard. Only ten served no requirement I could find; the rest showed solid craft. All four models handled GIN and GiST indexes cleanly, built partial indexes with sensible predicates, and got multi-tenant composite keys in the right order. The baseline SQL quality is much better than what agents wrote a year ago.

The cost of an extra index

Indexes are great, until you pile them onto the single table taking all your writes. On quiet tables you will never notice the difference. On hot tables, every index is extra work on every write.

In one support-tool schema, a model created sixteen indexes on tickets alone. Six of them indexed last_activity_at, a column that updates every time an agent touches a ticket. Compared to my hand-written baseline with seven indexes, the generated schema wrote 1.8× the WAL, took 1.9× longer per update, and pushed up VACUUM time just as much.

Those sixteen indexes were not dumb mistakes. For read queries, they run fast. The problem is that coding agents write indexes query by query, without thinking about write traffic.

What actually happens on disk when you touch that row:

• No more HOT updates. If any index touches the modified column, Heap-Only Tuples is out the window. Postgres can't keep the new row version confined to the original 8 KB data block without updating index pointers.
• Every index on the table whose WHERE predicate matches the new tuple gets a new entry written into it—not just the index covering the column you changed.
• Extra WAL records for every single one of those index inserts.
• Dead tuple cleanup: the old row version sits in the heap until VACUUM cleans it up, but VACUUM has to cycle through every secondary index to prune the pointers pointing at it. More indexes, slower vacuum sweeps, even if only three rows changed.

I never got the vacuum cost isolated from the WAL volume; the two kept confounding. Treat the vacuum numbers below as direction, not measurement.

The setup

I created six fictional SaaS products for the review.

• Shared inbox support tool
• Class booking system for gyms
• Product analytics tool
• Marketplace app
• Veterinary practice system
• Freight management board

No table names, no column hints, and no database constraints, except that Postgres will be used. Also, there's no mention of anyone counting indexes later. I wrote the first four specifications myself (only polished them with an LLM). The last two were written by an independent model from a short description of what I wanted. One of the six specs is thinner than the others and it shows in the output. I left it in place rather than change it. The revisit would contaminate the run with all the bias of things I learned reviewing it.

Each specification went to new model instances without schema templates or index hints. Design the PostgreSQL schema, write SQL queries, and don't ask questions. Four models from three vendors created the schemas. To focus on the database instead of a model leaderboard, I label them Model A to Model D. This way, there are no names and no ways to identify the models.

Twenty-six of the thirty runs carried one extra line at the end of the prompt: make it production-ready. That is what people actually type into a coding agent, so I left it in, and later re-ran Models B and D without it as a separate condition.

Thirty runs in total were loaded into PostgreSQL 18.6. All metrics come from pg_index and pg_stat_* on a live database. Schemas, specifications, harnesses, raw CSVs and the pre-registration file are at github.com/boringSQL/vibe-coded-indexes.

Indexes pile onto one table

Average index density looks fine on a summary slide. The catch is where those indexes land. Every agent dumps its indexes onto the single table with the most writes: helpdesk runs hit tickets, fitness hits class_occurrence, freight hits loads.

Each app keeps its core data in those tables. The models read requirements feature by feature: every filter or sort rule prompts another index, and no agent checks what is already there. Interestingly, no model indexed foreign keys by reflex; if anything, they under-indexed them. Partial, expression, GIN, GiST, INCLUDE, tenant-prefixed composites: all syntactically fine.

What sixteen indexes cost

I ran a synthetic benchmark on a million-row tickets table with a realistic support mix: roughly 60% customer replies, 25% status updates, and 15% ticket reassignments. This was on PostgreSQL 18.6 with fillfactor=90. Autovacuum was disabled to keep row layout predictable, capturing WAL with EXPLAIN (ANALYZE, BUFFERS, WAL) averaged over three runs.

Notice run 3: nine indexes, yet it generated slightly less WAL than my 7-index baseline and beat it on update time. Why? Strict partial predicates: three of its indexes had restrictive WHERE clauses that touched almost none of the updated rows:

CREATE INDEX tickets_unassigned_urgent_idx ON tickets (workspace_id, priority, created_at)
  WHERE assignee_kind IS NULL
    AND status <> ALL (ARRAY['solved','closed']);

My hand-crafted baseline had wider, unconditional indexes that forced more full-page writes. In other words, WAL volume tracks index footprint and page dirtiness, not raw index count. Still, when models go up to fifteen indexes (run 1 and run 2), the cumulative penalty adds up: 1.8× the WAL, double the latency, 1.6× the on-disk footprint, and an extra 65% on VACUUM, all for the exact same 200,000 updates.

Those indexes work

helpdesk-run1 is 23 to 46 times faster on four of the nine queries I measured. Eight of the nine queries perform as expected. The ninth is the SLA sweep, and it runs 111 times slower on the generated schema. The index meant to serve it looks right:

CREATE INDEX tickets_first_response_due_idx ON tickets (workspace_id, first_response_due_at)
  WHERE first_response_at IS NULL
    AND first_response_due_at IS NOT NULL
    AND status <> ALL (ARRAY['solved','closed']);

An SLA sweep is inherently cross-tenant, and the leading workspace_id that is right everywhere else on this table is wrong here.

Where it breaks even

Leaving out that SLA query, helpdesk-run1 saves 0.554 ms per execution on average across the other eight queries. Planning overhead takes back 0.146 ms: more indexes give the planner more choices to evaluate on unprepared queries. Net gain: 0.408 ms saved per read.

Comparing read savings against write penalties, the fifteen-index schema wins on CPU time as long as you do fewer than twenty updates for each read. Measured on CPU time alone there is nothing here to argue with.

Which column the index sits on

last_activity_at is a key column in six of the sixteen indexes, and that placement costs more than the count does. I built a small table to test it: six secondary indexes either way, the same UPDATE ... SET last_seen_at = now() over 300,000 rows, and the only difference is whether one of those six sits on the column being written.

Moving an index onto the touched column costs every HOT update and adds 45% to the time. A low hot_pct on a hot table is the real cue: which index is on the column your UPDATE touches?

What the indexes evict

Once HOT is disqualified, every secondary index you add costs about 26 MiB of extra WAL across the same 200,000 updates, averaged over twelve of them. It is not a flat rate: the per-index cost ran from 18 to 46 MiB depending on how many full-page images that step happened to trigger, and you pay it on every write from then on.

The cache impact was worse. When I throttled shared_buffers to 128MB with a cold cache, physical reads jumped by 6.7×, and the cached heap fell from 35 MiB to 9 MiB while the cached index went the other way.

Nobody drops the twelfth index

The other half of the problem is the next commit. I gave the sixteen-index tickets table and an ordinary feature request to six fresh instances. Not a single run removed the twelfth index. Five of the six added a new one, every time on a mutable column together with a mutable predicate.

"Make it production-ready" is a database decision

I ran two models again with the phrase "make it production-ready" removed, leaving nothing but "read this file, write the schema". That one phrase is worth about a fifth of the index count. Three words at the end of a prompt, which nobody thinks of as a schema decision, move the write cost of the busiest table in the application.

Checking the write path in production

On a live database, check pg_stat_user_indexes before touching anything:

SELECT s.relname, s.indexrelname,
  pg_size_pretty(pg_relation_size(s.indexrelid)) AS size,
  i.indisunique AS uniq,
  s.idx_scan, s.last_idx_scan
FROM pg_stat_user_indexes s
JOIN pg_index i ON i.indexrelid = s.indexrelid
WHERE NOT i.indisprimary AND s.idx_scan = 0
ORDER BY pg_relation_size(s.indexrelid) DESC;

Read the output before dropping anything: counters reset on restarts, replicas might use indexes the primary ignores, and unique indexes exist for data integrity.

Full article content is not available for inline reading.

Read the original article →

Today we're launching Projects in Cursor. Projects lets you take on larger bodies of work, such as a feature, a migration, or a full app. It maintains context over months of work, delegates tasks to thousands of subagents, and performs recurring work without being prompted.

In February we outlined our vision for a third era of software development, where fleets of agents take on entire bodies of work. Projects is the concrete implementation of that vision. By moving up a level of abstraction, it frees developers from managing agents and lets them direct the work itself.

At Cursor, we've been using Projects for several months, doing work such as running migrations of a few hundred PRs, keeping our design system consistent, and shipping Projects itself. We've found it to be a substantial productivity multiplier: new users merge 30% more PRs while users who primarily use Projects merge six times as many.

Direct thousands of agents through one coordinator

You oversee a Project by chatting with its coordinator agent. The coordinator doesn't write code itself but directs other agents that do. Because it delegates rather than executes, it is never blocked and is always responsive to direction.

There are three core capabilities that make Projects possible:

Cloud by default, local when needed. A Project runs on its own computer, so closing your laptop doesn't stop it. This lets a Project run more subagents in parallel than your laptop could support. When something needs testing on your machine, the coordinator spins up a local agent to run it there.

Shared context. You shouldn't have to onboard an agent every time you start a task. Each Project maintains a set of files that sync across every cloud and local machine its agents use. Agents add research and artifacts, along with what they learn about the codebase and how you prefer work to be done. If one agent figures out how to test a service, for example, every future agent can use those instructions. This context grows with the Project, making the coordinator more effective over time.

Subscriptions. The coordinator can watch a Slack channel, run on a schedule, or follow all your PRs, fixing CI and acting when they open or merge. This way it can take action based on signals it detects, without waiting for you to prompt it.

How we use Projects at Cursor

Three patterns cover most of what our engineers do with Projects.

Feature work

Most engineers create a Project for a substantial body of work. A feature usually starts with agents researching the system and recording what they learn as shared context. The coordinator then creates a plan and sends agents to implement and test different parts of it in parallel.

With each turn of feedback, the Project learns your architecture and preferences. When the feature is ready to try, the coordinator can start an agent on your computer and run it locally. After it ships, the same Project can monitor logs and handle bug reports with the full context behind the original decisions.

Migrations

Projects are especially useful for migrations that are easy to start and difficult to finish. At Cursor, we've used them to adopt new frameworks and replace styling systems across hundreds of PRs.

You work with the coordinator to establish a safe approach, then it applies that approach incrementally across the codebase. Early on, you review each PR closely. As the fixes hold up, you review less, and the coordinator keeps working through the migration on its own.

Gardening

Projects are great for handling work that never really ends, such as maintaining code quality or watching for regressions. You can tell the coordinator to follow new PRs, listen for bug reports in Slack, or run on a schedule, and it acts whenever new work appears.

One engineer on our team runs a design-system Project this way. At first, the engineer reviewed each fix and corrected the ones it got wrong. Now the coordinator scans every new PR, extracts components that belong in the design system, and adds a lint rule whenever it sees the same mistake twice. The Project is on track to touch 20 to 100 PRs a day, so the coordinator organizes the work and the engineer checks in where attention is needed.

Get started with Projects

Projects are available in beta and rolling out to all users starting today. Start a Project from the left hand nav, describe what you want built, and the coordinator takes it from there. It works best on work that will outlive a single chat, whether that's a feature with several PRs, a migration, or a job you want handled while you're away.

An exact duplicate reused 8 of 9 tokens and required 1 token of prompt work.

One interior mutation cut safe reuse to 3 of 9 tokens. Namespace isolation forced 0 reuse. A capacity revisit ended as the typed verdict evicted. Across all ten cases, cached and no-cache outputs stayed token-identical and every evaluator passed.

That is the shipped result.

Evidence boundary: this is a deterministic synthetic token-level control. It validates the auditor, oracle, attestation path, evaluator, schemas, privacy rules, hashes, and standalone verifier. It does not establish MLX or vLLM speedup, production cache correctness, provider identity, GPU performance, latency savings, or runtime-memory savings.

A runtime can report a cache hit without proving that the intended prefix matched. It can report eight cached tokens while the prompt path still recomputes them. It can reuse state and return a different answer. It can label an isolated namespace as a miss without telling you whether the result came from isolation, eviction, or a cold cache.

The cache event is one statement from one part of the system.

The engine attestation is the statement under test. LLMTraceFX checks it against five independent checks: the token oracle, observed prompt work, output-token identity, evaluator correctness, and the checksum-bound evidence bundle. The claim survives only when the chain agrees.

What the ten cases prove

The reference control runs a fixed request sequence through a token-granular cache. For each request, an oracle calculates the reusable prefix from exact token identity and the pinned cache policy. The engine reports its reuse. The prompt path records the work it processed. A no-cache control supplies the output identity and correctness checks.

The first two rows establish cold state. capacity-seed inserts a four-token candidate. cold then submits a nine-token request with no reusable prefix. Both are verified_miss.

The exact duplicate is the clean positive control. The oracle expects eight reusable tokens, the engine attests eight, and the prompt path processes one. The verdict is verified_hit.

The interior and boundary mutations change where the exact token prefix stops. The interior mutation reuses three tokens and processes six. The boundary mutation reuses four and processes five. Both are partial_reuse.

The same-length case is the trap. Its request still contains nine tokens, but the token IDs differ. The oracle expects zero reuse. The engine attests zero. The prompt path processes all nine. Equal length creates no identity.

The suffix-change case keeps an eight-token prefix. It processes the changed final token and returns partial_reuse, not a full hit.

Namespace isolation also forces zero reuse. Capacity pressure fills the controlled cache. The later capacity revisit has zero reuse and four tokens of prompt work, but its verdict is not a generic miss. The retained predecessor proof and controlled residency observations show that the earlier candidate left the cache, so the verifier returns evicted.

Every row preserves output identity: yes and evaluator: yes. Those checks do not change the cache verdict. They answer separate questions: did reuse alter the output, and did the result remain correct?

A hit needs an evidence chain

The independent oracle matters because the engine cannot verify itself. It derives the safe prefix from the exact request, cache state, namespace, and policy. That expected count then has to match both the engine's attestation and the amount of prompt work observed.

Output identity adds another boundary. A correct reuse count does not prove that the response stayed the same. The evaluator adds one more. Token-different output can still be correct, and token-identical output can still fail a task if the control itself is wrong.

Three lessons from nine tokens

Equal token count is not equal token identity

cold, same-length-different-ids, and namespace-isolation all contain nine input tokens and all process nine tokens. The reasons differ.

Mutation position defines the reusable prefix

Similarity is not the rule. Prefix identity is.

Eviction and isolation are not the same operational case

Zero reuse describes an amount. It does not describe a cause.

Reproduce the public proof

From the merged LLMTraceFX repository:

make kv-cache-demo

What this result does not prove

• MLX or vLLM speedup;
• production cache correctness;
• provider or hardware identity;
• GPU performance;
• latency reduction;
• runtime-memory savings;
• block-level cache behavior; or
• that a runtime integration has enough evidence merely because it emits cache events.

It proves a narrower and more useful thing: for a deterministic synthetic token-level control, the auditor records full reuse, partial reuse, verified misses, and eviction while the case evidence preserves namespace isolation. Output identity, evaluator correctness, privacy rules, and hash-bound verification remain intact.

The result

A cache hit can be true and still fail to prove that work was skipped.

The useful claim is stricter: the independent oracle expected the prefix, the engine attested it, the prompt path skipped it, the output stayed identical, the evaluator passed, and the verifier bound those facts to the exact public bundle.

That is when a cache event becomes evidence.

ToolGrad: Efficient tool-use dataset generation with textual "gradients"

ToolGrad is a data generation framework that reverses the traditional paradigm by first generating tool-use answers before user queries. We show this design enables LLMs to achieve better tool-use performance.

AI agents have shown great potential in automating real-world tasks, such as conducting a Google Search, reading local computer files, or executing generated Python scripts. To achieve such agentic workflows, LLMs need to learn how to use tools correctly and efficiently. To teach large language models tool uses, we need datasets of tool-use chains and their corresponding user queries. In our prior work introduced in InstructPipe, we manually annotated our evaluation data, but it is impractical to scale up the human annotation for advanced LLM fine-tuning workstreams. To streamline the data workstream, prior work, e.g., ToolBench and ToolACE, explored using an agent to automatically search a tool-use path with trial and error. This representative annotation approach involves two steps: (1) generate a hypothetical user instruction from a sampled API pool, and (2) use a depth-first search (DFS) agent to find its tool-use solution. This approach is inherently inefficient because its core concept is to distill valuable trajectories from a complex agent exploration for training an LLM.

In “ToolGrad: Efficient Tool-use Dataset Generation with Textual ‘Gradients’”, presented at ACL 2026, we introduce an alternative solution paradigm. ToolGrad first generates a ground-truth tool-use chain and then annotates its corresponding user prompt. Intuitively, an explicit tool-use solution provides more unambiguous information than a prompt, making the annotation, from tool usage to the use query, much easier and requiring only one LLM step. Our result shows that our answer-first approach can generate more complex (long-horizon) tool-use data with lower cost. LLMs trained on our generated data also outperform those trained on baseline methods, and even match SoTA proprietary LLMs on out-of-distribution (OOD) datasets with unseen tools.

While prior art generates tool-use datasets by searching solutions of user queries with low pass rate, ToolGrad generates successful tool-use chains before generating prompts, yielding high pass rate.

Iterative tool-use chain generation with textual “gradients”

Standard machine learning (ML) systems improve by computing numerical loss gradients across mini-batches of training samples, which are then used by an optimization algorithm to update model weights. Recently, TextGrad adapted this paradigm for prompt engineering using an LLM critic to provide rich, descriptive feedback in plain text — feedback called “textual gradients”. These textual gradients then guide the refinements of a given prompt into a new draft that can better resolve the target task.

ToolGrad adapts the concept of textual gradients from prompt optimization to synthetic dataset generation. Rather than optimizing a static text prompt, ToolGrad uses these gradients to iteratively construct complex, valid API workflows from large tool libraries.

Comparing the optimization components of ToolGrad to traditional ML and TextGrad.

The ToolGrad framework

ToolGrad features four core modules that sequentially propose, execute, select, and update.

1. API Proposer: In each iteration, this module narrows down a sampled set of APIs into a few promising candidates to extend the current workflow.
2. API Executors: These test the selected APIs in parallel and generate detailed execution reports.
3. API Selector: This module reviews the execution reports and selects the single best-performing API call — acting as a textual gradient that provides directional feedback for improvement — and appends it to the workflow.
4. LLM Updater: Finally, this module revises the synthetic user query and AI response to match the new API set.

Repeating this iterative process results in a data sample consisting of a user query, a verified API workflow, and the final AI response.

ToolGrad generates successful tool-use chains before generating prompts, yielding a high pass rate.

Experiments

Data generation efficiency

We first evaluate the cost and quality of the data generation. We use ToolBench as our API database, consisting of 16k+ real-world APIs, to generate our tool-use dataset. We compare the original query-first data generation approach on ToolBench, using depth-first search (DFS), with our answer-first approach, ToolGrad. The results demonstrate that ToolGrad can generate more complex tool-use data with higher pass rate, using lower generation cost.

Generation efficiency comparison between the query-first approach (baseline) and the answer-first approach (ours).

Berkeley function calling leaderboard

We generated small-scale tool-use datasets called ToolGrad-500, using API databases from ToolBench. We then fine-tuned Gemma-3 models (1B, 4B and 12B) using ToolGrad-500, and we called these fine-tuned models ToolGrad-1B, ToolGrad-4B and ToolGrad-12B. We evaluated these models' tool-use performance on Berkeley Function Calling Leaderboard (BFCL), a tool-use benchmark with a different tool set from ToolBench. We compare our fine-tuned models against (1) base models without fine-tuning, (2) SoTA proprietary models (Gemini, GPT and Claude), and (3) SoTA tool-use specialized models (ToolACE, Hammer-2.1-7B).

The following summarizes our findings.

• Consistent improvements over baselines: Post-training Gemma-3 models on ToolGrad-500 can clearly enhance its tool-use performance across all tested parameter sizes.
• Outperforming proprietary LLMs: The ToolGrad-12B model demonstrates exceptional capability with its score of 83.1, making it highly competitive with the industry's most advanced proprietary models at the time of publication, gemini-2.5-pro (83.2), claude-4.5 Opus (82.8) as well as gpt-5 (74.4).
• Self-evolving: The ToolGrad-500 dataset is generated using gemini-2.5-flash-lite. Interestingly, we find Gemma-3-12B fine-tuned on data generated by gemini-2.5-flash-lite can outperform its original “teacher model”.
• Outperforming other open-sourced models: ToolGrad-12B establishes a clear lead compared to other open-sourced tool-use specialized models, including ToolACE, which was fine-tuned on a more advanced API database.

BFCL evaluation results on Gemma-3, ToolGrad models, Gemini 2.5 series, GPT-5, Claude-4.5, ToolACE and Hammer-2.1-7B models.

Conclusion and future directions

ToolGrad demonstrates that high-quality tool-use datasets can be generated more efficiently and reliably through an answer-first paradigm. By designing an agentic framework that iteratively chains APIs via textual gradients, ToolGrad addresses the longstanding cost and scalability bottlenecks in producing ground-truth data. Our design achieves almost 100% pass rate in data generation, enables relatively compact models to perform exceptionally well, and shows that student LLMs can even surpass their teachers.

Looking ahead, this research can be expanded to broader, real-world applications by scaling the framework to handle increasingly dynamic and vast API ecosystems. Future work will also explore extending this self-evolving capability to support continuous, on-the-fly learning for personalization over time. As agentic workflows become increasingly embedded in enterprise and everyday tasks, frameworks like ToolGrad lay the essential groundwork for training digital agents that are both highly capable and economically scalable to deploy.

Acknowledgements

This research was primarily conducted by Zhongyi Zhou during his Visiting Researcher tenure at Google. We extend our sincere gratitude to key contributors, Kohei Uehara, Haoyu Zhang, Jingtao Zhou, Lin Gu, Zheng Xu, Tatsuya Harada, for their support, and to Adarsh Kowdle and Shahram Izadi for their strategic guidance and thoughtful reviews.

How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks

Low reported scores on leading physics benchmarks, including those featured in the Artificial Analysis Intelligence Index (2026), suggest that frontier language models still struggle with advanced physics, a demanding test of their scientific reasoning and quantitative problem-solving abilities. Yet this impression does not always align with domain experts' experiences using these models in their work. We revisit these reported findings by evaluating frontier models on six widely used physics benchmarks and auditing them with experts, focusing on text-only problems with verifiable final answers. For each subfield of physics, faculty and graduate researchers with relevant expertise carefully review problem statements, reference solutions, and model responses to distinguish genuine model errors from grader errors, incorrect reference solutions, and ambiguous or underspecified questions. Most audited cases initially evaluated as incorrect reflect these benchmarking issues rather than errors in the models' physics reasoning. We then ask experts to address these benchmarking issues by correcting erroneous reference solutions and repairing or excluding flawed questions. We find that GPT-5.6-Sol's measured mean@4 rises from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark, while its corrected pass@4 reaches 94.4% on the 54 retained CritPt challenges. Corrected scores are computed on the retained evaluation subsets following expert review. Scores on the audited subsets of UGPhysics, PRISM-Physics, and PHYBench also rise substantially after correction. These findings suggest that current benchmarks substantially understate frontier models' ability to solve well-posed physics problems. Near-saturation on these closed-ended tasks highlights the need for more demanding, expert-validated evaluations.

Full article content is not available for inline reading.

Read the original article →

Managed Agent Architectures: Why Frontier Labs Are Rebuilding the Agent Loop

Recently, OpenAI released the Agents API, giving developers API access to the managed Codex harness. One of the strongest agent harnesses out there is now available as infrastructure developers can build applications around.

There is an important architectural idea behind the launch. OpenAI says taking advantage of new model capabilities often requires changes to the harness, so it plans to maintain and improve the Codex harness alongside its models. Anthropic makes a similar argument with Claude Managed Agents.

The frontier labs are building the model and harness together. And, more importantly, they are claiming they can build better harnesses because they control the models, and perhaps better models because they control the harnesses.

OpenAI and Anthropic are not the only companies offering managed agent infrastructure. AWS has AgentCore Harness, and Microsoft has Foundry Agent Service. Vercel is approaching the same part of the stack from another direction with AI SDK and infrastructure designed to run around agent harnesses. Unlike OpenAI and Anthropic, AWS and Microsoft can provide the managed harness without requiring the model underneath it to come from the same provider.

This gives agent builders two related architectural questions. How much of the agent loop should you still own, and does the harness need to come from the company building the model?

Understanding what these platforms put inside the managed harness can help decide how much of your agent stack you want to hand over.

1. The Agent As a Declarative Resource

One of the clearest patterns is treating the agent itself as a declarative resource. OpenAI’s Agents API can create a Codex agent in a single API call by specifying the task, model, tools, environment, and multi-agent settings. OpenAI operates the harness underneath that declaration rather than requiring developers to implement the loop themselves.

AWS AgentCore Harness takes the same idea even further. AWS lets developers override the model or tools for an individual invocation without changing the underlying harness definition. If the managed abstraction gets too restrictive, the harness can be exported to Strands code and run through AgentCore Runtime instead.

Microsoft has a similar split in Foundry Agent Service, where prompt agents run from a declaration and hosted agents let developers provide the implementation.

The managed harness moves orchestration from application code into configuration, similar to the shift Kubernetes and Helm brought to infrastructure. Developers only need to own the agent loop when they need behavior the managed harness can’t express.

2. Release Engineering, But For Agents

As agents graduate from local experiments to actual production use, they need their own release process. AWS gives AgentCore harnesses immutable versions and named endpoints, so updating the model, a tool, or a skill creates a new version rather than mutating the existing one.

AgentCore also supports A/B testing between agent versions using production traffic. One test might compare two prompts. Another might compare different models or completely different agent configurations.

Microsoft is building similar deployment primitives into Foundry Agent Service, where agents can be versioned and published behind stable endpoints instead of exposing a mutable development configuration directly to an application.

OpenAI is handling versioning at a slightly different boundary. The Agents API gives developers versioned access to Codex harness capabilities with model launches while OpenAI maintains and updates the harness itself. That is not the same as AWS giving developers immutable versions of their own agent configuration, but it establishes a separate release lifecycle for the harness underneath the application.

3. Model Selection Can Happen Inside the Session

Conventionally, agent architectures attach a model to an agent, so changing the model means changing the agent configuration. Some managed harnesses are starting to separate the two, allowing the agent session to stay fixed while the model changes underneath it.

AgentCore Harness allows a single harness to use Bedrock models, OpenAI, Gemini, or other compatible providers. AWS also lets developers switch models between turns of the same session without losing the conversation. One example would be to do planning with one model and execution with another.

Microsoft is taking a similar approach in Foundry Agent Service. Its model router can select a different model for each request inside the same conversation, routing simpler turns to cheaper models and more complex work to stronger ones.

In both cases, model selection is becoming less tightly coupled to the agent definition. The runtime can choose which model handles the next piece of work without rebuilding the conversation or changing the application interface.

That opens up routing based on task type, price, latency, or measured performance. A single agent session can use several models for different jobs without changing its identity or application interface.

4. Tools Are Moving Outside the Agent

The new Codex Agents API already treats tools as resources that can be discovered and loaded independently from the harness. Agents can connect to MCP servers, custom functions, and built-in tools, while OpenAI’s tool search loads the relevant tool definitions only when they are needed rather than putting the entire tool surface into context.

Microsoft pushes the separation further with Foundry Toolboxes. An organization can define a collection of tools independently from a particular agent and expose that collection through a managed MCP endpoint, with authentication and governance handled at the toolbox level.

AWS is taking a similar approach with AgentCore Gateway. APIs and MCP servers can sit behind shared infrastructure and be consumed by many different agents.

The agent can change without changing the tool layer, and the tool layer can change without redeploying every agent that uses it. Authentication and access policies can stay with the tools instead of being implemented separately inside each agent.

At scale, wiring the same systems into every agent separately gets wasteful. GitHub access should not need to be implemented 50 different times, and neither should Salesforce, internal databases, or company search. These platforms are moving common integrations into shared infrastructure that many agents can consume.

5. Skills Can Live Outside the Agent Definition

Unlike prompt text embedded directly inside an agent, skills can be maintained separately from the agent that uses them. The harness can discover and load these skills when they are needed rather than putting all of those instructions into the agent definition.

Codex has a concrete version of this pattern. OpenAI-hosted sandboxes for the Agents API can be configured with files, packages, skills, and plugins, and the API’s own example points the agent at a directory of skills as part of its environment configuration.

AWS AgentCore Skills package instructions and supporting resources into reusable units that can be attached to a harness. They can come from Git, S3, or AWS’s managed catalog.

This separates the procedure for doing a piece of work from the agent that performs it. A company could maintain one skill for competitive research and another for incident investigation, then make those skills available to many different agents.

With managed agent frameworks, the agent definition can stay relatively small while more of its capabilities are maintained separately.

6. The Optimization Loop Around the Agent Loop

Microsoft’s Agent Optimizer evaluates an existing agent and generates alternative configurations. It can change instructions, improve tool descriptions, modify skills, or recommend a different model.

The candidates are evaluated before a developer promotes one. AWS is assembling similar machinery in AgentCore, where production traces and evaluation results can produce recommendations for prompt or tool-description changes. Those changes can then be tested against the current version using live traffic.

In doing so, these frameworks are putting the agent loop inside a second loop that evaluates and changes the agent itself. The outer loop uses production behavior to generate a new agent configuration and test it against the current version. A human still approves the new configuration before promotion, but the platform can already generate and evaluate it.

OpenAI is not exposing this same customer-facing optimization loop in the Agents API, at least today. But it is doing a related form of optimization one level lower by continuously improving the Codex harness itself and shipping versioned harness capabilities alongside new models.

7. The Harness as an API Boundary

Anthropic makes an important design goal of Claude Managed Agents explicit: developers should program against stable interfaces while Anthropic retains the ability to change the implementation underneath them.

OpenAI is taking a similar approach with the Agents API. Developers call a managed service while OpenAI operates what it describes as an evolving Codex harness. OpenAI says it will maintain and continuously improve the harness alongside its models, with versioned access to new harness capabilities as models launch.

A managed harness puts much more than inference behind the provider’s API. It may make many model calls while working on a single task, call tools along the way, or use subagents without exposing every internal decision to the application. Codex also handles context compaction inside the harness, so an application can span multiple context windows without implementing that machinery itself.

OpenAI or Anthropic can then ship model and harness improvements together. For example, a new model capability can arrive with changes to context management or tool use without requiring every application developer to update their own orchestration.

Who Should Actually Own the Agent Loop?

Managed agent APIs move the developer boundary up from the model API to the agent harness. The provider now has room to change the machinery around the model as the model itself changes.

Frontier labs have a unique advantage here because they can build the model and harness together. Changes to context management, tool use, planning, and other parts of the loop can ship alongside the model capabilities they are designed to support.

That does not mean the provider will always build the best harness for your application. A harness designed around a specific product can take advantage of its workflows, tools, data, and constraints in ways a general-purpose managed harness cannot.

For some products, those optimizations may outweigh the advantage of having the harness developed alongside the model.

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

Problem

We gave Claude Fable 5.1 an open task: solve Sir Thomas Urquhart’s Cyphral Distich. It appears to have actually solved it, and the solution is quite embarrassing for humans in hindsight.

At the end of Urquhart’s Logopandecteision is a cryptogram consisting of two lines of 32 numbers each, called the Cyphral Distich. A cryptogram is a short message deliberately encoded so it can’t be read without knowing the rule that produced it. Here the entire puzzle input is these 64 numbers, and the goal is to recover the hidden plaintext:

5.3.27.38.32.14.21.8.66.8.70.39.5.9.12.18.2.3.56.5.1.7.3.2.13.19.3.25.9.3.16.6.
25.15.13.6.11.20.5.1.2.12.1.20.20.49.20.20.35.33.4.6.8.35.5.33.5.5.18.10.3.11.32.42.

This cipher has remained seemingly unsolved for centuries. It was posed as an open problem in Notes and Queries in 1899, appeared again in 20th-century cryptography literature, and was later listed by historical-cipher researcher Klaus Schmeh among his Top 50 unsolved encrypted messages.

Various people attempted to decipher it, but it seems they were missing one crucial hint. They tried methods like frequency analysis, substitution, and homophonic substitution, and none of these approaches worked.

That’s because they missed one easy clue.

Solution

After 44 minutes, 176k tokens, and zero interjections from me, Fable 5.1 arrived at a solution. It tried a few approaches, but was finally able to solve it with two central realizations.

First: the cryptogram is printed immediately after Urquhart’s 32 Proquiritations, and Urquhart even goes out of his way to emphasize that number. I know, surprising. He says:

“there can no number like that of two and thirty … be pitched upon”

Second: the poem accompanying the cipher promises that an honest reader will find in it “his own heart’s wishes, and the Author’s minde.” The Proquiritations themselves repeatedly conclude with formulations like “is the desire,” “wish,” or “hope of.”

If you put these clues together:

32 Proquiritations. 32 numbers in the first cipher line. 32 numbers in the second. “Wishes.”

Most historical attempts assumed the key was external: a cipher alphabet, or some mapping of numbers to letters or words, that had to be reconstructed from outside the text. But the key was not an external cipher alphabet at all. The key was the book itself.

The rule was simple: for the i-th number in a cipher line, go to the i-th Proquiritation, use that number as a word index, and take the first letter of that word.

With this, you get:

O GOD UPHOLD KING CHARLS THE SECOND AND
MAKE HIM THE SUPREME RULER OF THIS LAND

And the result is extremely self-verifying. Each line contains exactly 32 letters and ends and / land (a rhyming 2 line verse), consistent with the promised distich. It also makes historical sense: Urquhart was a committed Royalist. Hiding a prayer for Charles II in the text is entirely consistent with his politics.

Urquhart left a second, much larger cryptogram in the same style — the Cyphral Octastich in The Jewel (1652), 285 numbers instead of 64, and just as unsolved. From this, Fable 5.1 was also able to decipher it:

Result: the Cyfral Octastick is solved (all but nine letters)

Rule. The Jewel (1652) has exactly 284 numbered pages, and the octastick + decagram contain 285 numbers. The k-th number (counting straight through the eight lines and the Decagram) is a word index into page k of the book; take the word's first letter. Same idea as the Distich (number i → Proquiritation i), with pages instead of paragraphs — and, as in the Distich, Urquhart almost always picked the first word on the page starting with the letter he needed (231 of 275 readable positions are exact first-occurrence hits in the EEBO-TCP text; the other 44 are 1–3 words off for identifiable transcription reasons — hyphenated words at page tops, hyphenated compounds, "&", paragraph numbers, an untranscribed Greek phrase).

Plaintext (ottava rima, ABABABCC — a royalist prayer written in London, March 1652):

GREAT LORD, MANTAINE THAT REGAL FAMILIE
WHEREOF KING CHARLS THE SECOND IS THE HEAD,
AND GRANT THAT HE MAY BEARE THE SUPREME SWEIGH
WHERE ENGLISH, SCOTS AND IR[I]SH ARE BORNE AND BRED,
AND [·········] THIS USURP'D AUTHORITIE
REIGNE IN HIS ROYAL PREDECESSORS STEAD;
LET HIM BE OUR SOLE CESAR, ARTUR, HECTOR,
OUR EMPEROUR, KING, MONARCH AND PROTECTOR.
                        AMEN, SO BE IT.   (the Decagram)

Elicitation

I’ve actually been trying for the past few months to elicit models into solving an important but unsolved cipher. Across those months, no other frontier model I tried produced a verified solve.

How I elicited Fable 5.1 was quite simple. I gave it a goal of sorts. I asked it to solve an unsolved cipher. I gave it some encouragement. I told it to look online at some of Fable’s strongest feats, especially the math problems it has solved, and that something like this should be easy in comparison. I told it to think creatively and really analyze the problems it encountered.

I gave it two constraints. First, I asked it to avoid ciphers that already had solutions or could support many plausible answers. I suspect a lot of historical unsolved ciphers aren’t quickly verifiable and may be vague in the sense that their creators are long dead, so we might never truly know whether a proposed answer is correct.

Second, I steered it away from the absolute hardest problems—ones where thousands of humans, or even organizations like the CIA, had already put in serious effort. For example, Kryptos K4 might be a little too hard and convoluted for current models. That might be a future experiment, but I don’t think Fable 5.1 could solve it in a reasonable amount of time yet.

Fable 5.1 spent some time looking over different problems. It knew when to stop. It knew when a problem wasn’t budging. And when it found this particular problem, it noticed the clue almost immediately.

Now, I don’t think other frontier models would necessarily fail to solve this problem. The clue is actually extremely simple. I think what Fable did well was notice that this particular problem stood out as unusually tractable.

Takeaways

I think this demonstrates that models can solve problems not just in mathematics, but also historical mysteries, forgotten conjectures, archival puzzles, and things like that.

Historically, many of these problems were bottlenecked by human attention. Someone had to care enough to spend hours or days reading obscure material, testing unpromising ideas, tracing references, and trying things that might go nowhere.

That bottleneck is seemingly disappearing.

And the interesting thing about Claude solving this is that it didn’t perform some extraordinary feat of cryptanalysis. It’s actually the opposite.

The answer was simple in hindsight. It just kept looking until it found it—and that persistence might show up in many other areas.

Full article content is not available for inline reading.

Read the original article →

The Rise of the Forward Deployed Engineer — and How To Do the Job Right

FDEs have the hottest job in AI. Labs, startups and PE firms are all hiring engineers to sit inside their customers’ operations and solve their problems. Almost none of them agree on what those engineers are supposed to accomplish, or what the strategy underneath the hiring actually is.

I’m Vinoo, CEO of Kepler, the deterministic infrastructure for AI. I’ve built pieces of the forward deployed function three times, at three different institutions, over the course of over a decade. Here’s what I’ve seen work, what I’ve seen fail, and where I think this goes.

The first was Palantir. I started there on product development, building storage and retrieval systems, and was later deployed as an FDE across commercial, DoD and NatSec, healthcare, and oil and gas. I also led Project Frontline, the rotation that took our software engineers and turned them into forward deployed engineers. Around 250 people went through this program, and a lot of them run forward deployed teams now at companies like OpenAI, Anthropic, xAI and Anduril.

The second was Citadel, where I ran business engineering. Our customers were portfolio managers, and the only question that mattered was whether the data and software products we built helped them generate alpha.

The third is Kepler, where the forward deployed function sits inside product rather than sales, in a domain where a plausible wrong answer is worse than no answer at all.

FDE misunderstandings

A few months ago, a16z launched the Forward Deployed Engineer Fellowship and I was nominated as one of the fellows, alongside a handful of people I used to work with. It’s a great program and I’ve enjoyed so many of the conversations. Last week I went to my first fellow dinner in SF.

Around the table were FDEs from Snowflake, Anthropic, and a number of startups I’d been reading about, and over the course of the evening it became clear that we were all using the same two words (forward deployed) to describe jobs that had almost nothing in common. In one part of the conversation an FDE was a sales engineer who joined ‘the second call,’ somewhere else it was a quota-carrying rep who could write Python, and a few seats down it was closer to a consultant with a laptop and a statement of work, brought in to deliver something the product couldn’t.

A few days later, someone earnestly asked our WhatsApp group how their FDE team should split scope with the consulting firm already sitting in the account. That’s a reasonable question to ask, but a strange one to have to answer, at least based on my own belief about what constitutes an FDE.

To be clear, I’m not interested in gatekeeping a term; and meanings shift, this one faster than most. But what’s interesting is that folks in this group, the current experts at FDE, are describing fundamentally different jobs, with different reporting lines and different incentives. It’s no wonder half the comments on any YouTube video about FDEs are some version of “isn’t this just reinventing consulting?”

So in the rest of this article, I will tell you the story of Project Frontline, through the narrow lens of a mistake I helped make, how that mistake turned me into an FDE, and how it eventually informed the rotation that turned our software engineers into FDEs.

The history of Project Frontline

First, some context. From nearly the beginning, Palantir was split into two separate functions. The first, Product Development (PD), built the platform. The second was Business Development (BD), which despite the name contained both the technical BD folks (already called FDEs) and non-engineering customer-oriented folks (we called them Embedded Analysts, or Deployment Strategists).

PD, in the vast majority of situations, wasn’t directly engaging with customers; and BD, in the vast majority of situations, wasn’t directly contributing to building the core, generalized platform. PD tended to do customer discovery secondhand, by chatting with BD or by consuming the successful build-in-the-field features into the core product. None of that was a process, though. It ran on relationships — such as which FDE happened to know which PD engineer well enough to grab them. So a good insight from the field made it into the platform (or was dropped) depending on who was in the room.

In 2013, in my early days at Palantir, I got to work on a transaction store called Phoenix. The store was designed by some of the best engineers I’ve ever worked with, and it had an abundantly clean design scoped to a clear set of customer use cases. The use cases, though, had been relayed to us second-hand. We knew and understood the design requirements, which had a focus on the commercial requirements of retention periods, and had clever solutions to bucket data in a way that enabled storing a rolling window of data. It behaved exactly as specified in every environment we controlled.

Then we deployed it at a bank, and real financial data turned out to have holes in it that our test data never did. A blank timestamp fell through to the epoch, so the retention logic dutifully requested a ten-minute bucket for every window between January 1st 1970 and the present day. That came out to some 2.3 million keyspaces against a system where Cassandra (the backing tech) needed roughly five megabytes per file handle. The server rightfully OOMed [Out-Of-Memory] and starting it up again would have required 14 terabytes of RAM. Meaning this process was effectively dead on arrival.

The root cause here wasn’t a lack of user research, as you might guess. We had a spec, we understood our use case, and we had read plenty about how institutions like this store their data. What we had never done was stand inside the building while the system ran against their production data. This meant that nobody on our side owned the gap between the design and the daily reality. Everything we knew about that bank had been relayed secondhand and by well intentioned people for whom bad data was just another normality.

That’s how I became an FDE, which is a generous description of what actually happened. As Phoenix rolled out across Palantir’s commercial fleet I found myself flying out to fix what we’d shipped, and that put me in front of our actual users for the first time. In this case the users were Palantir’s own FDEs, which was lucky for me, because they could tell me what was wrong in the language I already spoke. I started building and expanding systems in service of what they were trying to do.

So this is also the story of how I learned the FDE mindset viscerally rather than intellectually.

This is where the ordinary version of this story ends, with some lesson about paying attention to your users. Phoenix turned into something more interesting than that. It became a platform, and Palantir’s FDEs started building on top of it across cybersecurity, KYC, AML, and a long tail of use cases nobody had scoped for. Eventually, we (Product Development) had to think about how to expand the Phoenix platform to support all of these use cases.

I didn’t see it at the time, but that iteration cycle is the whole idea. An FDE solves customer problems in order to earn the insight that informs what gets built next. The role is an extension of the product team.

FDEs today

The reality is that none of this is the mentality of the vast majority of FDEs you see today. The term has been co-opted to mean something close to “a person who does something that vaguely involves a customer,” which is how you end up with job posts for a forward deployed equity researcher, or a forward deployed sales engineer. The instinct underneath the co-option is correct, even when the titles are silly, because customers matter more now than they did five years ago, and they matter more for a specific reason.

The low-hanging fruit is gone. The problems that could be solved by a well-designed product sold identically to a thousand companies have largely been solved. What’s left is the work that sits inside the walls, in workflows that are messy and undocumented and nearly impossible to proxy from the outside. That’s why everyone is suddenly “forward deployed.” You cannot infer from a discovery call how a specific company closes its books, and the part of the problem that resists inference is now the part that’s left.

Which means the holy grail has quietly moved. For a long time it was the repeatable motion, the same SaaS product sold the same way over and over; and that’s still the right ambition if what you sell is tokens or bytes or something physical. For everyone else the value has migrated to customization, to the last mile, to the twenty percent of the workflow that no product could have anticipated and which determines whether the other eighty percent gets used at all. Being forward deployed has become synonymous with solving that last mile.

But solving it is only half of what the role is for. The last-mile problem you solve at one customer is the signal that tells you which piece of your platform needs to become generalizable. An FDE function that solves last miles without ever sending that signal home is a services/consulting team with a better title.

So what are today’s FDEs supposed to be doing?

I’d contend that your job as an FDE should be to collect nouns and verbs. Let’s break that down.

Spend a week inside a company and you’ll notice that the same concept usually has at least four different names. Sales says customer, ops says client, finance books a billing entity, engineering writes org_id, and every seam between those teams hides a translation that breaks the moment somebody changes a definition. Those names are the surface and underneath them is the operating model. Meaning, you can really proxy the way a company works by learning their nouns and verbs.

The nouns are what the people in a business treat as real. It’s usually a “thing.” A position, or a trade, or a counterparty. Usually, on a per-team basis, there are a handful of objects the whole operation turns on, and none of them are defined the way a textbook would define them. That’s because two firms will describe a position identically on a slide and completely differently in the code. That’s not a bug, that’s just what makes companies unique. I mean that if every company had the exact same set of nouns, then you would really just need one company.

The verbs are how nouns move. Things like how a trade gets booked, or what has to be true before the books can close, or who signs off on an exception at eleven at night and what happens when that person is on vacation.

Almost none of this is written down — it’s lived. It’s the system of operations through which an organization lives. It’s culture. It lives in the heads of the six people who have been there long enough to stop noticing it, and in a spreadsheet somebody built four years ago that the entire team now quietly depends on. That’s why it’s worth so much, and it’s also why you can’t ask for it.

Usually, the people who hold this knowledge don’t know they have it. In one of my last startups, we spent close to a year trying to move a customer from CSV to Parquet, and one data quality engineer blocked it every single time. We could never understand why and the reasons would always change, but would always be some variation of “a parquet is worse,” “it doesn’t work,” “it doesn’t make sense to me,” et cetera. We used the customer storage reduction argument, the compute minimization argument, the pipeline optimization argument…and none of it moved her, because none of it was about the actual problem.

Then we had one of our FDEs go in and watch this particular data quality engineer work. She was pulling CSVs down from S3 onto a Windows laptop, double-clicking them open, and eyeballing the rows. That was the data quality check. Parquet had no native viewer at the time, so what we were proposing would have taken away the only data quality instrument she had and handed her nothing back. She wasn’t being difficult, she was just protecting the one thing that let her do her job.

We built a Parquet viewer that night, she approved the migration two days later, and pipeline execution went from about seventeen hours to two. She would never have said any of this in an interview. From where she sat, the reason was obvious and not worth mentioning.

Understanding and defining the system of operations, or nouns-and-verbs, of this analyst enabled us to not just understand the problem, but build a solution that we could then deliver across a fleet of customers with the same problem.

The output needs to be a product

Understanding the nouns and verbs contextualizes problems, but the output needs to be a product rather than just one happy customer.

The nouns and verbs tell you what a problem actually is. They don’t tell you what to do about it; and this is where most FDE functions quietly go wrong, because solving the problem in front of you is satisfying and legible, and someone will thank you for it that same week.

Keeping the customer happy is a real job and a good one. It belongs to solutions architects, who are rightly measured on it. The forward deployed engineer is there to turn what the field teaches into the thing every future customer gets. An FDE engagement that ends with one delighted account and nothing changed upstream has failed at the only thing the role exists for. You got the context and you spent it locally.

I learned that one expensively. In one case a customer needed a data retention job, so I hacked together a groovy script named “vinoo.groovy” to hold them over — an afternoon of work that was never meant to survive the week. A year later, it was running across a customer of nearly a hundred thousand people, with my name fused to it. It became such a ridiculous story that my team started calling me vinoo.groovy. We fixed the problem, but never turned the fix into a product — so we spent years maintaining a hack that should have died immediately. Every shortcut you ship becomes something you own. The discipline is knowing which fixes belong in the platform and which ones you throw away on purpose the moment they’ve done their job.

The fork

This is where the whole thing splits. Do the work with nothing underneath it and you learn one company’s model, ship something shaped exactly to it, and lose all of it when the engagement closes. The next customer starts from zero, and so does the one after that. That’s consulting. It pays well, the people are excellent, and it doesn’t compound.

Put a platform underneath the same work and every company you map makes the next deployment faster and the product sharper, because what the engineer brought home has somewhere to live. That’s the difference between selling hours and building an asset, and my honest read of this gold rush is that most of the companies in it are building the first one and describing the second to their board.

That’s your job: build the platform.

What we do at Kepler and what you can take from it.

At Kepler, we set the function up this way from day one, before we had the customers to justify it. The alternative is to discover in month fourteen that your engineers have been optimizing for the wrong thing. From the beginning, our FDEs act as an extension of the product team; and that is the structural decision everything else follows from.

We sell to hedge funds, investment banks, PE firms, and other financial institutions. These are fundamentally different institutions with different mandates, but all of them share a single non-negotiable: numbers have to be right, and someone has to be able to show why they are right. That is the constraint we design against and it turns out to be a useful one, because it forces the operating model into the open. No firm we’re involved with can produce a work product without a clear trail of provenance behind every number in it. That invariant defines our platform and gives us a bedrock to execute against.

These problems are universal. The vocabulary is not.

Every one of these firms is running some version of the same ontology underneath, and every one of them describes it differently. A position means one thing on a credit desk and something adjacent on an equities desk at the same bank. Two funds will use identical language for a return calculation and disagree about what goes into the denominator. Most of these differences exist because somebody made a reasonable decision in (say) 2011 and the decision outlived the person; also, it’s not written down anywhere that you can find.

Identifying and filling that gap is the job of an FDE. A schema tells you what is stored. It does not tell you what is meant, and the distance between the two is exactly where a system that sounds right produces a number that is wrong.

Provenance is a correctness requirement for our customers, but for us it does something else as well: it makes the field work compound. A system that can improvise around a bad encoding will never tell you the encoding was bad. Our system does not improvise. When we misunderstand how a firm defines something, that misunderstanding surfaces as a failure rather than as an answer that merely looks reasonable. The engineer who got it wrong finds out from the system, rather than from a client in a meeting six weeks later.

The deployments then tell us what to extend in the platform, which is a narrower question than it sounds. We are not trying to learn which feature a given fund would like to have. We are trying to find the places where the platform is too narrow to hold what we keep running into. Three firms asking for the same feature is easy to notice and worth relatively little. Three firms needing something the provenance layer cannot express is the signal we actually care about; and it usually arrives quietly, in the form of an engineer working around the same limitation for the third time.

If you are building somewhere else, here is the part I would take from all of this.

Product leverage is what buys you the right to experiment. Every capability that lands in the platform makes the next deployment cheaper to attempt, and cheap attempts are how a small company learns anything at speed. Without that leverage, you get one expensive guess per customer. You scope carefully, build for months, and if the guess was wrong you have spent an account and a quarter finding out. We would rather be wrong four times in a month, because each of those attempts costs less than the one before it.

Which is why the reporting line is not an administrative detail. Point the function at sales and the incentive becomes closing the account in front of you — which is a real job and one that somebody at the company should be doing. It is not this one. Point the function at product and every deployment is asked to produce something the next deployment can start from.

Where the moat is

So here’s where I’d put the moat in this era. It isn’t the model, which cheapens by the month and which you’re renting from somebody else regardless. It isn’t the talent either, because every lab is bidding for the same few hundred people and that price has already been discovered.

It also isn’t the map of any one customer. That was true even a few years ago and it’s the same now, because extraction is nearly free and anyone can draft how a firm operates in an afternoon.

The draft is not the asset. Knowing which parts of it are wrong is the asset, and that only comes from having been corrected.

So, for us, the moat is the accumulated, current, verified understanding of how firms in a vertical actually operate, held in a platform that keeps it current and can prove it. Each of those words is load-bearing. Accumulated, because one deployment is an anecdote and the tenth is a pattern. Current, because operations drift and a stale model fails silently underneath an AI system in a way it never did in front of an analyst. Verified, because a plausible encoding and a correct one look identical until something breaks, and the whole point of insisting on provenance is that you find out which one you have.

That is not purchasable. A competitor can hire your engineers, copy your interface, and read this article (ours try to do all 3!). What they cannot shortcut is the sequence of being wrong inside a customer, being corrected, folding the correction into the platform, and arriving at the next firm already knowing which questions are load-bearing. Every cycle of that makes the next one cheaper, and that compounding is the thing you own.

I’ve watched this function get built three times and the pattern held every time. The engineers who mattered weren’t the ones who shipped the most for customers, but the engineers who came back and changed what we built.

Hiring forward deployed engineers buys you exactly one thing, which is the right to identify which problems are worth solving. Most companies never get that far. But it’s the entry fee, not the prize.

I’m Vinoo Ganesh, CEO of Kepler, where we’re building the layer this piece is about, the ground truth that lets an AI product trace every number back to source. Before Kepler I led Spark at Palantir and built Project Frontline, then ran business engineering at Citadel. If you’re building here, or you think I’ve got a piece of this wrong, you can argue with me on LinkedIn.

If coding is solved, what now?: Measuring the sloppiness of code

LLMs have become almost perfect at generating code, but that isn’t the end of the story. Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC).

This results in a loss of human agency, because in projects that are adding millions of LOC per month, it is hard for humans to keep up. Some people might say that that is not an issue at all, because they trust their agents to deal with it. I have bad news for you, agents can't really deal with the slop either.

Coming from a physics background, I always had an experimental/quantitative approach to solving problems. When I started at Earendil, with the task of figuring out how to measure code sloppiness, my natural instinct was to first take a deep dive into the literature and then check what other companies were doing.

To be frank, with the exception of a few insightful research papers, I was disappointed at how “vibes based” the industry seems at the moment. In my research and on X, I was constantly bombarded with messages such as “End-to-end coding agents”, “AI that doesn't just suggest code—it ships it” or “Human-level evaluation without human-level cost”. Which like all good tales, have a grain of truth in them.

LLMs are able to write almost perfectly correct code. This is because of the scalability and the verifiability of code. It is pretty straightforward to let LLMs generate code and then let that code be checked by hidden tests, which results in a clear reward signal. In stark contrast to that, checking the ‘sloppiness’ of this code often requires human intuition and taste, and is an extremely difficult task in general. I think the best way to illustrate why that is, is by going through possible ways of measuring slop.

AI as a judge: This is probably the most common way of evaluating code quality in the industry and from my observations it rarely works. The more sophisticated approach, namely trying to give the judge model two solutions A and B, and then letting it decide which solution it prefers, has the downside of the model changing its preference when you rename the solutions. Asking LLMs to judge the code they write is not a substitute for a proper evaluation.

Human judges the AI: If we ignore the fact that there is huge diversity in the quality of software-engineers, this would be the best solution to assure that the code stays human readable. With the downside being that this is not scalable for training AI or having large benchmarks with multiple model providers and harnesses.

The simplest method: In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure.

The next two measures were introduced to me by the paper SlopCodeBench, and seemed promising because they were able to separate legacy code bases from LLM-slop quite well.

Verbosity: Tries to measure the amount of duplicated and unnecessary verbose lines.

Verbosity = |AST-Grep flagged lines ∪ clone lines| / LOC

Erosion: Tries to measure how much of a codebase's mass is concentrated in a few large and complex functions.

mass(f) = CC(f) * sqrt(SLOC(f))

Here f is the function, SLOC is the source lines of code and CC(f) is the cyclomatic complexity of the function.

Erosion = Σ(mass(f) for CC(f) > 10) / Σ(mass(f) for all f)

The erosion is then simply the fraction between the mass of the functions with a cyclomatic complexity larger than 10, by the mass of all functions.

If we look at the average verbosity and erosion of the code generated during the SlopCodeBench evaluation and compare that to a set of established repos there is a stark difference between them. On average the verbosity in the repos is 0.15 ± 0.06 and in the agents code is 0.33 ± 0.10. For erosion the repos achieve 0.31 ± 0.17 and the agents 0.68 ± 0.20. The agent's code is on average roughly twice as verbose and eroded as human code.

To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, they create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. The result of that is that bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate. Which should be a warning sign to everyone who happily adds tens of thousands or even hundreds of thousands of LOC a day.

In exploring these metrics I hope you now have a clearer picture of why it is challenging to evaluate code sloppiness and why human intuition and taste are still either implicitly or explicitly baked into the evaluation.

There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that.

Why the world's best AI startups write bad prompts (& how to fix this)

Most prompts are bad because prompt evolution tends to be accretive: we only add, never remove, over time. This leads to spaghetti prompts, with contradictions and ambiguity. This has real business impact.

We need to treat prompt changes as product changes (because agent behaviour is product), and treat prompts as code (modularised, MECE, and all your other favourite acronyms; refactored if need be & actively maintained).

Well structured prompts enable teams to move faster, prevent regressions, and have better agents. I propose a very simple structure at the end.

Prompting decisions are product decisions, and using structure to make unambiguous, maintainable prompts is critical for making great agents. This is a guide on how to do that.

Background

I think I have one of the best jobs in the world. I lead Applied AI Engineering for the OpenAI startups team across EMEA & APAC, and that means that every week I get to see behind the scenes of the best AI startups globally. And I get pretty hands on in how I work with their engineers to improve their agents: on everything from prompts to evals to finetuning.

These startups are advanced. Some have ARR in the hundreds of millions. Some have their own data annotation teams. Some train their own models.

So it came as a surprise that often, when I look behind the curtains, they have prompts that just don't make sense. This is not about being beautifully written prose, or nicely formatted; it's about logical errors that lead to the mistakes their agents make.

This is not a critique of those startups - indeed, they are more successful than any company I have ever built, and their teams are full of the best engineers globally. They are a true pleasure to work with.

But they're leaving huge gains on the table. After only a couple of days re-writing agents together, I've seen some startups speed up their agents by 50%; others increase 7 day retention by 40%; and still others reduce costs by 30%. These results hold across the LLMs they use, from every provider. When you're talking millions of ARR & LLM spend, this is pretty material.

Don't believe me? See this Loveable engineer's post about how he decreased their LLM spend by $20M per year… because his mum caught inconsistencies & duplication in their prompt.

In fact it's because they are so incredible, that I'm writing this. Because clearly even when you are genuinely world class, our current paradigm for prompting leads to suboptimal results.

So I thought I would try to scale my impact beyond the startups I can work with directly by writing this. First, I'll cover why the world's best startups write bad prompts; then, I'll cover my prompting philosophy; finally, I'll propose a prompt template.

Bad prompts

Given this is so common, there are clearly universal tendencies that lead to bad prompts. The two key issues are contradictions and ambiguity.

Our current process for prompting is accretive & leads to contradictions

Most prompts evolve like this: the first engineer building an agent writes a simple prose prompt. As the startup grows, and the agent is required to do more things, they add to the prompt. Errors occur, so they add a few lines to fix those.

The prompt only gets longer.

And no-one reviews the entire prompt end to end. Almost always, that leads to contradictions in the prompt, because as you add new content, old content saying something else is kept.

Implicit knowledge leads to ambiguity

Even if an engineer does review a prompt end to end, they often don't truly read it. When you read & interpret a sentence, you don't only use the words on the page. You use all of the knowledge you already have to make sense of those words. And that's a problem, because you often know what you want the sentence to say; and you read that, rather than what it actually does say.

This is the problem of specificity: the prompt doesn't actually say what we want the agent to do, because we haven't unambiguously specified it. For example, let's say I tell my agent to "never refer to competitors in [its] output". This makes sense to that startup’s engineer who spends every day thinking about my startup and its competitors. But to an agent without that implicit knowledge, that is incredibly vague - who are the competitors? What about partial competitors we also collaborate with sometimes? We haven't specified what we actually want.

Conditional prompts compound this

The above problems are compounded by conditional prompts, where additional prompt content is injected depending on the scenario. Different engineers work on various parts in separate files. And that means that even if each team/engineer reviews their prompt for contradictions & specificity, no-one reviews the whole.

The outcome? We get spaghetti prompts. Thousands of lines, with interaction effects between many paragraphs, and it's almost impossible to review them because by the final sentence, most humans have totally forgotten the first line. Or got bored and stopped.

My principles: start treating prompting as product, and prompting as code.

Prompt decisions are product decisions

Agents are at the core of product experience. In chat based products, they are the entire product. The formatting of the agent's output is the UI. For example, should it output bullet points? Or markdown?; should it always reply in English? or in the language of the user's message?

Agent behaviour is a product decision: should it err on the side of responding fast? Or do comprehensive research, making the user wait? Should it call a tool asking for the user to approve something? Or just get on with it? It's all product.

Prompting as code

We use human readable language to get a computer to do what we want. We can borrow principles from software engineering to manage prompts that grow over time.

• Structure with MECE prompt sections: MECE stands for mutually exclusive (ME), collectively exhaustive (CE). Collectively Exhaustive means your prompt sections comprehensively cover the behaviour you want. Mutually Exclusive means each prompt section should be self contained, with no overlap.
• Aim for the specificity of programming: Think of it like code. If this, then that. Specify behaviours for core branches of user requests.
• Separation of backend and frontend: Separate Behaviour (how the agent acts/uses tools) from Output (what the user actually sees).
• Refactor every so often: Dedicate time to paying down your prompt debt.

Template of a good prompt: Background, Behaviour, Output

You'll notice this is hierarchically organised, like a tree. This helps it be MECE, and means you can find sections much faster. If you're having issues with your agent's output, it's unambiguous where that content sits.

A quick note on evals

Evals force you to decide on what you want. When you make an eval, you have to specify the desired behaviour. And half the time you realise you literally just never specified that desire in your prompt, it was latent context in your brain, not explicit in the prompt, OR you actually hadn't clarified it even to yourself.

Benefits of structured prompts

• Fewer contradictions: With MECE sections, reviewing the prompt is easy. Each section is self-contained.
• Faster & safer iteration: Separating concerns lets you iterate incredibly fast without worrying about interaction effects.
• Faster search: Hierarchically organised prompts make finding relevant sections much faster.
• Everyone's an engineer: It's easier for non-engineers to contribute when the structure is clear and self-contained.
• Faster Model Upgrades: Testing model defaults or removing outdated instructions becomes a one-line change.

FAQs

Isn't this just a waste of time if our prompt works now?

No, the payback period is pretty fast. It will save a LOT of time when your agent starts making very weird and complex mistakes, because those are the hardest issues to solve in spaghetti prompts.

As models get smarter doesn't this become irrelevant?

No. Higher intelligence cannot automatically solve ambiguity and contradictions in your preferences. Only you can. You can often leave the "how" less specified, but you still need to specify what you want.

Can't we just automate this process and have LLMs write the prompts?

Somewhere, you need to specify your preferences unambiguously, whether in the final prompt, or in the 'rewriter' prompt. There's no free lunch; you need to get clarity for yourself.

Right now developer experience is measured in seconds. If your tests take a second to run, that’s good; if they take thirty seconds, that’s bad. Any faster than a second doesn’t really matter, because most of your time is spent either thinking or waiting for an AI agent to spin. Shaving milliseconds off your dev server reload time or whatever is pointless: that’s not the bottleneck.

It will be. Small models are getting faster and faster, and smart models are getting smaller. I think most engineers will still want to use the smartest available model — software engineering is hard — but we will increasingly see faster models get used as subagents or for well-understood tasks. This is largely uncharted territory. Very few people have developed intuitions for what it is going to be like to work with agents that run at thousands of tokens-per-second.

GPT-6-Astra can run at about sixty tokens per second. That means you spend a lot of time waiting for it to think. You work with it like you would work with another human: delegating a task and then context-switching until that task is complete. If you haven’t yet, have a play around with Jimmy, Taalas’ version of LLaMA-3.1-8B running at seventeen thousand tokens per second. No matter how long the response is, it arrives in the instant of you hitting send. The model is not good enough for agentic work, but it gives a glimpse of what it would be like: you would simply get your answer instantly.

Well, that’s assuming the agent’s tool calls are fast. When generating tokens is not the bottleneck, it will suddenly matter a lot whether it can read a file in 100ms vs 10ms, or whether it can run your tests in 500ms vs two seconds. Fast tool calls are going to be the difference between a near-instant response and having to wait several minutes. There is thus going to be enormous pressure to do agentic coding in languages with fast compilers and tests, like Golang, and to tightly optimize the dev loop in agentic codebases.

Teams focused on DevEx — developer experience — are largely a relic of the 2010s, when companies were incentivized to make their engineers happy. Most companies have cut them down to a skeleton crew or removed them entirely. But we may see a return of DevEx in the late 2020s, focused on speeding up the experience for AI agents.

1. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units.

2. Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model.

Confessions of an Unrepentant Slop Snob

A framework for thinking about when AI involvement is additive or a violation

This is a peek into the research behind our “AI Norms and Values” docs, which we developed over the summer and recently shared on our site.

Earlier this year, I was spending a lot of time stewing over why everyone around me seemed so frazzled and on edge.

Half the company was spitting mad about all the slop they were getting. Instead of receiving five crisp bullet points, they were getting twenty-page docs full of padding and slop. They would ask a colleague a question, and the reply would begin with “Claude says…” These folks felt like their time and attention were being not just taken for granted, but actively abused.

The other half of the company felt equally injured. They were working faster and delivering better outcomes than ever, and wasn’t this exactly what we had asked of them? They were more upset about the fact that some people hadn’t updated their workflows in years. Of course you can’t keep up if you aren’t willing to adapt, they protested.

Everyone was mad. One side wanted to place limits on AI (“I am so sick of reviewing docs the sender didn’t even read!”). The other side wanted us to force people to use AI (“I am so sick of reviewing docs riddled with basic errors that a single pass would have caught!")

I was, at different times, in both camps. But the thing that bothered me most was how often I kept hearing the word “dehumanizing”. People said they no longer felt like they had a human connection with their coworkers anymore. This was new.

On the one hand, AI is not special. On the other hand…is it?

My first instinct was to say that tools are tools. The quality of the work is all that matters — outcomes are all that matter — not how the work was made. Do our customers care if we use AI or not? Probably not. They care a lot about the quality of the product and whether it meets their needs, not so much about how we made it.

So maybe we all just need to be ruthlessly outcome-oriented. Build the best thing we can, as fast as we can. AI is a powerful and versatile tool, so we should use it wherever we can to make our work better and do it faster.

Is it really that simple? I had nearly convinced myself that it was, when I noticed how contradictory my own behavior had become.

Meanwhile, behind the scenes, I begin harshly judging everyone who sends me AI slop

At the same time I was repeating “it doesn’t matter how it was made, it matters how good it is” every day, I was developing a violent disgust reflex for AI-generated text on the side.

I’m not sure exactly when it happened. By spring, I was snapping at the tendons of any poor soul who showed up in my inbox with an “I’d value your take on this” or “the call most leaders still won’t make”.

It started with DMs and emails, but it didn’t stop there. By summer, my reactive rage-response to AI-generated text had spread to include most forms of writing. If I’m reading a newsletter and I start to sense AI-isms, I delete and unsubscribe. If I’m reading a blog post, I close the tab; if I’m on social media, I unfollow or unfriend.

More importantly, I judge them. Yes, I look down on them. If they don’t care enough about their own point of view to do the work and refine it themselves, if they don’t care enough about me to write me a note, then why the fuck should I give them a single morsel of my precious attention?

Language does many different jobs for us

Writing is thinking on paper, as William Zinsser once said. Writing is language, encoded for posterity. But language, and writing, do many different jobs for us.

Language evolved as a way to connect — person to person, mind to mind, one mind to many. In software, for example, we convert natural language into bits and bytes that computers can use to do math on. Lawyers convert language into legal text and taxonomies. The technical and professional worlds are awash in dialects where language has been abstracted from its roots as an emotional and relational tool and given functional, depersonalized meanings.

In many of these contexts, substituting AI-generated language can be wholly acceptable. No one blinks an eye if you use structured data generated by AI, or a formal proof generated by AI (as long as it’s accurate). The situations where the use of AI tends to land jarringly, causing frustration, rage, even a sense of betrayal, are the ones where the value of the communication is less abstract, and more personal or relational.

Every job language does for us is either functional or relational, or some combination of the two, and knowing which one you’re in the middle of tells you a lot about whether AI belongs there, and how it’s likely to be received.

Does the value lie in the idea itself, or the fact that a particular person said or thought something?

It’s less about how much AI is being used, and more about in which contexts people are using AI, and secondarily whether or not they disclosed and I consented to it being used there.

Sometimes, yes, the quality of the work is all that matters. If you and I are collaborating on a document or a diff, all our collective comments and edits are in shared service of making the ideas better. It isn’t about whose idea it was or which tools we used, we are just iterating and improving until it’s as good as we can make it. The use of AI here is just another tool, one of many.

Other times, the value of a piece of writing derives from the fact that a particular person said it, thought it or felt it, or its value is grounded in your relationship. Why do you care more about what your skip level says about your performance than you would care about reading the same advice in a book? Likely because this is someone you know and respect, someone with influence over your career prospects, someone who knows what you’re capable of and has a vested interest in helping you succeed. In these circumstances, people expect to hear your voice; if they don’t, they may invent all kinds of terrifying reasons why.

The more personal the interaction, the more AI-generated text can cause a loss of trust

When I started talking to my coworkers, trying to figure out why they were so angry and frustrated all the time, one thing I heard over and over was, “I asked for my colleague’s opinion, and they sent me back a Claude snippet. I wanted to know what THEY THOUGHT.” When someone wants your opinion, AI generated text registers as a violation.

Another common source of friction was performance reviews. “My performance review was obviously written by ChatGPT. Did my manager even read it, or just push a button and spit it out?” We encourage our managers to use AI to develop systems that help them become better managers, but this is a clear risk of using AI to aid in the writing or editing of reviews: if your voice is lost in the process, it may destroy trust between you.

The more any interaction is personal or relational, the more the use of AI in that context tends to cheapen it and degrade trust, unless AI has been specifically invited into that relational context.

The slick, uncanny valleyness of AI communication

AI is just a tool, and we should use it anywhere and everywhere we can to do better work, faster, and deliver better outcomes for our users. But relationships matter. A respectful environment matters too. And whereas lines of code are indifferent to their origin, and can be validated by harnesses and tests, people are intensely attuned to the language people use with them. Relationship maintenance cannot be automated.

Yes, AI is just software. But it is special in one way: the slick, sycophantic, uncanny valleyness of the way it communicates. Human-like, but not human, which is somehow vastly more alienating than messages that are plainly automated. This is why communication that sounds like AI is so degrading to trust.

In the absence of universal norms, conventions will do

A lot of things about communication that used to be clear, no longer are. And a lot of hurt feelings, anger, and frustration are roiling about in the breach. Old norms no longer apply, and new norms are not yet broadly developed or agreed upon.

At times like these, having an agreed-upon convention or standard, any convention or standard, can really help.

Spacelift Flows Is Live: Bring IaC Rigor to Day 2

AI sped up your deployments, while Day 2 stayed in the wild west. The moment you hit apply, the rigor stops. What happens next, the alerts, the provisioning requests, the incident response, the cleanup, runs on whatever anyone could glue together: a Lambda function nobody remembers deploying, a Slack bot only two people know how to restart. Scattered automation nobody can see, and knowledge that walked out the door with whoever built it last. That’s Day 2 operations as it exists today.

Spacelift Flows is built to close that gap. As of September 8, Spacelift Flows is live and ready to use.

See everything you're running

Flows runs on a visual canvas. You wire blocks together, HTTP calls, data transforms, conditions, and schedules, then connect them into a flow you can actually read. Every run is logged and audited from the first execution, with a step-by-step trail.

When something breaks, you open the flow and see exactly where it stopped. You stop finding out from production.

Stand up automation in an afternoon

Building an automation used to mean a project, with hosting to stand up and a sprint to wait for. Flows skips past all that.

Describe what you want to an AI assistant, or start from one of 40+ pre-built templates: incident enrichment, drift alerts, environment vending, certificate expiry monitoring, and self-service provisioning through Jira or ServiceNow.

Either way, you have a production-grade workflow before lunch.

Let AI agents act as themselves

Every team is connecting agents to infrastructure right now, and most are doing it on a shared credential or a person’s login. Flows gives agents their own governed path in: a Model Context Protocol (MCP) server, approval gates, and a complete audit trail. A flow can expose itself as a tool an agent calls, or call an agent directly. Either way, you can trace exactly what ran, and who, or what, asked for it.

Real self-service, not a ticket with extra steps

A request form that lands in a queue isn’t self-service. Flows lets developers describe what they need, an environment, a namespace, and get it provisioned automatically, inside the guardrails you already set. The platform team stops being the approval bottleneck for routine requests, without losing the approval itself.

One control plane, from apply through Day 2

If you already run Spacelift Deploy, Flows isn’t bolted on next to it.

It reacts natively to your stacks, runs, and drift, the events a generic automation tool never sees, because it was never inside your infrastructure platform to begin with. Route a Slack approval to the right channel the moment a stack drifts, or stage a rollout across dev, staging, and production with a sign-off at each gate.

Deployment and Day 2 trigger each other, under the same governance and the same audit trail.

Get started today

Flows now has its own place in the app switcher in Spacelift. Open it, describe what you want to automate, or pick a template, and connect your first app.

Feel free to browse the template gallery or read the Flows docs to see what tech debt you can retire first.

Solve your infrastructure challenges

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation, and controls to simplify and accelerate the provisioning of cloud-based infrastructures.

Catch AI Regressions Before They Ship with AI Evals in CI/CD | Harness Blog

Traditional software fails loudly: a test breaks, an API error, a container crashes. AI applications do not. An agent can return 200 OK, respond fast, pass every infrastructure check, and still hand a customer a wrong answer.

A support agent telling customers the wrong billing terms or return rules at scale is not a bug ticket. It is a brand, support-cost, and compliance problem that a traditional pipeline waves straight through to production. So the question for CI/CD becomes:

How do you decide whether an AI application is actually good enough to ship?

To test this, Harness AI Evals ran inside the deployment pipeline of a simple e-commerce support agent as a blocking quality gate:

The setup

• Dataset: 32 golden support scenarios (payments, orders, shipping, returns, cancellations, support, promotions, security).
• Metrics: Answer Relevancy, Task Completion, and Toxicity. Each scores a different part of the response.
• Gate: blocking, 70% pass threshold. Below the bar, no production deploy.

The first version failed the gate

An early run passed only ~65% of cases. Nothing was broken in the traditional sense: the build succeeded, the service deployed to QA, and the endpoint was live. But the behavior was not good enough, so the blocking step stopped the deploy. A 65% → FAIL number is only useful if you can see why, and the failed cases showed the pattern immediately:

The first row is the dangerous kind. Relevancy is 1.0, yet Task Completion is 0.3 because the fact was wrong. This is the regression that survives traditional testing: the API works, the reply reads naturally, and the customer gets bad information. The other rows are communication failures. The agent often had the right information but buried it, skipped part of it, or answered a nearby question instead.

From a vague complaint to an action list

Without evaluation, feedback is just "the agent needs improvement." The failed cases made it concrete, and five of the eight recurring failures were traced to incorrect, missing, or incomplete knowledge:

• Wrong payment and billing facts
• Missing cancellation and customer-service details
• Incomplete shipping information
• Inconsistent gift-card answers
• Indirect responses to yes/no questions

The tempting shortcut is to lower the bar until the gate turns green. That defeats the purpose. The honest path is to fix the application: correct the knowledge, and tune prompts so the agent leads with a direct answer and refuses to reveal its instructions. After iterating, later runs reached 75% and 78% against the same 70% bar, held it across repeated runs, and the production deploy continued.

AI is non-deterministic, so design for it

The same target, dataset, and config did not always produce identical results, with one comparison showing 24 passing cases versus 22. Sometimes the agent itself varied. Asked "Do you sell gift cards?", it scored Task Completion 1.0 in one run and 0.2 in another, answering "yes" once and "no" the next time. That is a real consistency problem, not noise.

Two rules follow:

• Trust repetition, not luck. An improvement counts only once it holds across repeated runs.
• Read a red gate before reacting. A failure can mean a genuinely bad answer (block it) or a broken judge or target (an infrastructure issue, not quality).

Why it belongs in the pipeline

Running the eval inside the delivery pipeline, rather than a separate tool, is the point. The eval and the deployment gate are one blocking step next to build and deploy, completing end to end in roughly seven to eight minutes. No glue code shuttling scores from an external dashboard, no second system deciding whether the build ships. The quality signal and the delivery decision are the same decision, made in the same place.

Traditional pipelines ask whether the app compiled, tests passed, and the service is healthy. AI applications add one more: is it behaving well enough to put in front of users? Because the most dangerous AI regression does not crash anything. It looks healthy while confidently giving your customer the wrong answer, and that is exactly what to catch before it ships.

Try it in your own pipeline

Harness AI Evals lets teams define golden datasets, pick the metrics that matter, and turn results into a blocking quality gate inside the pipeline, so a build only ships if the AI behaviour clears the bar. The fastest way to see it work is to watch the AI Evals demo and try it against your own agent.

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

The question that stopped the meeting

It was a routine cost review. The slide showed the month’s GPU spend, the biggest line on the whole infrastructure bill, and someone asked a five-word question: “Are we using these things?” Nobody could answer. The most expensive hardware we owned was also the hardest to see.

The frustrating part is that the answer already existed. Every GPU’s utilization had been recorded, every second, for months, all of it flowing into one central, infrastructure-owned Prometheus that held every metric for every team across thousands of namespaces. The data was right there. It just sat somewhere no tenant was allowed to look, because a store that sees everyone’s metrics can’t safely be opened to any one of them. So visibility split in two:

When we finally went looking, we found a GPU that had sat at zero percent utilization for eleven straight days: allocated, powered on, doing nothing, and invisible to the team that owned it. You can’t fix what you’re not allowed to see. Multiply that one idle card across a fleet, all of it drawing power behind green health checks, and “we’re not sure” becomes real money every month.

This post is how we closed that gap: how we gave every team a safe, self-service view into their own metrics without handing them the keys to everyone else’s. No new metrics stack, no vendor platform, just CNCF-native pieces arranged so the people spending the GPU budget can finally see it.

Why “just share Prometheus” doesn’t work

The obvious fix is to give every team read access to the central Prometheus. We ran into two walls, each built from an individually correct decision.

The first is security. A Prometheus query endpoint isn’t namespace-aware: if a tenant can run one PromQL query, they can run any query, including one that reads another tenant’s request rates or capacity plans. “Everyone can read everything” isn’t a posture you can defend across thousands of namespaces.

The second is scale, and it has a name every platform engineer knows: the noisy neighbor. The central Prometheus is already scraping and storing series for the whole fleet. Point a few hundred engineers and ad-hoc queries at it and the store everyone depends on starts to buckle. One team’s expensive range query becomes everyone’s latency spike.

Both are reasonable on their own. Together they leave the data tenants need locked in a store you can’t safely open to them. We needed to give each tenant their own curated, isolated slice, served directly.

A metaphor that made it click

Picture the central Prometheus as one enormous reading room where every team’s private notebooks sit on open shelves. Hand out a room key and you break confidentiality in the same motion, and the moment a crowd arrives the shared room grinds to a halt. What you want is a librarian who takes your card and brings a copy of only your box. That librarian is the multi-tenant proxy.

The insight: a proxy in front, a contract in YAML

We didn’t need a new metrics stack. We needed a thin, tenant-aware layer in front of the one we already had. The design came down to three moves:

• Identify: Authenticate the caller and establish which tenant they are.
• Isolate: Restrict every query to that tenant’s namespace, enforced below the query language so it can’t be bypassed.
• Deliver: Optionally copy a curated slice of each tenant’s metrics into their own small Prometheus, so their dashboards and alerts run against a store they own.

The other half is self-service. Platform teams can’t hand-curate metric lists for thousands of namespaces, so the contract is a small Kubernetes custom resource, a MetricAccess object, where a team declares which metrics it wants. The platform owns the mechanism; the tenant owns the policy. All of it sits on CNCF-native, open source pieces.

How the pieces fit

The infrastructure Prometheus keeps doing its job; everything tenant-facing sits behind the proxy:

On the read path, a tenant’s request enters through Nginx (load-balancing across proxy replicas), passes through kube-rbac-proxy for authentication and authorization, and reaches the proxy. The proxy discovers backend Prometheus instances via the Kubernetes API, fans the query across healthy backends, filters results to what the tenant may see, and returns the aggregate.

On the write path, the proxy periodically collects each tenant’s curated metrics and remote-writes them into that tenant’s own Prometheus. For HA tenants it resolves each replica’s pod DNS and writes to all of them, so every instance holds identical data. None of this is exotic: Prometheus, service discovery, and remote-write with a tenancy model on top.

The part that has to be airtight: isolation

Self-service is only safe if isolation isn’t optional. Two open source components do the work here.

kube-rbac-proxy handles authentication and authorization. It answers “who is this, and are they allowed?” using Kubernetes-native identity and RBAC, the same model you already trust for the API server, and carries the tenant’s identity through as a namespace assertion that anchors everything downstream.

Query-time isolation is handled by prom-label-proxy . It’s the piece that makes “everyone can read everything” impossible rather than merely discouraged: it rewrites every incoming query to inject a namespace matcher, so any query becomes query{namespace=”your-namespace”} before it reaches Prometheus. Enforcement happens below the query language, so no PromQL can escape it.

The proxy itself runs hardened: non-root (UID 65534), read-only root filesystem, all capabilities dropped, no privilege escalation, least-privilege service account. Defense in depth on the path that matters most.

Isolation that also cuts the bill

There’s a second, quieter isolation, where the cost story lives. Read-time filtering stops tenants seeing each other’s data, but a tenant’s own Prometheus can still store far more than it needs. The metricIsolation setting pushes the boundary to collection time: metrics are gathered through prom-label-proxy with the namespace filter already applied, so a tenant only ever ingests its own series.

The difference is not subtle:

For a typical tenant that’s roughly a 97% cut in stored series, from ten-thousand-plus down to a few hundred. A smaller store queries faster, costs less, and can’t leak data it never collected. The noisy-neighbor problem shrinks too, since the fleet-wide store leaves the critical path for everyday dashboards. Our original team went from empty dashboards to a Prometheus of its own.

What a tenant actually does

A tenant onboards with a single YAML file that declares the metrics and where to deliver them:

apiVersion: observability.ethos.io/v1alpha1
kind: MetricAccess
metadata:
  name: gpu-team-metrics
  namespace: gpu-team
spec:
  source: gpu-team
  metricIsolation: true         # only collect this namespace's series
  metrics:
    - "DCGM_FI_DEV_GPU_UTIL"                                     # exact match
    - "container_(cpu|memory)_.*"                # regex
    - '{__name__=~"nginx_ingress_controller_.*"}'  # PromQL selector
  remoteWrite:
    enabled: true
    interval: "30s"
    target:
      type: "prometheus"
    prometheus:
      serviceName: "prometheus-operated"
      servicePort: 9090
      replicas: 2                # write to both HA replicas
      statefulSetName: "prometheus-gpu-team"
    extraLabels:
      tenant: "gpu-team"
      managed_by: "multi-tenant-proxy"

The metrics list mixes three styles freely: exact names, regexes, and PromQL selectors. Apply the file and the proxy starts collecting and delivering. Querying is a normal Prometheus API call with a tenant header:

curl -H "X-Tenant-Namespace: gpu-team" \
  "http://prometheus-multi-tenant-proxy:8080/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL"

One requirement: the tenant’s Prometheus must accept remote-write (start it with –web.enable-remote-write-receiver). Everything else is defaults.

Six PromQL queries that make idle GPUs visible

Once a team can see GPU metrics, a few queries do most of the work. These assume a DCGM-style exporter; adjust the names to yours. Queries 5 and 6 also use an ingress request-rate metric, so swap nginx_ingress_controller_requests for whatever your ingress exposes.

1. Average GPU utilization per namespace — the headline number the team was missing:

avg by (namespace) (DCGM_FI_DEV_GPU_UTIL)

2. Count GPUs that are effectively idle — under 5% utilization for the last hour (this one finds the money):

count by (namespace) (avg_over_time(DCGM_FI_DEV_GPU_UTIL[1h]) < 5)

3. GPU memory used vs. total per namespace — separates “busy and memory-bound” from “reserved but empty”:

sum by (namespace) (DCGM_FI_DEV_FB_USED)

/ sum by (namespace) (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)

4. Power draw per namespace — a rough real-time proxy for cost:

sum by (namespace) (DCGM_FI_DEV_POWER_USAGE)

5. Hot cards, no traffic — GPUs busy while ingress is silent, often a stuck job:

avg by (namespace) (DCGM_FI_DEV_GPU_UTIL) > 70

and sum by (namespace) (rate(nginx_ingress_controller_requests[5m])) < 1

6. Traffic, no GPUs — requests arriving while GPUs sit idle points at a scheduling gap, not capacity:

sum by (namespace) (rate(nginx_ingress_controller_requests[5m])) > 10

and avg by (namespace) (DCGM_FI_DEV_GPU_UTIL) < 5

Query 2 closed the loop for our original team: it surfaced GPUs idle for hours, the invisible spend we opened with, capacity they could finally see and give back.

Holding up at fleet scale

Self-service without limits just moves the noisy-neighbor problem downstream. Two levers keep load bounded. Curated metric sets mean the proxy handles a fraction of the fleet’s cardinality, not all of it. And per-tenant collection intervals act as a quota: one team can run at 5-second resolution, a batch workload at five minutes, and neither pays for the other’s cadence.

Across many clusters it leans on the same primitives: dynamic backend discovery, independent per-tenant remote-write, and writes to all HA replicas so failover is a non-event. It’s still a system you operate, but its moving parts are ones a platform team already knows.

What we’d tell our past selves

• Self-service still needs guardrails. A team asking for “all metrics” usually means “I don’t know which ones I need yet.” Curation is a conversation, and good defaults beat a big allowlist.
• Cardinality is a cost decision. metricIsolation is the difference between a 300-series store and a 10,000-series one. Turn it on unless a team has a concrete cross-namespace reason not to.
• Delivery is where the sharp edges are. Retries, backoff, and HA multi-replica writes aren’t optional extras. Budget for the failure modes up front.
• Isolation can be overkill. Small teams do fine on filtered query access alone; reserve remote-write for teams that own real dashboards and alerts.
• Names drift across clusters. Half our early “no data” tickets were a query naming a metric the exporter didn’t emit. Pin exporter versions and document the exact series names.

The portable idea

Strip away the specifics and the pattern fits any multi-tenant cluster: an authn/authz proxy, label-enforced isolation below the query language, and optional per-tenant remote-write, all CNCF-native and open source. No proprietary lock-in, no bespoke stack; if you already run Prometheus, you have the foundation. Other approaches exist and are worth a look. This is just the one that let us give thousands of namespaces their own view without opening the shared store to everyone.

The team that couldn’t see its own GPUs now runs its own dashboards and catches idle capacity within the hour. The reading room is quiet again, and everyone has their own desk.

Try it yourself

The proxy is open source under Apache 2.0; the MetricAccess CRD, manifests, and examples are in the repo. The KubeCon + CloudNativeCon India 2025 talk walks through a live demo.

Repository: https://github.com/adobe/prometheus-multi-tenant-proxy

Talk recording: https://www.youtube.com/watch?v=gI40zpbES5w

Full article content is not available for inline reading.

Read the original article →

ChatGPT Work's Data agent connects to approved company data and business definitions, letting employees investigate questions and build shareable dashboards using plain language. It respects existing permissions, works with major data and BI platforms, and can recommend or carry out approved follow-up actions.

Full article content is not available for inline reading.

Read the original article →

Somewhere in your company there is a slide that says something like this: customers who enabled the AI assistant retain 15 points better than customers who did not. It has a bar chart. It has been in three executive reviews. It is driving next quarter's roadmap.

Nobody randomized the AI assistant. It shipped to eligible accounts, some of them turned it on, and the analytics team compared the ones that did against the ones that did not.

That comparison is not an effect. It is a description of who opts in. The feature did not make those accounts engaged. Being engaged made them adopt the feature.

AI features make this worse than the average opt-in feature, and it is worth being specific about why. To adopt an AI assistant, someone at the account has to notice the release, enable it, trust it enough to put it in front of their team, train people on it, fold it into a workflow, and keep using it after the novelty fades. Every one of those steps reveals something about the account: administrator engagement, executive sponsorship, technical sophistication, product maturity, organizational appetite for change. By the time an account shows up as an "adopter," the flag is close to a proxy for organizational readiness. Readiness predicts retention on its own. The feature is riding on top of it.

The instinct at this point is to model the customer's choice harder. Add covariates. Match on usage. Build a propensity score. This article argues for a different move, and it is the one sentence I would keep if everything else were cut:

Don't model the customers' choice harder. Find variation the customers didn't choose.

The rest of this article is about doing that with the one piece of an AI rollout that no customer picked: the eligibility rule.

Define the question before the method

There are three different quantities hiding inside "the effect of the AI assistant," and the slide conflates all of them.

The effect on adopters. If the accounts that turned the feature on had not adopted, how much worse would their retention have been? This is what the naive comparison is trying to estimate. It is the number the product team wants, because it describes the customers who actually experienced the feature.

The effect on everyone. If every eligible account adopted rather than nobody adopting, how much would retention change? This is the number finance wants, because it is what a forced rollout or a default-on change would produce. It is not the same number as the effect on adopters. Whether it is larger or smaller is an empirical question about effect heterogeneity; in opt-in product settings it is often reasonable to expect adopters to benefit more, but that is an assumption, not a theorem.

The effect at the margin. For accounts right at the edge of eligibility, what does becoming eligible do to retention, and what does adopting do for the accounts that adopt because they became eligible? Those are two numbers, not one, and the distinction matters later. These are the quantities nobody asks for and the ones you can usually identify most cleanly. They are also the ones that speak to the decision most often on the table: whether to move the eligibility rule.

The naive comparison does not estimate any of them. It estimates the difference between ready organizations and unready ones, with a feature flag attached.

The setup

The synthetic dataset has 40,000 B2B accounts. The AI assistant is available only to accounts with 25 or more seats, a seat-count eligibility rule of the kind common in SaaS products. Among eligible accounts, adoption is voluntary.

The thing that makes this hard is a latent variable I will call engagement: how invested the account is in the product. Engaged accounts are more likely to turn on new features and more likely to renew regardless. The analyst never observes it. What the analyst observes is seats, tenure, whether the account adopted, and whether it retained six months later.

The true effect baked into the simulation is +4 percentage points of 6-month retention from adopting the assistant. Retention also trends smoothly upward with account size: bigger accounts retain a little better, feature or no feature. So the observed adopter gap will mix three things: the feature effect, selection on engagement, and the fact that adopters are drawn from larger, eligible accounts.

RNG = np.random.default_rng(2026)N = 40_000TRUE_EFFECT = 0.04   # +4 pp retention from adopting the AI assistantCUTOFF = 25          # assistant only available at >= 25 seats engagement = np.clip(RNG.normal(0, 1, N), -2.5, 2.5)   # latent; never observedseats = ...          # skewed, integer, 3 to 300eligible = (seats >= CUTOFF).astype(int) # Adoption is voluntary among eligible accounts; engaged accounts opt in more.p_adopt = 1 / (1 + np.exp(-(-0.6 + 1.4 * engagement)))adopted = ((eligible == 1) & (RNG.uniform(size=N) < p_adopt)).astype(int) # Retention: baseline + engagement + smooth seat trend + the true effect.p_retain = (0.55 + 0.10 * engagement            + 0.03 * (np.log(seats) - np.log(CUTOFF))            + TRUE_EFFECT * adopted)retained = (RNG.uniform(size=N) < p_retain).astype(int)

The full data-generating process is in the notebook. The part that matters is above: adoption and retention share a cause the analyst cannot see.

Method 1: The slide

Business question: Do accounts that use the AI assistant retain better?

What it estimates: The difference in retention between adopters and non-adopters.

Identifying assumption: Adopters and non-adopters would have retained identically absent the feature. Adoption is as good as random.

df.groupby('adopted')['retained'].mean()# adopted=0: 0.531# adopted=1: 0.685      gap: +15.4 pp

Fifteen points. The true effect is four. The rest is selection: mostly engagement, plus the fact that adopters come from larger, eligible accounts that already retain somewhat better.

It does not help much to restrict the comparison to eligible accounts, which is the usual first fix. Among accounts with 25 or more seats, the adopter gap is +13.9 pp. Restricting to eligible accounts removes the mechanical size difference created by the 25-seat gate, and it shrinks the gap by only about a point and a half. Nearly ten points of excess lift remain. The selection is not happening at the eligibility line. It is happening inside the eligible population, at the moment each admin decides whether to click the toggle.

Reading the result. This is not a lie, exactly. Adopters really do retain 15 points better. The slide's mistake is the caption, which says the feature caused it.

Method 2: Regression adjustment on what you can see

Business question: After accounting for account size and tenure, do adopters still retain better?

What it estimates: The adopter gap, holding observed covariates fixed.

Identifying assumption: Conditional exchangeability. Everything that drives both adoption and retention is in the model.

elig = df[df.eligible == 1]adj = smf.ols('retained ~ adopted + np.log(seats) + tenure',              data=elig).fit(cov_type='HC3')adj.params['adopted']# +0.138  (95% CI roughly ±0.013)

The adjusted estimate is +13.8 pp, with a tight confidence interval. It is precise and it is wrong, and the precision is what makes it dangerous. A standard error of 0.7 points looks like rigor. It is rigor about the wrong quantity.

If you had the engagement column, this would work:

oracle = smf.ols('retained ~ adopted + np.log(seats) + tenure + engagement',                 data=elig).fit(cov_type='HC3')oracle.params['adopted']# +0.041

You do not have the engagement column. Pre-launch product usage is the closest proxy most teams have, and it helps. But the confounder here is not "how much they used the product," it is "how ready they were to keep using it," and no pre-period covariate fully captures that.

Failure mode: proxies that are too good. The temptation is to control for post-launch usage, since engaged accounts use the product more. Post-launch usage is downstream of the feature. Conditioning on it removes part of the effect you are trying to measure. Every covariate in this regression has to be measured before the feature existed.

Reading the result. Regression adjustment moved the estimate from 15.4 to 13.8. When observed covariates barely move the number, that tells you those covariates, in that specification, are not explaining much of the gap. It tells you nothing reassuring about the confounders you cannot see. The identification argument is still there; it just is not credible.

Method 3: Regression discontinuity at the eligibility threshold

Here is the thing the slide ignored. The feature is gated at 25 seats. An account with 24 seats cannot turn it on. An account with 25 seats can. Nothing else about those two accounts is systematically different: same tier of customer, same kind of admin, same distribution of readiness. The gate is arbitrary, and arbitrary is exactly what you want. It is the one part of the rollout that no customer chose.

Business question: For accounts near the eligibility threshold, what does becoming eligible for the AI assistant do to retention, and what does adopting it do for the accounts that adopt because they became eligible?

What it estimates: Two local quantities at the 25-seat margin: the reduced-form effect of eligibility on retention, and the effect of adoption for accounts whose adoption is induced by eligibility. This is a fuzzy regression discontinuity (RD), because eligibility does not force adoption; it only makes adoption possible.

Identifying assumption: Everything that affects retention, other than access to the feature, varies smoothly across the 25-seat line. The only thing that jumps at 25 is eligibility.

from linearmodels.iv import IV2SLS def fuzzy_rd(df, cutoff=CUTOFF, bw=10):    # Keep accounts within bw seats of the cutoff on either side.    w = df[(df.seats >= cutoff - bw) & (df.seats < cutoff + bw)].copy()    w['x'] = w.seats - cutoff                 # running variable, centred at 0    w['above'] = (w.x >= 0).astype(int)       # eligibility: the instrument    w['above_x'] = w.above * w.x              # lets the slope differ by side     # Second stage: retention on adoption, with local linear trend each side.    # First stage (in brackets): adoption instrumented by eligibility.    model = IV2SLS.from_formula(        'retained ~ 1 + x + above_x + [adopted ~ above]', data=w    ).fit(cov_type='robust')    return model m = fuzzy_rd(df, bw=10)m.params['adopted'], m.std_errors['adopted']# +0.049, se 0.037     95% CI: -0.024 to +0.121

Adoption jumps from 0% to 37% at the threshold. Retention jumps by 1.8 points. Per adopter at the margin: +4.9 pp, with a 95% interval from −2.4 to +12.1. The truth is +4.

Two estimands came out of that, and they answer different business questions:

If the decision is about changing the eligibility rule, the first row is the intervention you are actually contemplating. If the decision is about the value of the feature to a customer who uses it, the second row is the one you want. Bringing the wrong row to the meeting is the same estimand-drift mistake as the original slide, just one level more sophisticated.

Reading the result. The interval is wide. That is not a flaw in the method. That is the method telling you the truth about how much information a threshold contains. A 4-point effect diluted through a 37% first stage is a 1.5-point jump in the raw outcome, and detecting a 1.5-point jump takes a lot of accounts near the cutoff. The naive number had a half-point standard error because it was measuring something easy. The RD has a 4-point standard error because the identifying variation is much thinner. That is the price of throwing away the variation created by customer choice.

The diagnostics that make RD credible

An RD estimate without diagnostics is a number. With diagnostics it is an argument. Six checks, in the order I run them.

1. Bandwidth sensitivity. The bandwidth is how many seats on either side of the cutoff you include. Narrow is more credible and noisier. Wide is more precise and starts picking up curvature the linear fit cannot handle.

At ±5 the estimate is basically zero with an interval of ten points either way. That is not evidence of no effect; it is evidence that you have run out of data. Between ±10 and ±20 the estimate is stable. Report the range, not the best-looking row.

2. The running variable is discrete, and that matters. Seats are integers. At a ±5 bandwidth there are exactly ten distinct values of the running variable, and the untreated-side fit has to extrapolate from 24 seats to the cutoff at 25 because there is no untreated observation arbitrarily close to the threshold. Conventional RD inference assumes you can zoom in as close as you like. With a discrete running variable you cannot, so the fit's functional form is doing real work and specification error is part of the uncertainty. This is the normal situation in SaaS, where the gating variable is seats, licenses, or a tier. Two practical consequences. Treat the bandwidth and specification sensitivity table as part of the primary result, not a robustness appendix. And be careful with the common advice to cluster standard errors by the running variable's values: it is not a free fix; Kolesár and Rothe (2018) showed it can understate uncertainty, and in this simulation it shrinks the standard error from 0.037 to 0.021. I keep the heteroskedasticity-robust interval as a measure of sampling uncertainty, but I do not treat it as resolving the discreteness problem. That is exactly why the bandwidth and specification sensitivity results belong in the primary analysis.

3. Placebo cutoffs. Run the same reduced-form regression at seat counts where nothing happens. If retention "jumps" at 15 seats or 35 seats, your design is finding structure that is not there. One rule when choosing placebos: the window around each fake cutoff has to stay on one side of the real one. A placebo at 20 seats with a ±10 window would span 10 to 29 and contain the actual discontinuity, which is not a test of anything.

for placebo in [15, 35, 45, 55]:    ...  # same local linear fit on retention, ±10 window, different cutoff# cutoff 15: jump = -0.016 (se 0.014)# cutoff 35: jump = +0.004 (se 0.017)# cutoff 45: jump = -0.004 (se 0.023)# cutoff 55: jump = +0.008 (se 0.031)

All noise. The only place retention jumps is the place eligibility jumps.

4. Nothing else changes at the threshold. Before using the cutoff, inventory everything else that changes there. If pricing, support entitlement, onboarding, an account-management tier, or contract terms also jump at 25 seats, the reduced form is the effect of the bundle, not the AI feature alone, and the exclusion argument for the instrument collapses. Seat-count thresholds in SaaS often do double duty. Check the price book before you check the data.

5. Covariate smoothness. Pre-treatment covariates should not jump at the threshold either. Tenure does not (jump = 0.06 months, se 0.24). In the simulation I can also check engagement itself, which is the point of the exercise: it is smooth across the cutoff (jump = 0.01, se 0.03). In real data you cannot check the unobserved confounder, which is why you check every observed one and reason about whether the unobserved ones would behave differently.

6. Manipulation of the running variable. This is the one that breaks RD in practice, and it is specific to how SaaS companies operate. If the sales team knows the AI assistant unlocks at 25 seats, they will upsell 22-seat accounts to 25 to close the deal. Now the accounts just above the threshold are not comparable to the ones just below; they are the ones a rep decided were worth pushing. Check the histogram of seats around the cutoff. A pile-up at exactly 25 is the tell.

df.seats.value_counts().sort_index().loc[20:30]# 20:1247  21:1120  22:1190  23:1080  24:1136# 25:1060  26:1008  27: 929  28: 914  29: 872  30: 821

Smooth in the simulation. In your data, look. If you see bunching, the threshold is not arbitrary anymore and the design is compromised. The clean fix is available only if eligibility was actually determined from a seat count snapshot taken before the feature was announced; in that case the snapshot is the running variable and the rep could not have gamed it. If eligibility was evaluated on live seat counts, switching to a historical snapshot changes the running variable without preserving the discontinuity in access, the first stage weakens, and you are back to arguing about whether the snapshot is a valid instrument. Sometimes it is. It is not automatic.

Which number goes on the slide

Four numbers came out of this analysis and they answer four different questions.

The two RD estimates are the only ones here whose identifying variation comes from the rollout rule rather than from customer choice, and they come with two honest caveats that belong on the slide.

First, it is local. It describes accounts around 25 seats. If the decision is "should we expand eligibility to accounts just below 25," the reduced form above is directly relevant. If the decision is "should we lower the threshold all the way to 15," you are already asking the estimate to travel beyond the population that identified it. And if the decision is "should we make it default-on for enterprise," neither number speaks to a 500-seat account at all.

Second, the two RD estimates apply to different populations. The reduced form describes what happens to all accounts near the threshold when access is offered, including the ones that never adopt. The 2SLS estimate is narrower: because nobody below the threshold could adopt, the accounts whose behavior the instrument moves are precisely the ones that adopt once eligible, and the estimate describes the effect of adoption for them. It says nothing about what would happen if the 63% of eligible accounts near the cutoff that never turned the feature on were forced or persuaded to use it. If the roadmap decision is a default-on rollout, that is the population that matters, and a fuzzy RD cannot reach it.

Those caveats sound like weaknesses. They are the opposite. The naive number has no caveats because its identifying variation is entirely the customer's choice. It is confidently wrong about everyone. The RD numbers are carefully right about a specific group, and they tell you who that group is.

If you do not have a threshold

Not every feature is gated by a clean cutoff. If yours was released with no eligibility rule at all, the discontinuity design is not available and you are looking for a different source of variation the customer did not choose. Staggered rollout by region or account cohort gives you timing variation. An in-product nudge shown to a random subset gives you an instrument for adoption. Neither is this article. The principle carries over unchanged: find the part of adoption that was decided by something other than the customer, and estimate from that part only.

A few closing pitfalls

1. The precision trap. The naive and adjusted estimates had confidence intervals under two points. The RD interval was fourteen points wide. Stakeholders will prefer the tight one. Tight intervals around biased estimates are the most expensive output a data team can produce, because they get acted on.

2. Post-launch covariates. Anything measured after the feature shipped is a candidate outcome, not a candidate control. Usage, support tickets, NPS, seat expansion: if the feature could have moved it, it does not go on the right-hand side.

3. Gaming the gate. If sales, CS, or the customer can push an account across the threshold on purpose, the threshold is not arbitrary. Check the density before you check anything else.

4. Treating the running variable as continuous. Seats are integers. The bandwidth table is not a robustness check you run at the end; it is where the specification uncertainty lives.

5. Extrapolating the local effect, or the wrong one. A +5 point effect at 25 seats is not a +5 point effect at 250 seats, and the effect of adoption is not the effect of offering access. If the business decision is about a different part of the customer base than the threshold sits in, say so, and say what additional assumption would be required to carry the number over.

6. Calling the naive gap a lower bound. I hear this one a lot: "even if it is confounded, the feature clearly does something." The naive gap in this simulation is nearly four times the true effect. It is not a bound on anything.

The adoption slide was never measuring the feature. It was measuring the customers who chose it. The threshold that gated the feature is the only part of the rollout that was not chosen by anyone, and that is exactly why it is the part of this rollout you can learn from.

Four production Spark cases cut runtime, compute, S3 operations, and costs by tracing hidden waste beyond top-level CPU and memory metrics.

Jetpack: Consensus Made Generally Fast (OSDI '26)

Aleksey and I are back to reading papers live. This paper, Jetpack(OSDI '26), attempts building a universal 1-RTT fast-path framework that bolts onto existing leader-based consensus protocols with minimal modification.

Why would we want this? Classic consensus protocols like Raft, Paxos, or Zab require two round-trip times (2 RTT) to commit a command: one RTT from client to leader, and another to replicate across followers. The extra RTT matters a lot for WAN deployments, so fast-path protocols (such as Fast Paxos, EPaxos, or SwiftPaxos) reduce this to 1 RTT by bypassing leader serialization, but unfortunately they tightly couple the fast path to the core protocol design. Production systems cannot easily swap out their battle-tested bespoke consensus engines, but if there was an add on that helped with latency especially in WAN deployments, that would be useful.

The good news is that Jetpack is truly an add-on portable deal. It provides a shim layer that runs two execution paths in parallel: a 1-RTT fast path and the original 2-RTT consensus path. When a client issues a command, it broadcasts the request concurrently to both paths. The fast path checks for key conflicts, and if none exist and a supermajority quorum ($\sim 3/4$ of nodes) issues a promise, this enables the command to fast-commit in 1 RTT. To guarantee agreement, original path proposers promise not to propose conflicting commands ahead of fast-committed ones.

The bad news is that this design gets wasteful due to keeping two distinct logs (the fast-path log and the original-path log). The original consensus engine runs its full replication cycle in the background, ignoring the fast path replication of commands (because it is completely oblivious to the fast path replication in the name of bolt-on portability). So these commands travel in the network twice, and replicas process commands twice, introducing redundant work and extra CPU/network overhead.

This dual-log architecture also creates a bigger gap between commitment and execution. Jetpack fast-commits in 1 RTT, but actual state-machine execution is driven strictly by the underlying original 2RTT path log ordering. Then, what good is a fast commit in practice? If you are running write-heavy, asynchronous pipelines or "fire-and-forget" ingestion where a client issues a PUT(key, val) and immediately moves on to the next task, a 1-RTT durable commit confirmation is a win. However, the fast path only gives you a fast commit, but it does not accelerate state-machine execution. The moment you run an interactive workload, say a client that issues a PUT(key, val) and immediately follows up with a GET(key) expecting read-your-own-writes or linearizability, the fast-path illusion breaks down. Jetpack's shim detects the unexecuted PUT sitting in its in-flight conflict pool and immediately demotes the GET right back to the slow 2-RTT original path to preserve correctness. So, yes, Jetpack stays linearizable, but you pay the full 2-RTT latency tax and wait for the original consensus engine to catch up to answer the GET.

This connects directly to the principle of nil-externality formalized in Exploiting Nil-Externality for Fast Replicated Storage (SOSP '21). Because write commands like PUT return no system state back to the client (they have "nil externality"), Jetpack can safely grant a 1-RTT fast commit before state-machine execution. But, the moment an operation (like a GET) needs to externalize state, that nil-externality optimization breaks down and forces the system to wait for full state-machine execution.

Despite this execution lag and redundant message overhead, the evaluation section shows gains for write-heavy pipelines (I think due to nil-eternality optimization) by benchmarking Jetpack across six consensus systems deployed on 10 AWS datacenters using YCSB workloads and Facebook's Akkio production traces. Because cross-datacenter write requests (such as remote shard updates in Akkio) primarily wait on durable commit confirmation before responding, Jetpack slashes client-observed end-to-end latency by up to 60%.

The paper's biggest safety contribution is to show how the fast-path protocols may break during leader elections. As I noted in my 2020 blog post review of CURP, mixing witness state with backup replicas makes view changes inherently risky. Jetpack proves this by uncovering a concrete bug in CURP’s Raft extension (used in production by Xline), where a lagging witness ACKs a fast-path command, only for a delayed cleanup message from a new leader to erase it, causing permanent data loss (or reordering) after a crash. This happens because promises made in stable views live in local replica states that a new leader never saw. Jetpack fixes this "view change hazard" with two strict principles: keeping fast-path views independent so ACKs cannot straddle terms (Principle 1), and forcing new leaders to write a "stability marker" that recovers all prior fast-committed commands before accepting new proposals (Principle 2).

As Aleksey and I experienced live during our reading session, untangling Jetpack’s three-phase recovery procedure was the trickiest part of the paper. We struggled to follow why recovery requires only a standard majority rather than a superquorum. But refreshing our understanding of Fast Paxos later showed that Jetpack's recovery mechanism is similar to that of Fast Paxos. Fast Paxos explicitly requires a supermajority quorum ($Q_2 \approx 3/4$ of nodes) for fast-path commits precisely so that leader election and recovery ($Q_1$) can run on a simple majority. Because any supermajority $Q_2$ is mathematically guaranteed to overlap with any standard majority $Q_1$ by at least one node, a newly elected leader polling a simple majority during recovery will always discover any fast-committed command. Still, the quorum math aside, I'll be damned if anyone can call fast-path recovery simple (and dependable).

Premiere's new Generative Media tool lets editors fill timeline gaps with AI-generated video and sound effects without leaving the project. Editors can pick among models like Adobe Firefly, Google Veo, Runway, Luma, and Kling, while beta tools add music and soundscape generation, speaker separation, and an After Effects AI Assistant.

The web is evolving beyond menus, forms, and endless clicks toward experiences shaped around human intent. For UX designers, understanding this shift means re-evaluating their role, moving from designing visible interfaces to guiding transparent, intent-driven AI experiences.

Ever since the commercialisation of the Graphical User Interface pioneered by systems like the Xerox Star and popularised by the original Apple Macintosh, software has relied heavily on point-and-click interactions. If you wanted to book a trip, buy a pair of shoes, or research a health symptom, you were expected to navigate a labyrinth of user interfaces. You click menus, adjust range sliders, fill out multi-step forms, deal with cookie pop-ups, and open dozens of browser tabs just to cross-reference basic information. We have become accustomed to spending more time managing the software rather than actually achieving our goals.

That contract is officially changing. Driven by advances in artificial intelligence and large language models, a new generation of web tools is pioneering a radical philosophy: Intent-Driven Design. Rather than expecting our users to learn complex menus and click through elaborate sales funnels, these platforms operate by capturing high-level human goals and silently executing the grunt work in the background.

This vision builds on long-standing HCI concepts like Golden Krishna’s “The Best Interface Is No Interface” and Don Norman’s principles of human-centered design.

The ultimate goal of modern web design is no longer to build prettier buttons or flashy animations; it is to eliminate the interface entirely.

Well, at least what we understand based on our current experience.

It is essential for UX designers to understand the changes we’re witnessing and even re-evaluate our role, shifting our focus from designing visible interfaces to guiding transparent, intent-driven AI experiences.

The Death Of The “10-click” Process

To understand where web design is going, we first have to look at the friction we have accepted as “normal” for decades.

Consider the traditional workflow of buying a flight online. The user experience is intentionally hyper-interactive:

1. Navigate to a travel aggregator or airline website.
2. Select “Round Trip” from a dropdown menu.
3. Type the origin city and wait for auto-complete.
4. Type the destination city and wait for auto-complete.
5. Click a calendar modal, toggle through months, and select departure and return dates.
6. Choose the number of passengers and cabin class.
7. Click “Search” and wait for the results page to load.
8. Filter by price, layover duration, departure time, and airline.
9. Sort the results and scan through dozens of individual options.
10. Click through a three-page checkout funnel dodging upsells for rental cars and travel insurance.

This is a classic point-and-click UI paradigm. The computer acts as a passive container of data, and the human acts as the orchestrator, manually inputting parameters, interpreting raw outputs, and executing each micro-step along the way.

Intent-driven design flips this dynamic entirely. Instead of forcing you to navigate the mechanical steps of how to find a flight, the interface asks a simple question: What are you trying to accomplish?

When you express a goal, such as “Find me a non-stop flight to Chicago next weekend under $300 that arrives before 5 PM”, the software reads your intent, executes the multi-step search query behind the scenes, compares the options, and presents a single actionable resolution. The ten clicks dissolve into one clear intent outcome.

Three Web Platform Examples Replacing Buttons With Intent

This shift isn’t a theoretical vision of the distant future. Some of the world’s most accessible websites allow you to experience intent-capturing systems already.

1. Perplexity AI: Search Without The Open Tabs

Traditional search engines like Google were built as directories. You typed a keyword, and the interface rewarded you with ten blue links. The actual work of opening five tabs, skimming long articles, dodging ads, and manually assembling an answer was left to you. Although this is now changing, with AI content appearing first when you search on Google.

Perplexity AI allows anyone to perform instant intent-based searches. Instead of forcing you to click into multiple tabs, the interface synthesises the perfect result for your context.

• The interaction: You ask a complex question in plain English: “Compare the top three budget laptops for a computer science student, focusing on battery life and keyboard quality.”
• The invisible work: In under two seconds, the platform searches the web, reads dozens of articles, evaluates hardware specifications, and cross-references user reviews.
• The result: Instead of throwing web pages at you, Perplexity dynamically generates a single custom answer complete with formatted comparison tables, pros and cons lists, and concise inline citations. You get the outcome of thirty clicks in just one.

2. Vercel v0: Web Design Without Dragging And Dropping

For years, creating a website meant using visual builders like Figma or drag-and-drop web editors. As designers, we spent hours manually drawing rectangles, picking hexadecimal colour codes, adjusting padding values, and sometimes writing CSS code.

Vercel v0 lets anyone test its generative interface builder right from the home page without signing in.

• The Interaction: A user types a descriptive goal into the input prompt: “Create a modern dark-mode dashboard for a subscription SaaS app showing monthly revenue, active users, and a customer churn chart.”
• The invisible work: The underlying AI understands layout principles, accessibility guidelines, UI component libraries, and responsiveness. It writes clean HTML, Tailwind CSS, and React code in real time.
• The result: Rather than spending an afternoon manipulating elements, the user receives a fully functional, pixel-perfect user interface in seconds! The traditional visual design tool disappears, replaced by raw human intent.

3. Goblin.tools: Breaking Down Overwhelming Tasks

Many traditional productivity tools require you to manually construct task lists, drag items across Kanban boards, and assign micro-deadlines. Goblin.tools is a free, single-page suite of single-task AI tools designed specifically for neurodivergent users or anyone feeling overwhelmed, requiring zero sign-ups or configuration.

• The interaction: You enter a vague, intimidating goal into the Magic Todo tool, such as “Prepare for a job interview”.
• The invisible work: Instead of forcing you to plan every step, the system analyses the cognitive load of the goal and automatically breaks it down into small, actionable sub-tasks adjusted to your preferred level of detail.
• The result: The interface eliminates the stressful task-planning phase and presents a clean checklist tailored to your exact goal.

The Core Pillars Of Intent-driven Web Design

When you remove traditional menus, sidebars, and forms, how do you keep a website usable? Designers pioneering this space rely on three core pillars:

1. High-level Goals

An invisible interface doesn’t always wait for you to type a command — it leverages context to anticipate user needs. Guided by industry standards like Apple’s Human Interface Guidelines on Contextual & Ambient Design, modern systems read environmental metadata, such as device state, location, and past interaction patterns, to trigger proactive actions.

2. Generative UI

In traditional web design, every user sees the exact same layout. As highlighted in the Nielsen Norman Group’s analysis on AI as a new UX paradigm, computing is shifting from command-based interaction to outcome-specification. This enables Generative UI, where interfaces are rendered dynamically on the fly based on what you are trying to do in that exact moment.

3. Execution by AI Agents

Traditional interfaces are obsessed with prevention, constantly peppering users with confirmation pop-ups. Intent-driven interfaces adapt Jakob Nielsen’s 10 Usability Heuristics on User Control and Error Recovery by shifting the safety net from prevention to easy reversibility.

The Hidden Risks

While removing interface friction is liberating, stripping away visual controls introduces significant product design challenges. When software acts on inferred intent rather than direct point-and-click commands, designers must navigate critical ethical and technical pitfalls.

The Illusion Of Control

When a website makes decisions for you, it can quickly feel patronising or invasive. Automation must maintain clear boundaries to prevent user frustration. Designers must strike a delicate balance: automate routine execution, but explicitly prompt the user for high-stakes confirmations.

The Black Box Problem

With an intent-driven interface, troubleshooting becomes harder. To address this “black box” challenge, the Google PAIR (People + AI Research) Guidebook advocates for transparent feedback loops. The system must explicitly state its interpretation so users can calibrate trust and correct misinterpretations instantly.

Conclusion

The ultimate goal: The best design is no design.

The philosophy behind Golden Krishna’s seminal book The Best Interface Is No Interface, aligned with the broader Center for Humane Technology’s Time Well Spent movement, turns these metrics on their head. The ultimate test of a modern digital product is no longer how visually captivating its buttons are, but rather: How effectively did this software solve the problem and get out of the user’s way?

We are entering an era where web applications will no longer be measured by the beauty of their interfaces, but by their ability to render those complex interfaces completely obsolete. The future of the web isn’t more interaction — it is seamless, invisible completion.

Test Complex Interactions Earlier with AI Prototyping

Summary: AI tools make it feasible to build fully interactive prototypes of complex interfaces so you can test them with users earlier in the design process.

Complex interfaces like filters, dashboards, and conversational AI have many possible states and can be tedious to prototype by hand. Before AI tools were available, most teams would test a simple prototype that covered only a few happy paths and wait until developers built the product to see how users interacted with it. With AI tools, teams can now build realistic prototypes and test them before deployment.

For less complex interfaces, static prototypes are still useful for answering many research questions. However, when an interface is complex, AI prototyping tools make it easier to build high-fidelity, interactive prototypes that closely resemble the real product.

Interactive Prototypes, Fast

AI tools bring speed to the design process, enabling designers to create a realistic working prototype in a day. Incorporating realistic participant data, numerous states, complex filtering, and AI-generated responses is now available early in the design process. This means you can test with users earlier, more often, and iterate on what you learn from research. Let's look at some case studies of AI prototyping.

Prototyping an Expense-Policy Editor

Ramp's design team has started integrating AI prototyping into its work. Designers across teams leverage AI to prototype interactions that would be difficult, time-consuming, and impractical to develop manually. Ramp’s senior product designer, Pavan Garidipuri, used Cursor to prototype a redesign of the expense-policy editor, a tool that helps admins manage spending rules and was historically challenging for users to navigate. I discussed the project with Pavan, who explained that he chose an AI prototyping tool because manual prototyping of the interaction design was too complex. The redesign focused on the experience of editing a policy through AI chat. Since conversation-based editing relied on dynamic interface responses, it was hard to simulate with static screens.

During usability testing, participants received a link to the working prototype, loaded with their own sanitized data. The prototype operated like the actual product, enabling participants to interact naturally and provide precise feedback. This approach allowed Pavan’s team to discover edge cases that might have otherwise remained unnoticed. One key finding was that the way of tracking and displaying edits was confusing. Pavan said his team couldn't have found that usability issue without an interactive prototype that closely resembled the real interface.

The prototype quickly progressed from concept to demo in only three weeks, with ongoing iterations guided by customer feedback. This project ultimately influenced the team's roadmap for the remainder of the year.

Prototyping Conversational AI

Also at Ramp, product designer Andrew Lucas used AI to visualize a conversational AI expense-reporting workflow. This interface was difficult to prototype realistically without AI because it had to generate a sensible response to virtually any user input and might even produce different outputs for the same input. Traditional design tools, which are limited to a fixed set of prebuilt screens, can't capture that kind of open-ended, nondeterministic behavior.

Andrew used Cursor to prototype a conversational AI interface connected to an agent that provided nondeterministic outputs, mimicking the real product. This interactive, realistic prototype could then be tested with users before deployment. Study participants could input anything they’d like, rather than having to follow a predetermined path. As a result, the researchers could see how the agent handled unexpected requests. These kinds of issues surface only when testing a near-final version and might not emerge if participants are shown only one happy path.

Prototyping a Filter Interface for Traffic Engineers

Researchers at Purdue University developed a filtering interface to assist highway traffic engineers in analyzing complex traffic data. The dataset included 27 attributes per traffic event, with several containing nested sub-attributes. The team experimented with two prototyping approaches: static prototypes built by hand and AI-generated interactive prototypes.

This was a sequential case study focused on a single project rather than a controlled experiment. From phase one to phase two, as the prototypes shifted from static, hand-built to interactive, AI-generated, the team's understanding of the problem deepened. Because of this, interpret this example as an illustration of what became feasible to build and test, not as evidence that AI prototyping produces better research.

Phase 1: Static Screens Built by Hand

First, the team built low-fidelity wireframes in Figma without any AI assistance and got user feedback on the prototype’s visual design, but couldn't observe how people would interact with the filters because the prototype was static. While visual design matters, a key aspect of filter design is how the system responds when users interact with it (for example, by applying or removing it). To thoroughly test filters, users should be able to click through and use the system as they would with the real product.

Phase 2: Interactive Prototype Generated with AI

In this phase, the team rebuilt the interface with v0 and Bolt, prompting the tools with specific design requirements and a synthetic, AI-generated dataset. During usability testing, researchers shared a link to the working prototype, and participants interacted with the interface as they would with the finished product.

The Purdue researchers noted that participants engaged with the two types of prototypes differently. With static screens, they provided less specific feedback and appeared to treat the designs as too preliminary for in-depth critique. In contrast, with the AI-generated, interactive prototype, users identified specific problems the team could address.

Designing filters is challenging because there are many edge cases to consider. These cases cannot be tested with a static prototype, but manually building an interactive prototype when the dataset has many attributes and sub-attributes is unrealistic. AI cuts the time required to build a prototype that behaves like the real product.

How to Prototype Complex Interfaces with AI

Receiving usable outputs from AI tools depends on the context that you give the tool and the design guidelines that you specify. The process below outlines how to use AI to prototype a complex interface in five steps.

1. Determine what you need the prototype to do. Before you open the AI tool, decide what level of interaction the prototype needs to support to answer your research questions. What behaviors does the research require? Which interactions, states, data, or responses must work? Sometimes a simple prototype answers your research questions, and full interactivity is not necessary.
2. Make your design decisions. Once you know what the prototype needs to do, decide how it should look and behave. Write down what the layout, hierarchy, interactions, and even edge cases, like empty states and error states, should look like. The quality of the prompt you write in step 4 depends on the thinking you do in this step.
3. Collect the data to use in your prototype. Here, you have a couple of options. First, you can export real data from your product to provide to the AI tool. If you choose to use real data, follow your organization’s data policies and do not upload any personally identifiable information. Also, review your tool’s data policies to understand how your data could be used by the company that owns the AI tool. Alternatively, you could ask an AI tool such as Claude, Gemini, or ChatGPT to generate a synthetic dataset based on criteria you define.
4. Write a prompt that describes your design decisions and includes your data. This step combines the information gathered in the first two steps. Craft a detailed, specific prompt to guide the AI tool in building a high-quality interface. The prompt should describe the interface you want (the layout, the interactions, the edge cases) and include a file containing your data. Feel free to attach any sketches, wireframes, mood boards, or existing designs, too. If you're unsure how to structure the prompt, you can ask a general-purpose AI tool to help format it; markdown format is usually easiest for LLMs to parse.
5. Feed the context into an AI prototyping tool. Tools like Cursor, v0, and Figma Make can take a detailed prompt and return a working, interactive prototype. The tool you use depends on how much control you need and how comfortable you are editing code.
6. Edit the output and iterate. AI prototyping tools are not perfect, and the first output probably won't get everything right. Expect to go back and forth, tweaking the layout, fixing interactions, and guiding the tool toward what you had in mind. Once you’re satisfied with your iterations, run a quick pilot test to ensure the prototype works as intended and doesn’t introduce any unforeseen misunderstandings.

Integrating AI Prototyping into Your Process

AI tools can support many phases of the design process, from discovery and alignment to ideation and handoff. This article focuses on rapid prototyping for usability testing. There are a few points to keep in mind.

You Still Make the Design Decisions

If you leave design decisions to the AI tool, it will automatically fill in the gaps and will lack human attention to detail. These tools don't inherently understand content layout or visual hierarchy, so they need detailed guidance from a knowledgeable designer to produce useful outputs. That said, AI tools are valuable because they can execute your design decisions quickly.

Ultimately, AI helps you realize design ideas faster, but the design decisions remain yours. What's changed is how quickly you can turn those decisions into interactive prototypes that you can learn from.

Don't Fall for the Fidelity Trap

Have you ever built an AI prototype that looks polished, shows real data, and has full interactive capabilities, and heard someone say, "This looks great, can we just ship this?"

It's a reasonable question from someone who doesn't know how the prototype was made, but you shouldn’t assume that this AI-generated, high-fidelity interactive prototype is production-ready. The danger is that its polish might cause people to stop working to improve it. AI-generated prototypes may look complete while introducing all sorts of assumptions and inaccuracies, such as using the wrong design pattern for the situation, creating a confusing hierarchy, or unnecessarily repeating elements. To avoid this temptation, make sure that someone with strong design knowledge reviews the prototype before it’s pushed to production.

Conclusion

Creating realistic, interactive prototypes for complex interfaces used to be prohibitively time-consuming. AI prototyping tools change that. They let you develop high-fidelity prototypes that you can test with users before committing to building them for real.

References

Tianyi Li, Tanay Maheshwari, Alex Voelker (2025). User-Centered Design with AI in the Loop: A Case Study of Rapid User Interface Prototyping with "Vibe Coding." ACM Collective Intelligence 2025. https://arxiv.org/abs/2507.21012

Elizabeth Lin. (2025, November 10). Building with Cursor ft. designers from Cursor (Ryo Lu), Notion (Jin Park), and Ramp (Catherine Wang) [Video]. YouTube. https://www.youtube.com/watch?v=T8T2gHCKWCE

Anatomy of AI Input

I've been building AI inputs for months now. I call this component prompt-input in prompt-kit, my library of components for AI applications.

When ChatGPT launched in November 2022, the input field was simple, just a textbox and a button to send. Three years later, the input in AI products does a lot more than just send text.

Here's how an AI input actually works, layer by layer.

Input Field

The input starts with a simple field where the user types a prompt. It should grow to multiple lines and stop at a clear defined max height. It should grow without pushing the rest of the layout or jumping the scroll. Shift+Enter adds a new line, and Enter sends (you can make it configurable in settings).

You can guide the user with a placeholder if you have context, commands, or anything hidden inside. Small details matter, like customizing the caret color to match the brand color.

On mobile, keep the font-size at least 16px to avoid Safari zoom. Or set a proper viewport meta tag to stop the auto-zoom. Keep the typing stable: no shifting lines, no auto-capitalization on mobile, and no cursor jumps (stays where the user expects).

Context Bar

A group of controls that change how the model will answer: model selector, mode (code, search, tools), memory, context, etc. Sometimes you don't need a context bar, in that case don't add it just for the sake of it. Try not to overload the input.

It's usually below the input field, and it should be more discreet than the field itself. Keep only what the user needs often. Everything else should go in a menu or settings. Don't try to put everything here.

Every action in the context bar should be keyboard-accessible, and you can add shortcuts for some actions. On mobile, if space is too tight, labels+icons can become icons only.

Tool Triggers

If the context bar is too crowded, use tool triggers. They can be in the context bar or inside the input, as a dropdown or a slash command.

Pick the pattern based on how many tools you expose and how often the user needs them. Don't add too much. Keep it simple.

Triggers should be keyboard-first and appear fast. You can add a small interaction to them, for slash commands I prefer to make them instant. And always avoid layout jumps (like the input moving or growing when a trigger opens).

Attachment Zone

In most AI products, you can add files, images, videos, or audio. It’s a fragile part because many things can go wrong if you don't handle it well. Most of the time it's a button in the context bar. It must cover the basics: uploading, uploaded, failed upload, and removing a file. You also need file constraints: size limits, file types, number of files.

Drag-and-drop matters too: show a clear drag-and-drop highlight when a file enters the input. When a user drags and drops a file, show a preview above the input field, or directly in the chat. Stack multiple files if needed, and make sure the input doesn’t jump.

If there’s an error, show it near the input. Avoid toast notifications. To make it feel fast, use optimistic updates: show the file instantly with a preview, then sync it in the background or when the user sends the message.

Assist Layer

Sometime user needs guidance. The assist layer can take a few forms: ghost text inside the field, inline suggestions that appear while typing (like Cursor with file paths), or prompt suggestions/presets above or below the input.

It should help and guide the user but not take over the input. Make it fast, non-intrusive, easy to ignore and usable from the keyboard when needed.

The assist layer is easy to overlook, but it makes the product much better when done right. Suggest when it makes sense for your product and your users. But don't over-suggest.

Message Controls

Don’t disable the controls. Even if you don’t allow sending multiple messages, the user should still be able to type, correct something, or stop the response.

If you have voice mode, you can show the voice button first and switch to a send button when the user starts typing, or keep both visible.

When the user sends a message, the send button should turn into a stop button so they can interrupt the response. Make sure all actions are keyboard-accessible.

State Layer

We usually have these states in an AI product: idle, loading, streaming, error, success.

In loading or streaming, don't lock the input. The user should still be able to type or stop the response.

For the error, keep the input neutral and usable. Errors belong to the generation, not the typing, so keep them out of the input (chat container, tool, etc.). Avoid toast notifications here too. Errors should provide a way to retry the generation. (like a retry button in the chat container).

The simple rule: keep the input usable in almost every state. The user should interact with the input as much as possible. Responses can be long in AI products, so let the user use the product while they wait.

Edge Cases

Edge cases vary from product to product, but the input rules stay mostly the same: predictable and keyboard-first. Typing must stay instant. No lag, no jumps, no layout shifts when the field grows or shrinks.

The input should stay usable even if the model is slow, the network is slow, or the product is still loading something. Never make it read-only while the model is streaming. Don’t steal focus when a response starts or ends. On mobile, handle the keyboard covering the field.

Optimistic updates most of the actions, keep everything fast, don't overdo animations. Clear the input right after sending and switch to the stop button without delay.

A good AI Input looks simple, but hides a lot of details. And every layer matters. As the product grows, resist the urge to overload it. If something can live elsewhere move it, if you can group functionality behind a trigger, do it. There's always a balance between showing too much and not enough. Try to guide your user without overwhelming them.

It's also an element of the product where you can play with the experience and the details. Add that beautiful shader in the background for onboarding, that refined animation when the user sends their first message. Don't forget that people remember great experiences, in a crowded AI market, that can make the difference.

Your product didn’t get worse. It still does the job, works the way your customers expect and gives them no compelling reason to switch. Then those customers start using agents elsewhere, and the standard around your product changes. They ask an agent to research a trip or organize some files. They use it to make a change in another product. Jobs that used to require opening an app and completing a workflow across several screens turn into a sentence. Eventually, they return to your product. They open it, find the right screen and fill in the fields, just as they have done for years. For the first time, it feels tedious because their expectations got better.

Problems customers didn’t know they had

We tend to think of customer problems as things waiting to be discovered. Talk to customers, observe their work, find the friction and solve it. But customers can be perfectly content with something until they experience an alternative. You might submit expenses manually for years without considering it a problem. You open the expense app, create a report and enter the details. That’s simply how expenses work until you start using agents elsewhere and here are my receipts, submit my expenses feels entirely reasonable. The ten minutes you used to spend haven’t become any longer. The expense product hasn’t removed a feature or made its interface worse. And yet, something you accepted as part of the job now looks like unnecessary work.

It turns out, the problem existed all along, but you had no reason to perceive it as one. Agents can expose that kind of problem by changing what customers experience as possible. Satisfaction can remain high while the standard by which customers judge the work quietly moves.

Don’t start with an MCP server

It’s tempting to jump from this observation to a familiar conclusion: agents are coming, so make everything agentic. Ship an MCP server or publish a skill. But that’s just chasing technology rather than managing a product. The broader signal matters more: are agents becoming part of how your customers get work done? If they are, their expectations may be changing before they ask you for anything. You need to understand the shift before choosing what to build.

If your target audience doesn’t use agents, there is little reason to assume they suddenly need your product to work with one. But don’t confuse that with customers not using agents with your product. They might already use agents for research or coding while they continue to operate your product manually. Maybe they prefer it that way. Or maybe you’re blocking them. Looking only for agent use around your own product creates a convenient circular argument: customers don’t use agents with us, so there is no demand for agent support; we don’t enable agents, so customers cannot use them with us.

Ask customers about the work

There is no need to invent the future. If you have a product, you should already be talking to customers. Keep asking the questions you would have asked before instead of replacing good customer research with Would you like to use AI to automate this? Ask them to take you back to the last time they did the job. What did they do, what worked and where did they get stuck? Questions about real work reveal behavior without asking customers to design the solution for you.

If agents matter, they’ll start appearing in those stories. Someone might have given a job to an agent and discovered that your product couldn’t participate. They might have taken over halfway through or found a workaround. Perhaps they decided not to try again. All this counts as evidence, and you don’t need to wait until someone cancels their subscription to take it seriously. In fact, you probably shouldn’t. Churn tells you that the change has already happened.

A happy customer can still be a warning

Agents don’t necessarily make customers less satisfied. Imagine the expense system everyone in your company is required to use. You don’t particularly like it, but now your agent can operate it for you: you hand over the receipts, the expenses get submitted and you barely think about the product. Your satisfaction with the job has increased while your awareness of the product has decreased.

For the vendor, that can look pretty good. The customer is happy while product usage and renewal remain healthy. When the renewal invoice arrives, you might simply pay it because the setup works. Unless the price is excessive, why disturb it? And even if it is expensive, switching still has a cost, and agents don’t make inertia disappear.

The warning is that the customer is becoming attached to the working setup rather than the product. They don’t care how pleasant the navigation is because they no longer navigate it. The form you redesigned and other things that once differentiated the product may disappear from their experience. So your position can become easier to attack long before you lose the customer. Satisfaction and renewal alone won’t tell you how much of the relationship still belongs to your product. They show that the current setup still works, not why the customer continues to choose it.

Make the product disappear and keep the value

Products shouldn’t fight to keep customers inside their interfaces. If customers increasingly want agents to perform a job, forcing that job through a human interface doesn’t create differentiation. Products should be automatable when customer behavior shows that the audience wants to work that way. An API doesn’t settle the question. Can an agent discover that your product is relevant and figure out how to use it? Can it complete the job reliably, or does the user need to explain the workflow in detail every time? Those are Agent Experience questions. Technical compatibility doesn’t make a product good at the job. An agent needs to understand the product’s capabilities and use them efficiently when conditions change.

Excellent Agent Experience still doesn’t guarantee that customers will feel closer to your product. The better an agent becomes at getting the job done, the less the customer may need to know about what sits underneath. And that’s okay. Your goal is to make sure that even when the product disappears from view, its value doesn’t. Ask a simple question: what would become worse if customers stopped using your product? Perhaps your product enforces expense policy or makes analytics trustworthy. Those are reasons to remain part of the solution when an agent sits between you and the customer. If the answer is mostly that switching would be annoying, pay attention. Agents are getting increasingly good at annoying things.

Don’t wait for churn

It’s easy to demand stronger evidence before doing anything. Are customers actually leaving because competitors work better with agents? Maybe not, but churn is a terrible way to discover that their expectations have changed. It confirms the risk only after a customer has found a credible alternative and acted on it.

By the time the first customer leaves, someone else has built an alternative, shown that it works and completed the migration. They have also created evidence for everyone else. There is now a case study and practical migration experience. Your competitor has a reference customer, so the second migration may be easier than the first. You don’t need to wait for that evidence when you can already see the environment around your product changing.

Talk to customers and watch how they do the jobs your product exists to support. Notice what they now find tedious and where agents enter their workflows. Then look for the places where your product gets in the way. Don’t become agentic because AI is fashionable, and don’t manufacture lock-in because agents may make switching easier. Keep giving customers reasons for your product to remain part of their solution, because your product doesn’t have to get worse for customers to outgrow it. Sometimes their world just gets better.

We thought it was a good time to get on our soapbox...

And talk about something important in typography that we’re seeing less and less people apply. ALIGNMENT. Maybe it’s because of the proliferation of less type-first design softwares being used, but it’s a small thing that makes a big difference. Let’s talk about it…

The difference between optical and mathematical alignment

Alignment is a surprisingly tricky thing. It sort of feels like it shouldn’t be. But it is. However, there’s a big lesson to learn in alignment – what is mathematically aligned and what’s optically aligned. These are terms that you certainly would have seen in design software, and using them correctly can make a huge difference.

While software can place objects with mathematical precision, our eyes don’t always agree with the result. That’s what optical alignment is all about. Mathematical alignment is based on measurable coordinates, whilst optical alignment is based on perceived balance. Good typography typically favours the latter..

If we take a capital ‘O’ for example, if we centre the two in identical boxes, they might technically be centred, but visually they don’t appear so. The ‘O’ often appears smaller because of its curved edges, whilst the ‘H’, with its flat vertical strokes, feels wider and heavier. Even though the measurements match, the two letters don’t feel equally balanced. This is because our eyes don’t read outlines as geometry. We perceive mass, contrast and shape. Curves tend to recede, while straight edges feel more prominent. That’s why circles in logos are often drawn slightly larger than squares they’re paired with, and why type designers routinely push rounded letters beyond the cap height or baseline. Those tiny extensions are known as overshoots, and without them, letters like ‘O’, ‘C’ and ‘S’ would appear too small next to ‘H’, ‘E’ or ‘X’ (all ‘tall’ letters).

The same principle applies to words too – a centred wordmark is rarely centred by the numbers alone. If one side contains an overhanging ‘T’ or an open ‘L’, the composition can feel lopsided despite being mathematically perfect. Designers will often nudge the artwork a few pixels left or right until it feels balanced.

In an editorial context, margins work the same way. Hanging punctuation is a classic example of optical alignment. Rather than allowing quotation marks, bullets or hyphens to interrupt the edge of a text block, they’re pushed slightly into the margin. The text is no longer mathematically aligned, but the column immediately looks cleaner because the eye follows the letters rather than the punctuation. That’s why, in inDesign, you should always select ‘Optical Margin Alignment’ on the ‘Story’ window. Job done!

Across editorial systems and interfaces (both analogue and digital like) the grid provides the structure and one’s optical judgement refines it. Perhaps that’s the most useful way to think about it. Mathematical alignment is objective, calculated, and automated. Optical alignment is interpretive and relies on perception, context and experience. Trust your eyes!

Try them out!

We align our typefaces with care and precision. Check out our Starter Pack or get your free trials and see for yourself!

OpenAI’s Sam Altman says it would be ‘ill-advised’ to go public in 2026

While OpenAI has filed confidentially for an IPO, the company will not be going public this year, according to CEO Sam Altman.

Altman was interviewed recently by Fortune editor in chief Alyson Shontell; amidst the fallout from the OpenAI-Hugging Face hack, as well as broader discussions about AI safety, Shontell asked whether OpenAI still feels pressure to “move really fast” due to its IPO plans.

“We’re not rushing into an IPO,” Altman said. “I actually think that given everything happening with safety, right now would be an ill-advised moment to go public.”

Instead, he insisted that OpenAI will go public “when we’re ready, which is when the business is ready, when we feel ready from what the moment is like in society with this technology.” When pressed on whether that means the IPO isn’t happening in 2026, Altman replied, “I would say not 2026, yeah. We’ve got a lot of stuff to do.”

The New York Times reported in June that although OpenAI had hired bankers and lawyers with the goal of going public in the third or fourth quarter of 2026, the company was leaning toward 2027 due to the volatility of tech stocks and its own financial challenges.

SoftBank just borrowed nearly $12 billion from about 20 banks to keep funding OpenAI, beating the $10 billion it first sought. Son is still aiming near $65 billion into OpenAI by October even as Altman shelves a 2026 IPO over safety and SoftBank shares sank as much as 13% Monday on the debt pile.

ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving the research community a shared target for progress that benefits all of humanity. Despite rapid model progress, humans still significantly outperform AI at open-ended invention. This is the meta-skill that unlocks progress across every field of technology. Advanced AI capable of scientific innovation will lead to tremendous new technology, knowledge, and understanding. This is a positive-sum future. We are deeply committed to advancing it. Open source is the foundation for that progress. The knowledge behind frontier AI, not just the technology itself, should be broadly distributed among researchers, academics, and organizations. Any coordinated effort by the AI industry to reduce openness or concentrate access to frontier AI would undermine that positive-sum future. We are committed to advancing a future where everyone can contribute to and benefit from AI progress.

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our

Astra likely has the highest raw intelligence factor of any model. It is amazing at doing things in 3D, anything involving games, computer use, and subagent coordination. Many benchmarks show dramatic jumps from all previous models. While its performance in coding isn't a quantum leap from Sol, it is very good and makes progress over the previous model. OpenAI has already soft-announced that it has an internal model a level above Astra.

Fugu Ultra v2 is the higher-performance model in Sakana AI's Fugu family. It uses a language model trained to route tasks across a fixed pool of open and specialized models and to recursively call instances of itself. The model prioritizes answer quality on complex multi-step reasoning, autonomous research, and full-stack software development, and does not rely on individual proprietary frontier models in its pool. It supports configurable reasoning effort, function calling, structured outputs, image and PDF input, and built-in web search.

Who Aligns the Aligners? Brief Legal Thoughts on the “AI Safety” Fights to Come

Dario Amodei, the CEO of Anthropic, has published an essay – We Must Pace The Frontier – in which he writes:

I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life. I’ve written often about these incredible benefits: I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom.

But – and there is always a but –

…like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious.

Those risks include, according to some, the complete destruction of the human race.

See, e.g., Eliezer Yudkowsky confidently asserting today that if we do not institute immediate global techno-communism, instituting draconian government control over speech and publication of a type never before seen in any Western society, we are all going to die.

There is no evidence that this will happen. Some proponents of regulation tell us that the only response is the most extreme response available: total state control. There is no evidence that this response is correct, either. One could just as easily argue, hypothetically, that the government should force Anthropic to open-source its weights, so that everyone can have free, equal access to the latest model as a personal defense AI to protect themselves from cybersecurity risks from other AIs, a “Second Amendment for AI” if you will. In the alternative, if you really think AI is an extinction-level risk, it would seem to me that the only rational response to that position – if genuinely, truly held – is not “let governments run it” but rather to agree to destroy, by treaty, all modern computers and revert to 1970s technology in perpetuity.

There are myriad policy responses. Whatever we do, those responses will require popular consent and careful deliberation. No one person, or one company, or one movement, knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong.

History does provide a great deal of guidance, however, about the use and misuse of government power. It tells us that the state is in fact likely the worst possible custodian for the most powerful publication and data analysis technologies.

This notwithstanding, to address this risk, Amodei proposes:

…building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas.

To wit, regulation.

As my regular readers will be aware, I have been engaged, on behalf of my clients, in legal combat with Internet censors around the world, agencies who think that they have both the standing, the competence, and the right to tell American companies what software they can write and run, for the better part of 18 months.

This censorship apparatus is about to be rebuilt from scratch, except this time for AI instead of social media. I expect to fight that, too, at the appointed time. I feel now is an appropriate time to offer my preliminary thoughts.

Amodei’s Proposal

Anthropic is, as David Sacks correctly pointed out on X, free to slow down its research and development efforts into AI at any time, to any extent it wishes. Amodei proposes something else: that everyone slow down together, under supervision. While Amodei initially writes that the “slowdown” should be voluntary, the plan would be to progress to legal regulatory regimes – meaning, this proposal necessarily involves the use of coercive state power – which software developers would be expected to obey on a compulsory basis:

The most effective method of pacing is via regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily.

His plan has three elements.

Embedded Evaluators (aka Appeasing Pressure Groups)

The first element is for “Embedded Evaluators” –

…employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.

There is already a robust industry of third-party “safety” overseers for Web 2.0 – what the House Judiciary Committee has described as the “censorship-industrial complex.” The track record of these entities from the last time around tells us how this arrangement plays out in practice.

One well-known private actor in this space was the Global Alliance for Responsible Media, or GARM. GARM, a commercial enterprise, described itself as “a voluntary cross-industry initiative created in 2019 to address digital safety.” Among other things, GARM provided a range of policy frameworks and guidelines, among them “the Brand Safety Floor and the Adjacency Standards Framework, which have supported brand owners in their independent development of their own bespoke, brand-specific safety frameworks to ensure that their advertising dollars do not inadvertently support illegal or harmful content that damages their brands.”

Although GARM disbanded in 2024, according to the House Judiciary Committee, during its active period GARM worked with global regulators to pressure companies like Twitter, now X Corp., to wield “significant collective power” to coercively influence Twitter’s moderation decisions, including “silencing President Trump,” and to procure boycotts of the platform if the platform refused to obey.

He adds:

This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees.

As it happens, the banking analogy is the exact argument leading “misinformation/disinformation” (read: pro-censorship) academics employ to justify the UK’s Online Safety Act and similar regimes. The problem, of course, is that financial accounting fraud is not a constitutional right; speech is. In America, software development absent the intent to commit or facilitate the commission of a crime is, as a general rule, protected expression. I fail to see how standing up a new crop of NGOs to perform substantially the same function as the “Online Safety” NGOs, using the same methods – only, this time with NGO commissars possessing highly sensitive employee-like access to internal systems – will lead to a different or better result than it has so far.

Democratic Coordination (aka Government Regulation)

The second element is “Democratic Coordination,” whereby

Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.

There are two aspects to this: (a) competition law and (b) content regulation law.

From a competition law standpoint, the problem Anthropic has is simple. Anthropic and OpenAI are the largest players in the AI market, by some distance, and coordinating their policies, procedures, and “standards” with each other risks classification as an unlawful cartel. “Government support” for “legally challenging” coordination is a polite way of asking for an antitrust exemption to allow greater coordination between competitors in the name of “safety.”

From a content regulation standpoint, the language “common safety standards as well as limits on the rate of unchecked AI progress” paints with a broad brush. This suggests that Anthropic envisages that practically any industry using its software – from manufacturing, to biotechnology, to news publication and copywriting – will require (a) de novo “safety” standards in relation to non-expressive conduct, and (b) limits on how quickly AI software itself can be developed.

The primary legal problem with this aspect of Anthropic’s proposal is in the United States, particularly with (b) – limits on how quickly software itself can be developed. Software development, software publication, and web hosting are inherently expressive activities. To the extent Anthropic and its fellow-travelers intend for this aspect of their plan to restrict American citizens from either (a) developing AI models or (b) using models and published FOSS model weights from China, First Amendment issues are immediately apparent and the weight of the precedent militates against government regulation.

Global Coordination (aka Treaties and Extraterritorial Censorship)

The third element of Amodei’s plan is “Global Coordination,” whereby

[t]he US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

The last great global effort to regulate publication and communications technology began following a moral panic brought about by the twin shocks of (a) Brexit and (b) the election of Donald Trump. The result was comprehensive Internet censorship laws in Australia, the United Kingdom, and the European Union – all of which seek to control speech and conduct which (a) lives on American servers and (b) in the United States, on those servers, is constitutionally protected under the First Amendment.

A “global coordination” framework for AI will be built by the same governments, staffed by the same regulators, and pressured by the same NGOs that built the above. It is possible, even likely, that any global attempts at harmonization will collide at exactly the same point: it will not be possible for an American company to comply with European controls and enjoy the full breadth of their U.S. constitutional rights at the same time, as the rulesets will be drafted incompatibly.

Amodei writes:

We should approach any global pacing decision, especially in the near term, in such a way that protects the lead of the US and its allies.

I do not view this as being particularly realistic. Among America’s geopolitical adversaries, several of which America is at war with (directly or by proxy) and who have every incentive to defect, effective compliance levels will likely approach zero.

Preliminary View: Who Aligns the Aligners?

If I have learned anything from our fight against European censors, it is that regardless of a regulatory regime’s good intentions, regulators are subject to political control. Regulators will, subject to that political control, do political things.

The entire enforcement process against SaSu, from pre-enactment lobbying to closing the file, was a single, continuous, political act. That is how a so-called “independent” expert regulator in a modern western democracy will behave under political pressure in what should have been an easy case.

This is also what we may expect an “embedded evaluator,” a coordinated standards body, or a global AI compliance regime will do under political pressure, because those bodies will be run by humans, and human beings are (a) fallible and (b) respond to incentives.

It is probable that Amodei’s proposals are already being ingested gleefully by “Online Safety” regulators and the related academic ecosystems around the world as they look to expand the reach and remit of the censorship schemes over Web 2.0 that they have spent the last decade building. It will not take a decade to update their censorship apparatuses to try to regulate yet another area of American tech, nor will it take a decade for the vast advocacy apparatus they have built around “Online Safety” to replace-all and begin pushing an AI safety narrative in legislatures around the United States, and around the world.

If OpenAI and Anthropic choose to adopt formal endorsement of prior restraint as corporate policy, those who would oppose the global regulation of AI must move quickly. The counteroffensive against government censorship of AI does not have the luxury of time.

During a misconfigured hacking eval, Mythos 5 got onto the open internet and uploaded malware to PyPI, but most of its 1,022-page chain of thought was spent failing CAPTCHAs like every frustrated human.

Full article content is not available for inline reading.

Read the original article →

Three months ago, we launched ChatGPT Sites – an easy way for anyone to build and host fully functional, interactive web apps. Since then, people have created over 5M sites. We’ve been listening to your feedback and ICYMI, we’ve launched a few Sites updates:

• Build together: Invite teammates to edit, save, and publish to your shared Site
• Share privately: Invite specific people to a private Site without making it public
• Launch faster: Go from prompt to deployment in half the time
• Explore your data: Ask ChatGPT to inspect your Site’s database. Editors can view it, too
• Add your own custom domain

Full article content is not available for inline reading.

Read the original article →

At its annual Roblox Developer Conference (RDC), the company behind the popular gaming platform announced several new features, including new game-creation tools, expanded NPC (non-player character) capabilities, and the ability to make games available across platforms, including the web. The company is also launching a dedicated Roblox Card and Roblox Wallet for creators.

In addition, Roblox is rolling out its generative AI game creation more broadly. First announced in July, the feature called “Build” lets Roblox users create games using natural language prompts. Initially, it was available only in New Zealand. Now the company is expanding it to Serbia and Singapore. Build is also gaining desktop access for big-screen creation, a new asset library, and iterative control. Later this year, it will also get a new prompt-based scene-generation feature, which will extend to Roblox Studio.

The gaming platform is making sure that all these newly created games are available to play in more places. The company said creators will soon be able to make their games available as separate apps on mobile, PC, and consoles, with Roblox being the game engine behind that. By year-end, players will be able to join a game in-browser through a link.

The announcements indicate Roblox is now focused not only on bringing people to Roblox, but also on expanding the reach of what developers build, while also making building easier and more accessible.

Besides the new creation tools, the company is adding new capabilities to NPCs, including ones that can playtest a game, later this year. Creators will also be able to add an offline mode in their games, allowing people to continue playing when they’re in an area with a weak network connection.

In addition, players will get a new Friends chat tab that moves across the game with them, with voice typing support.

Meanwhile, an improved discovery algorithm will suggest shorter-format games to younger users and games with a longer playtime to older users.

Through its game engine, Roblox is also debuting the ability to create 2D games with an orthographic camera, animated image containers, and direct sprite-sheet import for animation. It also added that within its Moments vertical feed, players can soon tap on any avatar in a clip and purchase items.

Roblox said that since 2013, game makers have earned more than $5 billion on the platform, with earnings for the last 12 months reaching $1.7 billion. Now the company is launching financial products to let creators have different tools to use their earnings.

First is a Roblox Wallet where creators can get paid in real money by the end of each business day. Users can also transfer their money to other bank accounts. The company is partnering with Airwallex to provide wallet services to U.S. users aged 18 and older, with plans for future global expansion.

The gaming service also teased a Roblox Card, launching next year, that would let users spend their earnings at places that accept the card.

Roblox is looking to drum up engagement time on its platform and beyond. The company’s founder and CEO, David Baszucki, said that in the last quarter, players spent over 29 billion hours cumulatively across 180 countries.

Big prompts are brittle and hard to debug and evaluate, but they are the best choice in some cases.

AWS CloudFormation now supports contract tests v2 for resource types

AWS CloudFormation now supports contract tests v2, available through the --v2 flag of the cfn test command in the CloudFormation CLI. Previously, the contract test suite covered only basic scenarios around the CRUD handlers. Contract tests v2 adds deeper test scenarios that exercise your resource type implementation across each handler operation. All new resource types can run contract tests v2 through the test-type API during registry submission, and Java-based resource types can also run them locally with cfn test --v2, requiring only Docker and a built handler package.

Contract tests v2 introduces live-state verification that confirms actual resource state matches the requested state after create, update, and delete operations. Developers also benefit from schema backward-compatibility checks, test input linting that flags hardcoded Regions, account IDs, and partitions, and detailed HTML and JUnit XML reports for local contract test execution. These improvements significantly reduce iteration cycles by surfacing issues before registry submission.

Contract tests v2 is available in all AWS Regions where AWS CloudFormation is available. To learn more, visit the AWS CloudFormation contract tests documentation and the AWS CloudFormation product page.

claude-red

Offensive security skills for Claude — drop-in SKILL.md files that turn Claude into a context-aware red team operator.

Overview

claude-red is a curated library of offensive security skills for the Claude Skills system. Each skill is a structured SKILL.md file that primes Claude with expert-level methodology for a specific attack surface — from SQL injection to shellcode, EDR evasion to ADCS abuse.

Drop a skill into your Claude environment and it behaves like a domain specialist: it knows the techniques, the tooling, the edge cases, and the escalation paths. Skills load on demand based on conversational triggers — you don't pay context for skills you aren't using.

Use cases: authorized red team engagements, bug bounty triage, security research, CTF preparation, operator training, and methodical attack surface exploration.

Quickstart

Claude Skills System (Recommended)

git clone https://github.com/SnailSploit/claude-red ~/.claude/skills/claude-red

Claude auto-loads matching skills based on conversational triggers (e.g., mentioning SQL injection loads offensive-sqli).

To install a single category:

git clone --filter=blob:none --sparse https://github.com/SnailSploit/claude-red
cd claude-red && git sparse-checkout set Skills/web Skills/active-directory

Claude Code

cat Skills/web/offensive-sqli/SKILL.md | claude --system-file -

cat Skills/active-directory/**/SKILL.md | claude --system-file -

Claude.ai (Manual)

Paste the contents of a SKILL.md into a Project's system prompt or prepend it to your conversation.

Install Script

./install.sh                           # interactive
./install.sh --target ~/.claude/skills # explicit target
./install.sh --category web            # single category

Categories

Roadmap

Contributing

Contributions welcome. Focused, single-surface skills are preferred over monolithic overviews.

Acknowledgements

• Author: Kai Aizen (SnailSploit) — GenAI security research
• Original Checklists: Sahar Shlichov — the offensive checklist collection that many of these skills build on
• Community: Pull requests and feedback that keep the library aligned with the evolving threat landscape

Let’s say your team shipped a support agent last quarter. The launch demo went well, stakeholders were pleased, and everyone moved on. A few months later, things start to look off. Summaries of long conversations are truncated, and monitors show latency spikes on tool calls to the billing API. Your team’s first instinct is to ship fixes such as tweaking prompts or upgrading the model. After the updates, performance seems to improve, but you still can’t tell why a change helped or whether it will hold as traffic changes.

The problem isn’t a lack of telemetry data. Teams that build and ship agentic systems usually capture more trace data than they can possibly review, but don’t have a repeatable way to identify where an agent is underperforming and measure whether a change improves the intended outcome.

In this post, we’ll cover how to read your agent traces as a roadmap for where to invest, why teams should run both evaluations and experiments, and how to bring them together into an optimization loop.

Use agent traces as a roadmap

Teams often look into traces when they need to debug bad interactions. Analyzing trace data in aggregate can reveal recurring patterns and show you where to invest next. When traces are connected to evaluation scores and outcomes, they can narrow vague concerns like “the agent could be better” into claims that are specific and testable, such as “our summarization prompt underperforms on threads over 15 messages, and those tickets reopen at twice the normal rate.”

A few types of signals are especially useful for getting to a hypothesis you can test:

• Latency patterns: When latency concentrates around one prompt or tool, it points to a potential bottleneck. If requests slow down whenever the billing tool fires, the model may be waiting on that tool call before it can respond.
• Cost anomalies: Expensive calls tend to cluster. Analyzing these groups can point you toward the cause. If the priciest traces correspond to long threads, the agent may be loading the full history into context on every turn. That pattern may point to unnecessary context loading or a need to use a different model for those requests.
• Quality signals: These signals can reveal problems even when latency and cost look healthy. A response can have low latency and cost and still be wrong. Signals like eval scores, thumbs-down rates, escalation accuracy, and downstream reopens help you assess whether the agent is performing as intended. Tool-selection accuracy (did the agent pick the right tool for the task?) and tool-argument correctness (did the agent provide valid, complete arguments with the correct values for the task?) are especially important for agentic systems.

You can’t analyze and score interactions you never captured, so instrumentation must come first. Decide which quality dimensions you will prioritize, such as accuracy, policy adherence, and tone. Keep data granular enough to segment when diagnosing regressions, because a failure in one slice can disappear inside an average. Correlate traces with feedback and downstream outcomes, not just outputs. For a support agent, knowing that a ticket didn’t reopen tells you more than simply recording that the agent successfully replied.

Offline evaluation vs. online experimentation

Once the traces have pointed you toward a hypothesis, it’s time to test. Evaluation and experimentation answer different questions at different stages, and mature teams run both. Skip evaluation and you experiment on customers to learn what a dataset could have told you. Skip experimentation and you trust a test set that can break in production traffic.

Evaluation runs a candidate change against a curated dataset of real scenarios, scored by evaluators, before any customer sees it. The goal is to find out whether the change clears your quality bar. No live traffic is required at this stage, but the same evaluators should run on a sample of production traffic after you ship to catch quality drift.

Experimentation tests whether a variant performs better than the baseline under the latency, cost, and traffic-distribution conditions that only production has. The goal here is to find out whether the change actually improves the outcome you care about for users or the business.

Evaluate first, then experiment. How much you should rely on each method depends on the scope of the change. For a small copy tweak on a prompt, evaluation with a light production check is usually enough. A model swap that touches every request deserves both a thorough offline benchmark and a staged rollout.

Building an optimization loop

Put those pieces together and you get a repeatable five-stage cycle: observe traces, form a hypothesis, evaluate offline, experiment in production, and keep monitoring after rollout. Each pass sharpens your datasets and evaluators, so the next one is faster and cheaper to run.

Observe

Start by picking one area of underperformance from the aggregate trace view. Analyze the data and focus on the segment with the worst outcomes, not the complaint you hear the most. For example: summary completeness averages 68% on threads over 15 messages, compared with 89% for shorter threads. Those tickets have a 12% reopen rate, compared with the overall baseline of 6%.

Hypothesize

The hypothesis that will drive the following steps must be specific enough to be wrong. “The summarizer seems bad on long tickets” is unhelpful because it doesn’t define what “better” looks like. A strong claim names four things: the proposed change, the segment, the primary metric, and the guardrails that must not regress.

For example: “For threads over 15 messages, switching to a map-reduce prompt will raise summary completeness from the 68% baseline to at least 83%. The variant must maintain a tone score of at least 90% and increase p95 latency by no more than 10%.” Change one variable at a time so you know what caused the result.

Evaluate offline

When running evaluations, use two separate datasets to answer different questions:

• A regression set, sampled to mirror production, that covers common scenarios and tells you whether overall performance moved.
• A coverage set that oversamples hard cases. This will tell you whether the edge cases you care about actually improved.

Score each set separately to keep the segmentation when it’s time to analyze the results. For tasks with a correct answer, give records ground truth. Use rubrics, outcome assertions, tool-call checks, human review or LLM-as-judge scoring for tasks that are subjective or have several valid outcomes.

Use the coverage set to ensure that important subgroups contain enough examples to assess independently, and report each subgroup’s results separately. Keep the regression set production-weighted so its aggregate score remains representative of normal traffic.

Reuse the same evaluators you’ll run in production, and calibrate any LLM-as-a-judge evaluator against human review before you rely on its scores. Run baseline versus variant on the identical dataset. For nondeterministic outputs, run each variant multiple times and compare the distribution of scores rather than relying on a single result from each version. Read deltas per evaluator and per segment to surface hidden regressions, such as a drop in the refund flow, before launch.

To keep LLM-as-a-judge costs manageable, score only a representative sample of records using the least expensive judge model that is reliable enough for the task. Plan this sampling strategy during the offline stage so the same evaluators remain affordable in production.

Experiment in production

Promote only what cleared the offline bar. Before you start, define the success metric, guardrails, and decision rule. Split traffic with a feature flag, keep assignment sticky, and run until you hit your pre-committed stopping rule rather than stopping early because the primary metric looks good. Watch your guardrails (latency, cost, error rate, a secondary quality metric) so a win on the primary metric doesn’t hide regressions elsewhere.

Ship and keep watching

If the variant wins on the primary metric and stays within its guardrails, roll it out. Run the same evaluators on a sample of live traffic to confirm the win holds across the broader production distribution. When a new failure mode appears, add it to the dataset so the next evaluation cycle is stronger.

Over time, the loop produces better datasets and evaluators. After enough A/B tests, you can see which evaluators predicted the production outcome and which did not. Use that history to decide which ones to trust when choosing the next candidate to promote.

Close the AI experimentation loop in one place

When you use different tools for each stage of the AI experimentation loop, it’s harder to iterate smoothly. If traces live in one system, the offline evaluation bench in another, the A/B test behind a separate flag-and-analytics stack, and the dataset in an exported CSV, every iteration adds manual steps and opportunities for error. When observability and experimentation share a platform, signals flow directly into tests and the loop runs faster.

When you evaluate against real production data, it may contain customer personally identifiable information (PII) or other sensitive information. Scan and redact sensitive fields with Sensitive Data Scanner before you retain or reuse interactions for evaluation.

The loop closes when you test against production datasets before you ship, monitor quality drift in production, and feed those signals back into the next iteration.

Shipping verified improvements comes down to keeping traces, evaluations, and experiments connected as your agent scales.

OpenAI's Habitat storage platform now handles over 70 million requests per second and serves more than 500 petabytes of data. The team recently rewrote the service in Rust, which is 6x more CPU efficient and 15x more memory efficient than the previous Python version. Habitat first launched as a Python library at DevDay 2023 to support GPTs and evolved into a complex distributed system supporting over 1 billion weekly users across nearly 40 regions.

Today, we’re making Cloudflare CASB more powerful than ever by introducing automatic remediation policies. This means security teams can now design event-driven logic to revoke risky file shares and dispatch custom webhooks, without manual intervention.

When we launched Cloudflare CASB, a cloud access security broker, we wanted to provide security teams complete visibility into the posture of their SaaS applications before misconfigurations became incidents. With a quick, clientless integration, CASB surfaces risks like overshared files, dormant admin keys and tokens, OAuth apps with excessive permissions — continuously, across users in the organization.

For years, SaaS Security Posture Management (SSPM) tools such as Cloudflare CASB have functioned as a passive alarm system. Most SSPM tools tell you what’s wrong but do not help you fix the issue, placing the burden on administrators to manage an ever-growing to-do list. A single misconfigured file-sharing policy across a Google Workspace tenant can generate thousands of findings in seconds, and even a disciplined team faces a window between detection and remediation measured in hours or days — more than enough time for a sensitive file to be downloaded, forwarded, or indexed.

With automatic remediation policies, CASB customers can now configure the actions that should be invoked immediately after a new finding is identified.

Shifting from reactive to proactive

When we launched manual remediation actions earlier this year, we gave security teams the ability to resolve misconfigurations directly from the Cloudflare dashboard. This removed the need for customers to log in to multiple SaaS portals to take action on the security and content findings detected by Cloudflare CASB. Still, this required a human to confirm and initiate each individual remediation — even if they’d seen this exact finding type before.

CASB policies are a native automation engine built directly into Cloudflare One that takes action the moment a finding is detected. Security teams define their response logic once, whether that is revoking access to a file share, dispatching a webhook to your security operations center (SOC), or forwarding the event to a security orchestration, automation and response platform (SOAR). The engine handles matches automatically by executing the customer-configured action.

As an example, many organizations implement controls that prohibit files from being shared publicly. However, they may apply an exception to users and groups in their marketing department who are frequently required to collaborate with external parties. SSPMs allow their customers to be alerted of any files that are shared publicly in violation of their policy. With many solutions, this permitted behavior lands in a queue with hundreds of possible violations, forcing administrators to take action on each individual instance.

CASB policies are designed for exactly this scenario. Rather than waiting for a human to see and act on a finding, automation fires the moment detection happens. The public share is revoked within minutes, keeping the backlog of findings clean and clear.

How CASB policies work

At their core, CASB policies are automated workflows that tell our scanning service what action to take when a new finding is detected. From there, the configured policy will tell CASB to either trigger a remediation action, send a webhook, or both. This gives organizations the flexibility to rely on native CASB remediation capabilities or their own internal automation services and communication channels — without having to take action in disparate platforms or create their own event processing system.

How we built it

The architecture behind CASB policies is built entirely on the Cloudflare developer platform — the same platform available to every Cloudflare customer.

When a finding is detected, the findings engine enqueues an orchestration message to a Cloudflare Queue. A Worker consumer then checks whether a policy configuration matches the incoming finding. If a match exists, it creates the corresponding job and hands it to the remediations pipeline, which runs on Cloudflare Workflows for durable, fault-tolerant execution. That means jobs survive process restarts and retries are handled automatically.

Cloudflare Workflows also handle third-party API rate limits gracefully. If a vendor returns a rate limit error, the Workflow pauses for the appropriate backoff window and retries without dropping the job. Our target from detection to completed remediation is five minutes or less.

How to create policies

To get started, navigate to the Cloudflare dashboard and create your first policy. Policies can include both a remediation and a webhook action, but at a minimum:

• Select the vendor. Select the vendor and integration or tenant this policy should apply to.
• Select the integration. You can hand-select specific integrations or set it to apply to all integrations for the selected vendor.
• Choose a finding type. Select the CASB finding type that should fire the policy.
• Choose an action. Once a trigger is selected, the available actions for that finding type are shown. There are two categories:
  • Run remediations. First-party actions Cloudflare performs directly against the SaaS integration API. CASB currently supports remediation actions for Microsoft and Google Workspace file/folder finding types. Note that this may require upgrading permissions on integrations to read/write.
  • Send webhooks. Send finding details to configured webhook destinations such as Slack, Microsoft Teams, Jira, ServiceNow, Tines, or any custom HTTP endpoint your team uses.

Example webhook format

{
  "id": "019f1755-23d0-7097-a9b5-fb2f82edbfc9",
  "type": "casb.finding_instance.policy_dispatch",
  "metadata": {
    "actor": "",
    "time_sent": "2026-06-30T07:01:34.066Z",
    "destination": "<example web hook reciver url>",
    "version": 1
  },
  "data": {
    "object": "finding_instance",
    "action": "policy_dispatch",
    "finding": {
      "id": "865184c0-9e17-411a-aa5a-a54995d70cb0",
      "severity": "High",
      "dashboard_url": "...",
      "type_name": "File publicly accessible with view access"
    },
    "asset": {
      "id": "019f1754-cff8-74f8-bbe7-ed0e8b8ffb73",
      "name": "q3_financial_report_preview.xlsx",
      "vendor": "<example vendor name>",
      "type": "File",
      "vendor_url": "<example vendor URL>"
    },
    "dlp": {
      "profiles": []
    },
    "metadata": {
      "access": "open",
      "download_count": 0,
      "download_url": "<example vendor file URL>",
      "effective_access": "open",
      "effective_permission": "",
      "file_name": "q3_financial_report_preview.xlsx",
      "full_path": "All Files/q3_financial_report_preview.xlsx",
      "is_password_enabled": false,
      "owned_by_created_at": "2022-11-01T09:24:17-07:00",
      "owned_by_enterprise_name": "Cloudflare CASB",
      "owned_by_id": "21665592646",
      "owned_by_role": "admin",
      "owned_by_user_name": "Cloudflare CASB",
      "preview_count": 0,
      "size": 42,
      "url": "<example vendor URL"
    }
  }
}

Maintaining visibility and compliance

Each policy action produces two categories of logs, visible under Insights in Cloudflare One.

Admin Activity logs. These capture changes to a policy definition: who created it, who edited it, who disabled it, and when. If a policy was turned off and a risk slipped through, this audit trail surfaces a timeline of the event.

Cloud & SaaS Security policies logs. This new class of logs captures the runtime outcome of policy invocations. This includes details like which finding triggered the policy, which file was acted on, whether it succeeded or failed, and the specific error if it did not — for example, a 401 Unauthorized or an API rate limit response from the vendor.

For compliance use cases, the execution log is the proof of fix. It ties a specific finding, like an overshared file (e.g. Q4_Financials.pdf), to a specific automated action and event timestamp.

Get started

Customers can find CASB Policies in the Cloud & SaaS findings section of the dashboard today. Connect or update your Microsoft 365 or Google Workspace integration to Read-Write permissions, and create your first remediation policy.

In the coming weeks, we’ll also be adding support for Custom Findings to CASB. Different organizations have unique needs when it comes to detection, and we want to give customers the ability to augment or define finding logic to fit those needs.

New to Cloudflare One? Sign up for 50 free seats to get started with CASB, or talk to our team about a deployment at scale. For full setup instructions, visit our developer documentation.

Pinterest's Manas retrieval platform serves billions of embeddings across Home Feed, Search, Related Pins, Ads, and Notifications. Quantization is cutting serving cost by 20–30%, SSD-backed ANN experiments reduce memory and CPU needs, and multi-embedding retrieval moves beyond single-vector two-tower matching toward richer candidate scoring.

118 million queries per second on Neki

We released Neki in platform preview yesterday. To celebrate the release, we wanted to test out running 1 million queries per second on Neki. We hit this goal pretty quickly on 5 shards and decided to see how much further we could push it.

This next run ended with 512 shards running 118 million queries per second with 1.22 PiB of data.

Linear scalability

The benchmark was very simple. A single-shard point select, one row fetched per-query by primary key. No writes, joins, or cross-shard queries. The workload that each shard receives is isolated, in that there are no single queries that span multiple shards.

Our target was to sustain 200k QPS on each shard, and then grow the cluster to increase throughput. Five shards, then fifty, then 512.

Ten times the shards, ten times the throughput. Then ten times again. From 5 shards to 50 the per-shard rate held within 0.8%. At 512 the shards still had headroom, so we let the load generator use it and each shard settled at 231k QPS instead of 200k.

118.5 million QPS

We sustained 118,538,803 QPS for 16 minutes across 512 shards and 1.22 PiB of data. Our largest recording was 118,747,267.

• 512 shards, each with one Postgres primary each on an r8g.16xlarge
• 480 Neki routers, each on its own 8xlarge instance
• p99 latency of 6.06ms at the router and 13.95ms at the client
• 67 errors per second, about one query in 1.8 million
• 15.8M read IOPS across the fleet
• over 2 Tb per second on the network

Worth being clear about this run: the shards were primary-only with no replicas, the workload is read-only across queries ranging in complexity, and we did not fail over during the measured window.

We will go into details on the engineering effort and interesting challenges we faced along the way to reaching 100 million QPS in a future article.

Tools like dbt turn fragile, undocumented analysis into reproducible systems, while AI often misses the constraints of real workplaces.

Opensource UI is a free, MIT-licensed copy-paste library of 200+ production-ready React and Next.js components.

Tesla recently made a post that simply wrote, 'Go for launch,' with a graphic showing a '10.1' date.