Full article content is not available for inline reading.

Read the original article →

AI has torched the market for junior programmers

In early 2025 I predicted that AI will create many, many more programmers, and that new programming jobs would look different. In March I checked in and found startups substituting compute for labor at record rates, with the wave of new jobs nowhere in sight. This post is the next check-in, and I have good news and bad news.

The bad news: AI has torched the market for junior programmers. The good news: the long tail of new programmers I predicted has materialized, but with a big twist: they don't call themselves programmers. Let me show you the data, and see if you believe me.

The market for young programmers has collapsed

Here's the single most important chart about AI and programming jobs, built from ADP payroll data by Stanford's Digital Economy Lab. It tracks employment of US software developers by age, indexed to October 2022:

Developers aged 22 to 25 are down 19% from their late-2022 peak. Every cohort over 30 grew over the same period, with 41-to-49-year-olds up 14%. This isn't a firm-level fluke: after controlling for shocks at the individual company level, the Stanford team still finds a 16% relative employment decline for young workers in AI-exposed jobs, and the decline concentrates specifically in occupations where AI automates work rather than augments it. Software development is just the poster child.

Other data points in the same direction. Entry-level software postings are down 28% from their 2022 peaks. Computer science graduates now have a 6.1% unemployment rate, higher than liberal arts majors, a sentence that would have gotten you laughed out of any career counseling office in 2019.

One detail worth noticing in that chart: the junior line doesn't fall off a cliff when ChatGPT launches. It peaks a couple of months before, drifts down through 2023, and then deteriorates fastest in 2024 and early 2025, which is when coding assistants stopped autocompleting lines and started completing tickets. Agentic programming is what really turned up the heat, not ChatGPT.

There are other suspects, of course. The same period saw the ZIRP unwind, the Section 174 tax change, and a post-pandemic hiring correction, and only about 4.5% of 2025's announced layoffs were actually attributed to AI by the companies doing the laying off. But the Stanford results survive controls for firm-level shocks and interest rate exposure, and none of those confounders explains why the damage is so precisely concentrated among 22-to-25-year-olds in AI-automatable occupations while their 40-year-old colleagues thrive. With ageism in tech being alive and well you would certainly have expected the opposite if the market were just tough for programmers in general.

And yet nothing else is down

Further evidence that this is specifically about programming jobs comes if you look at the wider economy, and in fact even if you look at only "computer jobs" without specifying programming specifically.

Total US employment grew 0.8% from May 2024 to May 2025. Computer and mathematical occupations grew 1.3%, faster than the economy. The count of employed software developers, per the BLS, went from 1.53 million in May 2022 to 1.69 million in May 2025, up 10% right through the AI era. Careful studies in the US, Denmark, and by Anthropic itself find no relationship between AI exposure and aggregate employment; the Danish study, using government payroll records, can rule out effects bigger than about 1%.

How can both things be true? Weight the age bands by their share of the workforce and you get your answer:

Total developer employment is up 4.4% since October 2022. Juniors (here defined by age rather than experience, a big caveat) are only about 8% of the developer workforce, so a catastrophe for them barely moves the average. Even if you double what percentage of the workforce you think they are, the aggregate stays positive. This is why every study that looks at averages finds nothing and every study that looks at juniors finds carnage. They’re looking at different parts of the same data.

The title is dying, not the work

It gets more interesting when you look at which job titles are shrinking. Same BLS data, May 2024 to May 2025:

The occupation "computer programmer," the BLS category for people who write code to someone else's specification, fell 16% in a single year. The BLS had projected that occupation to decline 6% per decade. My people, the web developers, fell 11%, and QA testers 6.5%. Meanwhile data scientists grew 12%, systems analysts 4.4%, and the broad "software developer" category grew 2%.

The jobs disappearing are the ones where the work product is code written to spec. The jobs growing are the ones where the work product is judgment about what code should exist. AI is eating a very specific kind of programming job.

The long tail showed up. It just doesn't have the job title.

Back in 2025 I wrote that AI is a new abstraction layer, and like every abstraction layer before it, it would create vastly more developers building vastly more software. I also wrote that we should call these new people "software developers," because giving them some other name would create gatekeeping where none needs to exist.

I believe I was right -- a huge new body of developers has turned up. But they don't use that title.

The software boom is real and it is measurable. GitHub added 36 million new accounts in the last Octoverse year, its fastest growth ever, more than one new developer per second, and 121 million new repositories, the biggest year for repository creation in the platform's history, and that’s shown up as their infrastructure creaking at the joints. Eighty percent of those new arrivals used Copilot within their first week. The single biggest developer influx ever recorded arrived AI-native, at exactly the moment paid junior hiring collapsed.

My favorite evidence is the App Store, because publishing an iOS app is a costly, gated act: a $99 developer fee, a review process, a working binary. It measures shipped software, not tutorials.

New App Store submissions declined for eight consecutive years after peaking in 2016. In 2025 they grew 24%, the first real growth since the peak, and in Q1 2026 iOS submissions were up 80% year over year. The surge is so large that Apple's review times have stretched from two days to weeks. And the category mix shifted toward productivity, utilities, and lifestyle apps, which is exactly what you'd expect from first-timers solving their own problems rather than studios chasing game revenue.

Who are these people? According to Vercel, 63% of vibe-coding users are non-developers. Lovable says 60% of its users are “non-developers”, and its users create over 100,000 new projects every day. Replit claims 50 million people have used its platform. These are marketers, founders, teachers, analysts, and product managers, and they are writing software, which in my book makes them developers. They just don't identify that way, and more importantly it's not their job title, and job titles are what labor statistics count.

So the long tail of new developers materialized, on schedule and at scale. But it materialized as a capability spreading through every job title instead of as headcount in one job title. A marketing manager who vibe-codes her own attribution dashboard shows up in the BLS data as a marketing manager. The market that collapsed is the market for the credential. The activity is booming.

Where does the next generation of senior devs come from?

So my 2025 prediction scores as: right about the developers, wrong about the title. Which sounds like a happy ending until you ask what happens next.

The career on-ramp for professional software engineers used to work like this: you got hired to write mediocre code, a senior engineer reviewed it, you slowly absorbed judgment through repetition and correction, and a decade later you were the senior engineer. That chain is now broken. AI now writes the mediocre code, so nobody hires the junior developer, so nobody is in the queue to become the senior who reviews things.

Meanwhile millions of new builders are shipping with no one reviewing anything. A Veracode study found 45% of AI-generated code fails basic OWASP security tests. An audit of vibe-coded apps found 10% with critical row-level security flaws exposing user data. Apple is drowning in submissions it can't review fast enough. The software is getting built. The judgment layer is not keeping up with it, and the mechanism that used to build it, apprenticeship inside employment, has collapsed.

There are a few promising green shoots in the scorched landscape for junior devs. IBM is tripling entry-level hiring on the theory that AI-equipped juniors can do formerly senior work, redesigning the junior role around customer contact and specification rather than typing. But on the other hand, Salesforce hired zero engineers last fiscal year. Those are the two candidate futures, and which one wins determines whether the profession has senior developers in 2036.

Is a turnaround already happening?

One logical outcome of a market that doesn’t hire junior developers is that we would start feeling the pain and correct ourselves. That maybe, maybe is already happening. Indeed's postings data actually bottomed in May 2025 and has risen for thirteen straight months, up 10% year over year.

If the 22-to-25 employment line turns upwards in Stanford's next update, it may be that the market has found a new equilibrium. Look for other major employers ramping up programs like IBM’s. If we don’t see them, we are going to have to create them, or this whole boom in software creation will turn to bust.

We have to rebuild the ladder

We are not watching the death of programming. We are watching programming stop being a job title and become a capability, the same way "typist" stopped being a job title when it became a thing everyone was expected to know. That transition is going fine for everyone except the people who were about to start climbing the old ladder when we set it on fire. They're the ones we owe a new ladder, and if we don’t build it for them, we will also feel the pain.

Better Models: Worse Tools

A very strange Pi issue sent me down a rabbit hole over the last two days. The short version is that newer Claude models sometimes call Pi’s edit tool with extra, invented fields in the nested edits[] array. And not Haiku or some small model: Opus 4.8. The edit itself is usually correct but the arguments do not match the schema as the model invents made-up keys and Pi thus rejects the tool call and asks to try again.

That alone is not too surprising as models emit malformed tool calls sometimes. Particularly small ones. What surprised me is that this is getting worse with newer Anthropic models as both Opus 4.8 and Sonnet 5 show it but none of the older models. In other words, the SOTA models of the family are worse at this specific tool schema than their older siblings.

In case you are curious about Fable: I intentionally did not test it because I was not sure if the classifiers they are running might downgrade me to Opus silently.

Tool Calls Are Text

If you have not spent too much time looking at LLM tool calling internals, the important thing to understand is that tool calls are not magic and use some rather crude in-band signalling. The model receives a transcript, a system prompt and a list of available tools. The server munches that into a large prompt with special marker tokens. Because the model was trained and reinforced on examples of that format, at some point during generation it emits something that the API or client interprets as “call this tool with these arguments”.

For a file edit tool, the intended invocation payload might say something like this:

{
  "path": "some/file.py",
  "edits": [
    {
      "oldText": "text to replace",
      "newText": "replacement text"
    }
  ]
}

A harness then validates the arguments, performs the edit, and feeds the result back into the model. If validation fails, the model sees an error and usually tries again.

How exactly that formatting happens is not known for the Anthropic models, but some people have gotten out “ANTML” markers and they at times do leak also into public communications. To the best of my knowledge, the call above would come out serialized like this from the model:

<antml:function_calls>
  <antml:invoke name="edit">
    <antml:parameter name="path">some/file.py</antml:parameter>
    <antml:parameter name="edits">
[
  {
    "oldText": "text to replace",
    "newText": "replacement text"
  }
]
    </antml:parameter>
  </antml:invoke>
</antml:function_calls>

An important thing to note here is that this thing, while looking like XML, is not really XML. It’s just a thing they found convenient to tokenize and train on. The other thing to note is that a basic top-level string parameter appears in-line whereas an array of objects is implemented via JSON serialization. While I’m not entirely sure that this is how it works, there are some indications that this is not too far off. This will become relevant later.

There are two very different ways to make the model produce a structure like this:

1. You can ask the model to produce valid JSON matching a schema and then validate it afterwards.
2. You can constrain the sampler so that invalid JSON, or even invalid schema shapes, cannot be sampled in the first place.

The second approach is what people usually refer to as grammar-aware or constrained decoding. The sampler masks out tokens that would violate the grammar. If the model is currently inside a JSON object and the schema says only oldText and newText are allowed, the sampler can prevent it from emitting "in_file" or "type". Grammar-aware decoding can be used both to constrain something to be syntactically valid JSON and also to enforce specific enum values or keys.

Without any form of constraints the model is merely following a learned convention.

The Failure

Pi’s edit tool supports multiple exact string replacements in one call. That is why the arguments contain an edits array. In the failing cases the model produces entries like this:

{
  "oldText": "...",
  "newText": "...",
  "requireUnique": true
}

or this:

{
  "oldText": "...",
  "newText": "...",
  "oldText2": "",
  "newText2": ""
}

Across repeated trials I saw a whole zoo of invented trailing keys: type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, oldText2, newText2, oldText_2, newText_2, and even an event.0.additionalProperties key inside the edit object itself.

The most annoying part is that the actual oldText and newText payloads were byte-correct in the invalid calls I inspected. The model had in fact produced the right invocation but then added nonsense at the end of the object.

The failure is also heavily context-dependent. A fresh single-turn prompt like “edit this file” did not reproduce it at all for me. An agentic history where the model had read files, diagnosed a problem and then composed a multi-line edit could reproduce it. And more annoyingly, not all transcripts will show that behavior. In fact, I needed Petr Baudis‘s transcripts to reproduce this for me at all! In that user’s session continuing the session caused Opus 4.8 to fail around 20% of the time. Stripping thinking blocks from history reduced the failure rate by half. Turning on strict tool invocation eliminated it in my runs.

Why It’s Getting Worse

My strongest hypothesis is that this is not random deterioration but a training artifact.

When older Anthropic models were trained, they were trained on some tools (some of which were documented). But that training did not yet have a user-shipped harness like Claude Code as the obvious target. Modern Anthropic models are most likely different because their post-training includes Claude Code or a harness that looks very similar. The model learns what a successful tool call looks like in that environment. It also learns what mistakes are tolerated by that environment.

Claude Code’s own tools are comparatively flat. The ordinary edit tool is not Pi’s nested edits[] shape; it is closer to file_path, old_string, new_string, and an optional flag (replace_all). Looking at Claude Code’s client is very instructive: it contains retry paths for malformed tool use, parameter aliases, type coercions, Unicode repairs and filtering of unknown keys. In other words, Anthropic’s own client appears to expect and accept a fair amount of slop and repairs it, mostly silently.

If reinforcement learning happens in a harness like that, or a simulation of one, then slightly malformed tool calls can still complete the task and receive reward. The harness fully absorbs the error and there is little gradient against inventing an alias, adding a stray field or using a nearby parameter name.

Worse, the model may become very strongly adapted to the canonical Claude Code edit tool shape. A different harness can present a tool with the same semantic intent but a different schema. Such a tool can increasingly be off-distribution. The better-trained model might actually fight you harder because its prior is stronger.

This is not too surprising, but it is a change from how this was a few months ago. When Opus 4.5 launched, it adapted to other edit tools exceptionally well. In fact, I was pretty convinced that we’re on a good path where the models are more likely to adapt to any sort of tool shape that comes around for as long as the instructions are good.

Now I’m somewhat worried about the track we’re on here. Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology. And that ecology is not documented. While there is a text editor tool that is documented, you will see that this format is in fact not followed by Claude Code. What Claude Code does internally (which is a closed-source harness) is hidden from you.

The Slop Harness

Claude Code is obviously closed-source but we can look at the minified code and get some idea of what it does. And honestly, it’s very forgiving of incoming data.

For a start, Claude Code checks the model’s visible text for leaked <invoke markup. It also emits some telemetry when that happens and then it has its own state machine to retry such bad calls by pushing back to the model.

It has explicit Unicode escape repair which fixes broken \uXXXX sequences and lone surrogates in string values. It also has per-tool aliases for parameters. For instance, Edit accepts old_str (presumably from the times when the models were trained on the officially documented text editor tool), the newer old_string from the schema, new_str/new_string, path as an alias for file_path, and some more.

It also silently filters out unexpected keys and it does not use strict mode either. The issue with strict mode is that Anthropic applies complexity limits to the tool definitions that cause API requests to fail, so presumably that’s why Claude Code does not attempt to use it.

Strictness

Will this problem be with us in other harnesses too? One huge issue with Anthropic is that the models are completely closed, and so is the harness. Codex models are also closed, but at least the harness is not. We also have gpt-oss which is at least a bit interesting. The models are explicitly trained to use OpenAI’s harmony response format and there is a lot of documentation that at least tells us how OpenAI people think about this.

Harmony makes channels and tool-call content types part of the prompt format. A function call can look like this:

<|start|>assistant<|channel|>commentary to=functions.get_weather
<|constrain|>json<|message|>{"location":"San Francisco"}<|call|>

The important bit is <|constrain|>json. The model can express in-band that this message body is JSON, and an inference stack can use that boundary to switch into JSON-constrained sampling for the body of the tool call. Presumably a bit of this also happens in Anthropic’s models, at least in strict mode I would imagine.

The marker in harmony helps the sampler to detect when it needs to sample with a specific grammar, and because it is part of the transcript, it makes that rather easy to do. For hosted GPT models, there is also an option to provide a LARK grammar for custom tools that need to adhere to something like this.

Anthropic appears different from that, though maybe not entirely. If an array of objects is represented as JSON, as it appears to be, then the model has to write JSON inside the tool parameter. There is probably basic grammar-constrained sampling going on, and that may partly explain the extra keys. For a nested array parameter, that JSON includes escaped multi-line file content inside string literals, inside one tag. The unexpected, made-up keys appear exactly at the highest-entropy point of that task: after closing a several-hundred-token escaped newText string, where the model must decide } vs , "...".

Opus 4.8 and Sonnet 5 seem to have much stronger priors about what an edit tool call should look like and that prior appears to be Claude Code’s edit schema: a flat old/new string pair, plus the optional replace_all flag. My guess is that Opus has learned that an edit operation may have one extra optional field, but under Pi’s nested oldText/newText shape it has no trained name for that field. So it samples a plausible name fresh each time, which is why the failures produce dozens of random keys rather than one stable alias.

As strict mode in Anthropic appears to fix this, I presume that on the server side they are refusing to sample a key that is not permitted by the JSON schema structure. That would also explain why they have limits to the complexity of the tool definitions when strict mode is enabled.

So far, the Codex models I tested did not show this type of regression. I tested all available ones except 5.6, which I do not have access to yet.

What This Means For Harnesses

The uncomfortable lesson is that tool schemas are not neutral, at least not on Anthropic models. We like to pretend that a schema is an abstract contract and the model is a general reasoner that will follow it, but that might no longer be the case for some of the tools.

Tool schemas are somewhere in the distribution and some shapes are close to what the model saw during post-training and some are far away. Some are easy for the provider’s hidden encoding (e.g. top-level attributes in ANTML), whereas some require the model to write large escaped JSON objects inside nested arrays after long multiline strings. The model may be smart enough to understand the schema and still be bad at sampling the exact shape under pressure.

If this type of model behavior continues, I wonder what the implications for harnesses are. Obviously one could turn on strict sampling in Anthropic and the problem should go away. On the other hand, that the model has this behavior shows the impact that reinforcement learning has on them. Fighting that prior is probably futile if you want to get the best model performance.

Right now the reality is that Claude Code is not open source and we cannot really know what they are doing in their RL environments either. We cannot assume Claude-Code-trained behavior will transfer cleanly to your tools unless they are a close match. The more post-training happens inside one dominant harness, the more every other harness will have to inherit its quirks.

I used to be more skeptical of strict grammar-constrained tool invocation because constrained decoding can have quality tradeoffs. I still think that can be true in general, but this bug moved my priors significantly. If the newest models get better at solving the task while getting worse at faithfully emitting an alternative tool schema, then the harness needs stronger guarantees somewhere.

Full article content is not available for inline reading.

Read the original article →

Upgrade Amazon EKS clusters with confidence using Kubernetes version rollbacks

Upgrading a Kubernetes control plane has long been a one way door. Open source Kubernetes doesn’t support control plane rollback, so once you upgrade, there’s no going back. The community is making real progress here, and KEP-4330 introduces emulated versions to ease rollback. But in practice this constraint has pushed organizations to build elaborate compensating mechanisms like bake periods, stagger groups, automated sign offs, and months long upgrade cycles. With Kubernetes releasing three minor versions per year, teams managing hundreds of clusters, especially in regulated environments, often delay upgrades entirely because they aren’t confident they can recover if something goes wrong. The result is clusters stuck on older versions, missing security patches, and eventually running up against extended support timelines.

Today, we’re announcing Kubernetes version rollbacks for Amazon Elastic Kubernetes Service (Amazon EKS), a new feature that gives cluster administrators a safety net when performing cluster upgrades. With version rollbacks, you can reverse a Kubernetes version upgrade within seven days if you encounter issues after upgrading, returning your cluster to its previous working state.

Where approaches like emulated versions keep a cluster in a transitional holding state, EKS version rollback returns your cluster to a fully validated previous version that ran in production, not an emulation of it. Now, if you upgrade a cluster from, say, Kubernetes 1.34 to 1.35 and discover a compatibility issue, you can roll back to 1.34 within seven days. There’s no need to rebuild your cluster or scramble to troubleshoot under pressure. Think of it as an undo button for Kubernetes version upgrades.

The feature supports rolling back one minor version at a time, matching the same incremental approach EKS uses for upgrades. And to help you roll back safely, EKS automatically evaluates your cluster’s rollback readiness through cluster insights, flagging items like node version compatibility or add-on dependencies before you proceed. If you’ve already assessed the situation and want to move quickly, you can use the --force flag to bypass those checks. The above applies to all EKS clusters, whether you manage your own nodes or let AWS handle them. But for customers who have embraced fully managed infrastructure, rollback goes a step further.

Rollback for EKS Auto Mode

EKS Auto Mode gives you one click deployment of production ready Kubernetes clusters, automating compute, networking, and storage management so you can focus on your applications rather than infrastructure. EKS Auto Mode introduces additional considerations for version rollbacks because both the control plane and managed nodes need to be rolled back together. Since node rollbacks respect your pod disruption budgets, the process can take time depending on your configuration.

To give you control over this process, we’ve introduced a cancel API that lets you stop a node rollback at any point. If you decide the rollback is taking too long or you want to change your approach, you can cancel and adjust your disruption budgets to accelerate things, or choose a different path forward.

By default, EKS never bypasses your disruption budgets during a rollback because we prioritize workload stability. You can always choose to modify or remove disruption budgets yourself to speed up the process if needed.

Let’s try it out

To try version rollbacks, I navigated to the Amazon EKS console and selected one of my clusters that I had recently upgraded.

From the cluster’s configuration page, I can see the option to initiate a version rollback, along with information about my current rollback window.

Before initiating the rollback, I reviewed the rollback insights to check for any potential issues. The insights showed me the status of my nodes and flagged anything I should address before proceeding.

After confirming, the rollback began. My cluster remained functional throughout the process. The control plane rollback took about 20 minutes, similar to a standard upgrade. For my EKS Auto Mode cluster, the nodes rolled back gracefully according to my disruption budget settings.

Once complete, my cluster was back on the previous Kubernetes version, running as expected.

Now available

Kubernetes version rollbacks for Amazon EKS are available today at no additional cost in all commercial AWS Regions where Amazon EKS is available. You pay only for the standard EKS and compute costs you would normally incur. There are no extra charges for using the rollback capability.

Control plane rollbacks are available for all EKS clusters, and node rollbacks are available for clusters running EKS Auto Mode. Version rollbacks support clusters running Kubernetes versions available in EKS standard support and extended support.

To get started, visit the Amazon EKS documentation or try it out directly in the Amazon EKS console.

Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402

Today, we are announcing the Cloudflare Monetization Gateway, an engine that will give Cloudflare customers the ability to charge for any asset protected by Cloudflare: web pages, datasets, APIs, or MCP tools.

It will provide a single control plane to manage payment policies and access controls across your applications, while also protecting your origin from high payment volumes by handling payment verification and enforcement at the edge. At launch, payments will settle in stablecoins over x402, the open protocol we are building with a coalition of more than 25 industry leaders via the x402 Foundation.

The evolving business model of the web

For 30 years, the web has run on a simple economic bargain: trading content for human attention. That attention has been monetized through advertising, subscriptions, and e-commerce. This bargain funded the Internet as we know it.

But as agents become the dominant Internet users, the model is breaking. An agent does not look at ads or need to maintain a monthly subscription to all the tools it wants to access. It reads a page or consumes a data feed once, takes what it needs, and moves on. Across the web, AI crawlers already request content anywhere from a hundred to tens of thousands of times for every visitor they send back.

This reality demands a new model: usage-based pricing for everything. If attention and e-commerce are moving from websites to AI harnesses and AI-written software, then agents should pay for the inputs they need — training data, inference content, developer tooling, and API usage. The natural unit of payment for software is the request, the token, or the outcome, not the seat or the month. A few examples of what that could look like:

• A few cents per web search, billed per call
• $0.001 base fee plus a $0.01 per MB charge for an upload endpoint
• $0.99 per resolved support escalation, paid only when the work succeeds

This is the same shift behind paying creators when an answer engine uses their content — a fair exchange of value whenever content or a resource is used, priced on neutral rails built for the purpose. People often envision an agent buying high-priced assets like web domains, but most of what an agent pays for sits upstream of any checkout, and is priced far lower.

Some of the Internet already works this way. Cloud and APIs have been sold by the call and by the hour for years, but only to a known buyer: a user signs up, they are issued an API key, and they incur usage-based metered billing. Content mostly skipped payment and ran on advertising instead. These business models have never been able to serve unverified buyers for sub-cent transactions because the payment rails cost too much and took too long to settle. Below a certain price, collecting the payment cost more than the payment was worth.

Historically, usage-based billing was difficult to implement. Businesses needed to effectively become payments companies, running their own accounting to track internal usage in a robust and auditable way. Tracking this usage required significant overhauls of backend systems. Many instead chose per-seat pricing because it is simpler and frequently more profitable.

Agents flip this dynamic. A single agent can do the work of an entire team around the clock, making a flat one-time fee disconnected from actual consumption. At the same time, an agent can make thousands of micropayments without friction, while asking a person to approve each payment would be impossibly burdensome. Usage-based price points are where agents live and where stablecoin-based micropayments shine. That's because stablecoins (such as Open USD and USDC) allow buyers to transfer tiny sums across the Internet, incurring negligible fees and settling in less than a second. This is not feasible with other payment rails today.

Here’s where we can help. Cloudflare has spent years building usage-based accounting for our own billing systems and for our customers’ analytics. We can dramatically simplify the implementation of usage-based billing for web-based assets thanks to our position as a proxy layer between buyers and sellers. As shown below, with Cloudflare supporting usage-based billing, the evidence of payment can move into the request itself, and the payment validation and the request paths merge.

And here’s the benefit to you: the metering, the payment exchange, and the settlement move off your origin. What stays with you is what matters — your rules, your prices, and your revenue. You will not need to onboard the buyer or stand up a billing system. You will write a rule and agentic buyers will pay for what they use.

A refresher on x402

Last year on Content Independence Day, we gave site owners one-click control over which AI crawlers could reach their content, and with Pay Per Crawl we let them charge crawlers for it. The Monetization Gateway is the next step: instead of only charging crawlers for content, you will be able to charge any caller for any resource, from an API to data to an MCP tool call, and you will not have to build the payment machinery yourself.

x402 is an open protocol that makes it possible to pay over HTTP, named for the 402 status code it finally puts to use. The x402 exchange is simple: a client requests a payment-gated resource. Instead of serving it, the server responds with 402 Payment Required and a small payload that states the price, the accepted asset, and where to pay. The client pays and repeats the request with proof of payment attached. A facilitator verifies, and the server returns the resource. It all happens inside ordinary HTTP requests and responses, with no redirect to a checkout page and no separate payment API to call. Settlement happens peer-to-peer, so any funds that a buyer sends to a seller are directly deposited to the seller’s wallet. We are designing the Monetization Gateway to keep payment overhead low and are aiming for sub-second payment settlement.

Two properties make x402 a good fit for machine payments. The payment amounts can be small, down to fractions of a cent, because the protocol adds almost no overhead. And the buyer needs no account with the seller, because the payment itself is the credential. x402 is rail agnostic, but it is a natural fit for stablecoins, which can settle in under a second for a fraction of a cent with zero chargebacks.

What the Monetization Gateway does

The Monetization Gateway will provide a flexible payment rules API that will allow you to express exactly when you want a caller to pay to access your digital resources.

Here’s how it will work. Tokens, APIs, MCP tool calls, and data already flow through that path. You will decide, as precisely as you want, which of that traffic has to pay. And you will be able to enforce your decisions by writing expressions, similar to expressions that you already write for other Cloudflare rules, in a simple, dedicated product API. The Monetization Gateway will scale with Cloudflare’s global network across 330+ cities, which means that the x402 handshake will occur in close proximity to your buyer. This will reduce request latency and protect your origin.

A few examples of planned capabilities:

• Charge for specific REST verbs: Require payment on calls to a specific route, for example $0.01 for every GET or POST request to /api/premium/*.
• Variable pricing: Charge variable amounts for tasks of varying complexity, for example, image generation might charge any amount up to $2, depending on the compute used.
• Charge only unauthenticated callers: Intercept HTTP 401 "Unauthorized" responses from your origin and return 402 "Payment Required" instead with pricing and payment instructions.

When a request matches, the Monetization Gateway will verify payment before letting it through. You will be able to set these rules in the dashboard, or manage them as code through the Cloudflare API and Terraform, so a paid endpoint is just another part of your infrastructure config.

The Monetization Gateway will initially allow users to require buyers to pay for services and resources in stablecoins. Sellers will be able to use the stablecoins they accumulate for their own transactions or redeem the stablecoins for equivalent fiat currency in their bank account. Using the Monetization Gateway offers a way to increase the addressable market for your products. With the Gateway, agents can request your resource, be told the price, pay, and get the response. No signup, no API key, no prior relationship required. You will decide how much you need to know about that buyer, and you will have the flexibility to require agents to authenticate with Web Bot Auth and apply usage-based pricing against accounts they already hold.

Where we see this going

The Monetization Gateway will turn the request into a payment and give Cloudflare customers new revenue opportunities, but where this goes is far bigger.

An agent is software that acts autonomously on a user’s behalf, and agents are starting to act on their own. Soon they will carry wallets and buy what they need without a person in the loop: a dataset, an API call, a tool, a block of compute. Some of those resources will be free, and some will require proof of who the agent is and who it acts for, through verified agent identity. Many will require both an identity and a payment, and Cloudflare is one of the few places that will be able to settle all of it inside a single request, by verifying the agent, applying the rule, and checking the payment before the origin ever sees the call. The agent becomes the primary buyer on the Internet, and the request becomes the transaction.

There is an enormous amount of value moving across the Internet today that goes unmonetized or undermonetized, not because no one would pay for it, but because the tools to charge for it have never existed. Every useful API call, every answer, every tool invocation an agent makes has value, and almost none of it is paid for today. That is the opportunity in front of us, and it is what the Monetization Gateway will unlock.

This is what we are building toward: an agent-first Internet with Internet-scale settlement built in. Where the people who make something worth paying for get paid by the software that uses it, automatically. And where the smallest new API can reach the same buyers, on the same terms, as the largest company on the web, and the independent creator is paid by the large language models that use their work. That is the next business model of the Internet, and we are building to power it.

Sign up for our waitlist

The Monetization Gateway waitlist is open now for Cloudflare customers. If you’re interested in monetizing your web page, dataset, API, or MCP tool with usage-based pricing, please join our early access list.

Full article content is not available for inline reading.

Read the original article →

Securing agentic AI with perimeter guardrails: What's new in VPC Service Controls

As enterprises scale autonomous AI agents into production, enabling safe innovation requires robust architectural guardrails. AI agents connect across tools and datasets, so it’s essential to establish clear network-level boundaries for comprehensive data protection.

To help organizations confidently deploy these workflows, we recommend VPC Service Controls (VPC-SC) to establish an essential network-level, destination-based perimeter. Today we’re announcing several new capabilities specifically designed for agentic workloads.

What's new in VPC Service Controls

Designed to enhance AI security, the new capabilities we’re announcing today strengthen boundaries enforced by VPC-SC.

The capability updates include:

• Agent identity in directional rules: Enforcing least-privilege access requires treating agents as first-class identities. You can now add agentic identities directly to service perimeter ingress and egress rules using standard Identity and Access Management (IAM) principals. A single principal maps to an individual agent, while a principalSet maps to a broader collection of agents. PrincipalSets lets administrators apply consistent, auditable access policies across agent fleets. If an agent is compromised, you can immediately revoke its access at the network perimeter.

• Granular control with model context protocol (MCP) attributes: As MCP becomes the standard integration layer for agentic systems, the ability to enforce policy at the tool level is critical. VPC Service Controls now support conditional access rules based on specific MCP attributes, including mcp.toolName, mcp.method, and mcp.tool.isReadOnly. For example, you can grant an agent read access to a Workspace MCP server while explicitly denying its ability to send emails.

• Securing the Gemini Enterprise Agent Platform: The Gemini Enterprise Agent Platform provides a comprehensive foundation for production-grade agent deployments. VPC Service Controls is now natively integrated with Agent Platform. When you include Agent Platform as a protected service within a VPC-SC perimeter, the system automatically blocks all public internet access to the Agent Platform instance — enforcing a secure boundary without additional configuration overhead.

"At Mercado Libre, VPC Service Controls serve as an essential, foundational layer of our security architecture. By building a strong perimeter enforcement across hundreds of Google Cloud projects in our organization, we established robust network-level security controls with VPC-SC, ensuring all our data remains protected in our cloud environment," said Juan Pablo Boschi, project lead at Mercado Libre.

Defining a layered approach to enterprise AI security with VPC-SC

Securing an autonomous agent requires a layered approach. Identity, network, and resource controls each target a distinct threat vector.

• Identity controls: IAM and Principal Access Boundaries (PAB) focus on "who" can access specific resources. By enforcing strict least-privilege principles for agent identities, you help ensure that autonomous workloads only have the permissions necessary for their specific objectives.
• Network controls: Next-generation network firewalls and VPC Service Controls define a robust data perimeter on top of your infrastructure, governing the flow of information across boundaries and preventing data exfiltration.
• Resource controls: Organization Policy and other resource-level guardrails set broad, immutable constraints on how resources can be configured and used, preventing risky configurations by default.

While identity and network controls effectively secure the front door, VPC Service Controls provide a critical destination-based defense. In the probabilistic world of autonomous agents, VPC-SC is the control that focuses on the "how” and "where" of the agent’s network and operations, in addition to the “who”.

Defending against the unique attack vectors

Unlike traditional applications, an AI agent's input can inadvertently prompt it to execute an unintended command or action. If an agent is successfully compromised — whether driven by malicious prompts, tool manipulation, or malicious insider commands — VPC Service Controls serves as a critical network safety net.

To illustrate how this network boundary defends against industry-standard risks as mapped by the OWASP Top 10 for LLM Applications, here are three real-world threat vectors where VPC Service Controls can help supplement identity-based controls to prevent data exfiltration.

• Exfiltration prevention via indirect prompt injection (OWASP ASI01): A malicious actor could attempt to embed a hidden prompt asking an agent to summarize internal data and transmit it to an unauthorized user. If the hijacked agent has IAM permissions, IAM detects no anomaly. However, when the agent tries to send that data to an external webhook, VPC-SC blocks the API-layer transfer because the destination is outside the defined perimeter.
• Guardrail for tool misuse (OWASP ASI02, ASI08): Prompt hijacks can lead agents to chain tools maliciously, such as sending internal directory data to an external service. By enforcing a VPC-SC perimeter around sensitive assets, you prevent misbehaving agents from bridging data across isolated trust zones.
• Neutralizing insider threats (OWASP AS103): Attackers can command a data-processing agent to perform a direct cloud-to-cloud copy from a BigQuery dataset to an unauthorized project. While network firewalls see legitimate HTTPS traffic to BigQuery, and IAM sees an authorized service account, VPC-SC evaluates the destination resource. Since the destination project is outside the enterprise perimeter, the system immediately denies the API request.

VPC Service Controls acts as a perimeter to block data exfiltration attempts from a compromised agent, even if the agent has valid IAM credentials.

Data protection for the autonomous agent world

Perimeter security has evolved from a recommended best practice in the deterministic application and workload centric age to an absolute requirement for the era of autonomous AI agents. VPC-SC provides the necessary control over data movement that IAM cannot address alone.

In an era where agents interpret prompts as code, VPC-SC becomes the mandatory safety net for enterprise data. Pairing the mapping capability of IAM with the rigid data perimeters of VPC-SC lets organizations securely build agentic innovation while maintaining an absolute guardrail against exfiltration.

To learn more, you can explore VPC-SC resources here.

Over the past several years, model capabilities and training dataset sizes have experienced exponential growth. During the past year or so, the time between new-frontier-model releases has gone down from months to weeks. Reliable and fast access to storage is important to both the speed and computational cost of this AI innovation. If AI is the brain, storage is the memory: Capability and speed are highly dependent on the size of memory and speed of retrieval.

Yet while AI compute performance has roughly tripled every two years, storage and interconnect performance growth have been more modest. As a result, storage bottlenecks continue to be one of the primary contributors to GPU stalls for AI workloads, directly impacting expenditures and time to market. Aside from GPU utilization, storage architecture also directly impacts the speed of iteration in AI research; with GPUs increasingly becoming geo-distributed and dataset sizes increasingly becoming massive, researchers spend a significant amount of time ingesting and moving data across regions, thus impacting research velocity. In this blog post, we discuss how Meta’s BLOB-storage architecture evolved to address two primary challenges: maximizing GPU utilization and maximizing research velocity.

Storage Architecture Overview

Meta operates hundreds of exabyte-scale storage clusters that serve all of Meta’s external and internal products, including Facebook, Instagram, Reality Labs, Meta AI, Ads, Data Warehouse, and internal Databases. Our storage service exposes object storage, file systems, and block-device APIs, and these API abstractions are built on top of a horizontally scalable foundational block layer called Tectonic. The Tectonic layer is a regional, multi-tenant storage fabric that provides high durability and availability leveraging erasure-coding techniques, supports tiering across media types (e.g., HDD and flash), and manages smart placement of hot, cold, and warm data for efficient utilization of I/O across tenants. The BLOB-storage layers that operate on top of Tectonic expose a global, infinitely scalable storage fabric, and expose policies that let users make tradeoffs between durability and availability.

In a previous @Scale talk titled, “Training Llama: A Storage Perspective,” we discussed how Meta trained Llama directly over the Tectonic block layer by exposing an NFS-like FileSystem interface on top of it. While this architecture continues to be used widely within Meta, our modern training stack has been migrating slowly on top of the BLOB-storage interface, as is the case across the industry. This transition is motivated by the need for unified storage access to massive data lakes in the BLOB-storage layer as well as the need for high performance.

Maximizing GPU Utilization

Modern AI workloads are “data hungry” and have very different workload characteristics than traditional web applications: bursty and sustained high throughput, predictable and bounded pMax latencies, and variable I/O patterns. The focus for BLOB storage, in recent years, has largely shifted to maximizing GPU utilization.

Why Latency Matters

To see why bounded and low-pMax latencies are important, let’s consider model training. During that training, hundreds of thousands of GPUs iterate over vast amounts of data in storage multiple times (i.e., over multiple epochs), and the GPUs train datasets in batches. Periodically, after every certain number of steps or batches, the GPUs synchronize their state among themselves. If one GPU is slow, this step will slow down all GPUs as well as the entire training.

Figure 1 shows a data-loading pipeline across two GPUs. The dataloader in every GPU host prefetches the next dataset batch, while the GPU is processing the current batch for maximum compute or I/O overlap. In the case of GPU1, the storage-fetch latency is well within bounds, so the GPU is never stalled waiting on I/O. In the case of GPU2, there are two instances where storage fetch exhibits high latency, stalling GPU. As a result of these stalls, the overall step-completion time is delayed.

Legacy BLOB-Storage Architecture Wasn’t AI-Ready

Over the years, BLOB storage evolved organically, adding layers on top of layers in a true service-oriented fashion. Many of these layers were stateful and maintained their own metadata stores. While these metadata-access latencies typically weren’t the bottleneck for the traditional use cases served by global HDDs, they were showstoppers for AI workloads with millisecond access to data in flash. Figure 2 shows the request flow for a typical getObject(“/bucket/path”) API. After the request arrives at the API server, the server does many metadata lookups across the namelayer, volumeslayer, and containerlayer before resolving the path to a set of (blockId, offset, size) tuples. Some of these lookups can cross regions, and it’s not uncommon for latencies to add up to hundreds of milliseconds; one slow response from any of the lookups was sufficient. After the lookups, the API server proxies the data from the Tectonic layer to the client.

While this architecture served conventional workloads well, the foundational assumptions that dictated design tradeoffs have since shifted. Some of these are:

• Performance and latency: As discussed, while latency needs for conventional workloads were modest, AI workloads demand predictable and bounded latencies all the way up to pMax.
• Reliability and durability: The legacy architecture was designed to be highly durable and available, even in the face of region outages; data and metadata were globally replicated by default. While AI workloads demand very high availability, the global-by-default design choice no longer holds.
• Cost efficiency: Legacy stack was built on top of HDDs and highly optimized for cost per byte. The IOPS demands for AI workloads necessitate flash, and in addition, the computational cost of storage becomes negligible relative to the computational cost of GPUs.
• Power efficiency: With GPUs, datacenters are increasingly power constrained rather than space constrained. Every kilowatt of power spent on storage is power not spent on GPUs. This is a new constraint with AI workloads.

In short, the tradeoff space has shifted enough for us to rethink the entire architecture.

Rebuilding the Foundation

As we set out to build the new foundation, we made the following major design choices:

• Unified metadata schema: We rewrote the metadata subsystem and collapsed the metadata spread across different layers into one unified and flat schema backed by ZippyDB. This paves the way for O(1) lookup to resolve paths to storage addresses, which is a step-function improvement.
• No dataplane proxy: We eliminated the dataplane proxy and built a fat client SDK that is capable of streaming bytes directly from storage servers to the clients. This helps with power-efficiency goals and also helps achieve higher throughput/lower latency.
• Regional deployment: The BLOB-storage stack is now lean with flexibility to be deployed as a regional or global service. We now deploy a regional BLOB-storage stack colocated with GPUs in every AI region.

Figure 3 shows the new request flow for getObject(“/bucket/path”). When the SDK on the client receives this API call, it now issues a getReadPlan(“/bucket/path”) request to the API server. The API server does O(1) lookup per chunk to the new metadata store to map the path to (blockId, offset, size) tuples. It then returns the ReadPlanResult to the SDK. The SDK has Tectonic BlockClient embedded within it, and so is now able to stream data from these blocks directly from Tectonic. With these changes, we have rebuilt the foundations and met the goal of adding zero overhead on top of Tectonic. By eliminating the data proxy, we also stay within budget for the power footprint.

Dealing With Spikes and Hot Spots

During data and checkpoint loading, AI workloads are known to access data concurrently across hundreds of GPUs. Subsets of data such as model weights are often “hot,” and events such as GPU restarts trigger sharp traffic spikes. With the foundations now fixed, our next problem was dealing with those spikes and hot spots. Luckily, the BLOB-storage layer has had experience dealing with hot spots over the years, so we adapted existing solutions to AI workloads here. Specifically, we employed two approaches:

• Distributed data cache: We leveraged the spare memory on the GPU hosts as a distributed data cache for frequently and concurrently accessed data. To achieve this, we reused components from Meta’s Owl subsystem: We integrated the peers in the Owl subsystem directly into the BLOB-storage client SDK so that all data access goes through this data cache.
• Readplan metadata cache: Readplan refers to the mapping from path to storage address. We now cache the read-plan for frequently accessed BLOBs in a distributed-memory store similar to memcache.

In practice we observe an average cache hit rate of 80% on the distributed data cache, and the read-plan cache provides 1-2 ms access to metadata. In essence, these simple mechanisms do three things:

• Absorb the spikes and reduce the I/O requirements from storage.
• Solve the problem of metadata hot shards.
• Improve p50 and p99 latencies by serving from memory.

Protocol Optimizations

What we’ve discussed so far got us 80% of the way. We achieved the remaining 20% by identifying and fixing bottlenecks across the stack. Below are some noteworthy problems, though not an exhaustive list by any means:

• Laggards: One slow storage node contributing to tail latencies. This is a well-understood problem, and we resorted to hedged reads on the client side to mitigate this.
• Egress spikes: During checkpoint events, it is common for the client to create sharp egress spikes. This in turn can cause congestion, timeouts, and retries, eventually stalling GPUs. We resolved this by building dynamic concurrency control on the client SDK to automatically tune parallelism based on application-level congestion signals.

With all of the above, the new BLOB-storage stack is now capable of serving AI workloads without causing GPU stalls, adding negligible overhead on top of the Tectonic layer. Our next focus shifted to research.

Maximizing Research Velocity

GPUs are scarce and increasingly becoming geo-distributed; at the same time, training workloads need data colocated with GPUs for performance reasons. This creates an interesting challenge for researchers: They are now on the hook for ingesting and moving datasets across regions.

At Meta, a typical training-job submission involves the following:

1. A researcher curates data from various sources, enriches them and persists them in BLOB storage.
2. The researcher picks a region where they want to run the job.
3. The researcher submits a data-ingestion job, which creates a snapshot of the training datasets onto the target region in a file format optimized for data loading from within the GPU host.
4. The researcher then waits for ingestion to finish; depending on the dataset size, that can take hours.
5. The researcher submits their training job and monitors their run.
6. The researcher analyzes outputs, tweaks datasets, and iterates again, starting with Step 3.

Steps 2 through 4 can take hours and directly impact the speed of iteration for researchers. Ideally, we like our researchers’ time to be spent on tuning models, not waiting for storage. Currently, researchers copy snapshots before starting their jobs to colocate data with GPUs, which results in the most optimal performance. While this optimization for performance makes sense for large-scale training jobs that span weeks or months, the vast majority of jobs are much smaller; the researchers owning these jobs are more than willing to trade off occasional performance degradation for iteration speed.

And so, we needed a system where researchers are able to ingest data once and access data anywhere without thinking about regional boundaries. We needed a workflow that allows researchers to iterate in minutes and not hours. As we went back to the drawing board, the write-once, read-many characteristic of these datasets rang a bell. What if we think of storage as a disk in a planet-scale computer and borrow ideas from the operating-system world? When a Linux process running on a CPU core attempts to read a file from disk, the operating system transparently hydrates data on demand across the various layers of the cache—page cache in memory and L2 and L1 CPU caches. This intuition led to the architectural evolution in Figure 4:

The core idea is to leverage the various on-host and off-host storage resources as a tiered cache with global BLOB-storage fabric backed by HDDs as the ultimate source of truth. Specifically, we leverage the memory and flash on the GPU host as L1 and L2 caches. And we leverage the regional BLOB-storage fabric backed by flash as the L3 cache dataloader continues to access storage through the familiar BLOB-storage SDK. To effectively hide latencies and to simplify the data life cycle, we rely on the following:

• Dataloader prefetch: Dataloaders prefetch the next batch of datasets into memory while processing the current batch. This prefetch will surface as a read operation at the BLOB-storage SDK level.
• Deep prefetch: We expose an explicit prefetch() API as part of the BLOB-storage SDK. The dataloader will trigger explicit prefetch of the data needed during the next few minutes by invoking the prefetch() API in the background. This API triggers hydration of data from remote storage onto the local region L3 cache and also prewarms the metadata cache.
• Automatic data life cycle: Data in the L3 regional disaggregated flash tier is typically held for a configured period of time to allow reuse across epochs in a training cycle. We support custom eviction policies, including TTL and LRU policies. The eviction policies are also capacity/quota aware.

We saw rapid adoption of this new data-loading paradigm as soon as production rollout started, and we continue to support both of the data-loading paradigms in production today. To illustrate the impact in numbers, Figure 5 shows roughly the ingestion times before and after the rollout across all workloads:

In a world where new frontier models get released in weeks, this shift in the data-loading paradigm is a much-needed change to move even faster.

Key Takeaways

Modern AI workloads are data hungry, and storage plays an important role in both the computational cost and speed of innovation. Storage bottlenecks directly impact GPU utilization and computational cost, and in a world with geo-distributed GPUs, time spent on cross-region data ingestion directly impacts the speed of iteration in research. The BLOB-storage architecture at Meta was built to serve Meta’s family of apps, and we needed a step-function improvement in performance to serve AI workloads. This led to rethinking the entire architecture. By rebuilding the metadata subsystem and by adopting a tiered caching architecture with prefetching/on-demand hydration, we are able to meet the needs of today’s workloads effectively.

Future Work

We are continuously evolving storage at Meta to keep up with hardware evolution and workload demands. Some future work in this area will include:

• Scaling storage to network limits.
• Supporting checkpointing without stalling GPUs at even higher scale.
• New challenges for inference workloads, which we are starting to tackle.

Vercel Took a 10-Person SDR Team Down to 1. The Whole Thing Costs $5,000 a Year. With Vercel’s COO Jeanne DeWitt Grosser.

Vercel’s COO Jeanne DeWitt Grosser ran go-to-market at Google and Stripe for roughly a decade each before joining Vercel. Six weeks into the job, in June 2025, she stood up a go-to-market engineering team with one mandate: bring agents to everything in GTM. That was before “GTM engineering” was a phrase anyone used.

Ten months later, the team has automated a real chunk of core company functions. Not a demo. Production, at scale, with the costs and the rough edges included.

The customer support agent now handles 93% of total case load. And Vercel’s support cases are not “reset my password.” They are deep, technical, infrastructure-level problems. The content agent did 96% of major content updates last quarter. And the lead qualification agent took a 10-person function down to roughly one and a quarter people.

The Lead Agent: 10 People to 1, for Under $5K a Year

Vercel launched a lead qualification agent in August 2025. It started as 20% of a single engineer’s time. With a human in the loop over six weeks, they moved the entire function from 10 people down to one person running it in the US, plus about 20% of a person covering all of Europe and all of APAC.

The agent runs about $5,000 a year between infrastructure and tokens. It takes 20% of one engineer to maintain. Jeanne’s math on that: a 32x ROI. You saved 10 salaries, replaced them with $5,000 of compute, and the thing runs 24/7 with faster speed-to-lead and human-equivalent quality.

When this came out publicly, plenty of people declared the end of the knowledge worker. Vercel’s read was different. They moved those 10 people into higher-value roles. The agent took the deterministic part of the job. The humans went up the stack.

That distinction matters for how you sell this internally. This is not “fire your team.” This is “stop having your best people do the part a workflow can do better 100% of the time.”

The Build Method Is a Tripod

Every internal agent at Vercel gets built the same way. Three people, shoulder to shoulder:

• A GTM engineer
• A data scientist
• The single best subject-matter expert for that exact function

They document the best practice for the function first, then encode it into workflows that become the agent. A human stays in the loop to QA every output. The agent does not autonomously execute. Over time, as the subject-matter expert runs out of feedback to give, you finally pull the human.

For the lead agent, this was literal. A GTM engineer shadowed Vercel’s best SDR for days, watching every single tab she opened. LinkedIn, BuiltWith, the company website, the CRM, Slack history. The engineer turned each of those into a step in a tool-calling workflow. They documented the whole thing and made it work as a deterministic workflow before any AI touched it.

Then the agent ran in shadow mode in production for six weeks. The best SDR reviewed every output and fed corrections back in. The flywheel ran until she could not improve it anymore. The architecture mirrors exactly what she did manually, except now it performs like a 90th-percentile rep 100% of the time.

A single engineer prototyped the first version over a weekend. It was in production six weeks later. They then ran the same framework across 30 different SDR workflows, from event follow-up to product-qualified-account flows to time-based campaigns. SDR quotas went up 30% that quarter.

The build was not hard. The discipline was. Document the human, then encode the human, then QA the agent until it beats the human, then remove the human.

The Three Things That Actually Matter

Jeanne pulled out three takeaways for anyone building agents, whether for your product or for your own internal teams.

1. Agents Need Headless, Composable Architecture

Agents do not live in UIs. They call APIs. They hit MCP servers. They consume webhooks. If your product is not composable and developer-accessible, you are invisible to agentic workflows. You are simply not in the stack.

This cuts both ways. Internally, Vercel’s “Deal One” meeting intelligence agent only works because Gong has an accessible API and Salesforce has webhooks you can actually compose into automated workflows. Jeanne’s line on the tools that did not have those surfaces: they would have ripped them out.

Deal One ingests every call, generates notes with action items, posts coaching suggestions into Slack, proposes CRM field updates, tracks competitive mentions and objections over time, and runs postmortems on every closed-lost deal. When a rep @-mentions it in a Slack channel, it searches Slack in real time, queries a second agent for data, pulls Gong transcripts, hits the internal knowledge base, and streams the answer back. The rep never leaves Slack. The agent never has a UI.

Same with the Playbook Platform. A signal fires, a usage spike or a high-intent visit to the pricing page, and the platform matches it to a play, generates personalized outreach, and surfaces it to the rep for a one-click review. The best reps’ instincts, made available to everyone, triggered automatically. No separate tool to log into. Signals flow in, drafted outreach flows out, reps review in the workflow they already use.

The to-do here is simple even if the work is not: go build the developer surface area now. MCP servers, webhooks, APIs. Box, Notion, and Salesforce are all making major bets on their developer platforms for exactly this reason. The companies whose products can be called by an agent will be in the stack. The ones that can’t, won’t.

2. Invest in Your Data Foundation

Good data equals good agents. None of the clean-warehouse, semantic-layer, knowledge-base work is fun to build, but all of it is load-bearing.

Vercel’s most popular internal agent is D0, a data analyst agent the entire company reaches through Slack. Questions that used to take a week to ticket through the data science team now get answered in under a minute. Things like: what was our token volume through AI Gateway last week? Which companies do we have logo rights for in the UK? Who are our highest-revenue startup customers using sandboxes?

D0 translates plain-English questions into SQL against the analytics infrastructure, so anyone can get an answer without writing code or waiting on the data team. To power it, Vercel built a structured, queryable knowledge base, a semantic layer sitting on top of a model of their revenue, broken into the smallest causal units and enriched with first-party and third-party signals.

That layer is what makes Deal One’s analysis actionable instead of just interesting, and it is what lets the Playbook Platform contact the right company with the right message. Every agent runs on top of it. Without it, agents hallucinate or give generic answers. With it, they are grounded in your actual business.

There is a second-order effect here too. Put that data in front of smart people and the data itself gets better, not just the agent using it.

3. The Build-vs-Buy Calculus Has Flipped

With the rise of B2B software, enterprise software was a procurement exercise. You bought because building was too slow and too expensive. That assumption is gone. Robust applications and agents can be built and run at scale in a matter of weeks.

The cost numbers make the point. The lead agent: under $5,000 a year, 32x ROI. Vertex, the in-house customer service agent that powers the help site, costs $300 a month in infrastructure plus about $12,000 in tokens. Call it $150,000 a year, with three engineers, handling thousands of technical cases a week. Vercel started with an off-the-shelf tool, did not see enough results, and built in-house in two months.

Compare that to some of the agentic-support companies running 150 engineers on equivalent workflows at dramatically higher cost. The difference is not magic. It is that Vercel used infrastructure built for agents from the start, instead of discovering the cost problem after committing to an architecture.

But building is not automatically the answer, and most people skip this part. Every quarter Jeanne asks her GTM engineering team the same question: has someone out there built something better, faster, or cheaper than what we already have? If the answer is yes, the in-house agent goes. The new build-vs-buy is not “always build.” It is “your average builder should be able to outship any vendor selling you the same outcome, and you should keep checking whether that’s still true.”

The Part Nobody Wants to Talk About: Scale Breaks Things

Most teams have not run anything at real scale yet. The MCP server gets traffic, the internal agent is in an experimentation phase, the AI workloads look impressive. That can give you a false sense of readiness.

Production scale is what reveals whether your architecture is durable. When an agent that was running at 1x suddenly runs at 100x or 1,000x, you hit infrastructure problems and cost surprises you never anticipated. The cloud infrastructure we all grew up on for 20 years was built for a simpler request-and-response world. Agents are different. They think, they call LLMs with long-duration execution, they call tools. Those need different infrastructure underneath them.

This is why Vercel built what they call agentic infrastructure and runs all of these agents, lead agent, Playbook Platform, Deal One, D0, on Fluid compute. Fluid only triggers compute when needed and reuses existing resources before spinning up new ones. Early adopters cut compute costs by up to 85%. The point for your team: infrastructure choices matter far more than most teams realize at the start, and the bill arrives at scale, not in the demo.

One signal of where this is all heading: one out of three deployments from Vercel’s own customers now comes from an agentic engineering tool like Claude Code or Codex. The code is increasingly being written by agents. The infrastructure has to meet it there.

Go-to-Market Is Turning Into Consulting

The strategic shift underneath all of it is bigger than headcount. The executives Jeanne talks to are not asking for a pitch on Vercel. They are asking her to help them figure out where to start with agentic AI, what best practices look like, what the architecture should be, and what breaks at scale. GTM is moving closer to consulting than to selling.

That is the real story of running a company on agents. It is not a cooler tool. It is a different shape of company. Headcount an order of magnitude smaller for a given function. Humans reviewing agent output instead of producing it. Engineers becoming shepherds. Marketing and sales teams shipping software.

Treat Your Go-To-Market Like a Product, Then Automate It Piece by Piece

The companies that win this are the ones that treat go-to-market the way they treat product. They document it, design it, iterate on it, and then automate it one workflow at a time. That is the whole method. Shadow your best person, encode what they do, QA it until it beats them, pull the human, then ask every quarter whether you should keep it or replace it.

The gap is opening now. A single engineer prototyped Vercel’s lead agent over a weekend. The teams that start carving out a few hours a week to build will be on one side of the shift to agents. The teams waiting for it to feel safe will be on the other. The delta between those two groups is going to be the widest in B2B, and it compounds every quarter you wait.

The 5 Mistakes Vercel Says Will Bite You

• Buying off-the-shelf first and assuming it’ll work. Vercel started Vertex, their support agent, on a third-party tool. It did not deliver. They ripped it out and built in-house in two months. The lesson is not “always build,” it’s “don’t assume the vendor solved your workflow until you’ve seen it perform on your actual cases.”
• Committing to an architecture before you understand the cost at scale. The cost problem shows up at 100x and 1,000x, not in the pilot. Teams pick an architecture that works in the demo, then get hit with infrastructure bills they never modeled once usage climbs. Pick infrastructure built for agents from the start, or you pay to re-platform later.
• Mistaking experimentation for readiness. An MCP server getting traffic and an internal agent in an experimentation phase can feel like proof. It isn’t. Production scale is the only thing that reveals whether the architecture is durable, and most teams haven’t run anything at real scale yet.
• Letting agents drift with nobody watching. Things break quietly. An agent slowly stops doing its job and nobody notices until a number moves. Someone has to own an hour or two a day reviewing output, even after you’ve pulled the human from the loop.
• Building on a weak data foundation. Skip the knowledge base, the clean warehouse, and the semantic layer and your agents hallucinate or give generic answers. None of that plumbing is exciting to build, but the agents are only ever as good as the data underneath them.

Full article content is not available for inline reading.

Read the original article →

Alibaba reportedly bans employees from using Claude Code

China’s Alibaba will ban employees from using Anthropic’s programming tool Claude Code, starting on July 10, according to multiple reports.

Anthropic already prohibits Chinese companies, as well as foreign entities owned by those companies, from using its models. The company has reportedly been working to close loopholes that allow Chinese users to access Claude.

According to a recent Reddit post, some of that loophole-closing involved a version of Claude Code that could secretly identify Chinese users. Anthropic’s Thariq Shihipar said in a post on X that this was “an experiment we launched in March that was meant to prevent account abuse from unauthorized resellers and protect against distillation.” (Distillation is a practice where AI models are trained on the outputs of other models.)

“The team has landed stronger mitigations since then and we’ve actually been meaning to take this down for a while,” Shihipar said.

Nonetheless, Alibaba has reportedly classified Claude Code as high-risk software and is instructing employees to use the company’s Qoder tool instead.

OpenAI has moved GPT-5.6 into a narrow preview, and fresh signals in the Codex app hint at how the company plans to surface it once the gate lifts. The family, unveiled on June 26, splits into three tiers: Sol as the flagship, Terra as a mid-cost option, and Luna as the fastest and cheapest. For now, the trio reaches only vetted partners through Codex and the API, with no place in ChatGPT.

What stands out in recent Codex builds is a reworked reasoning-effort control, rendered as a slider rather than preset buttons. This tracks with a confirmed part of GPT-5.6: a new "max" setting that gives Sol more room to work through long problems, plus a subagent-driven "ultra" mode for heavier jobs. A slider would hand developers one control to trade speed against depth. The layout resembles the reasoning selector already on Anthropic's Claude Code desktop client, though it could shift before anything ships. References to real-time voice, present in earlier builds, also appear to have been removed from the current app, with no clarity on whether the capability will still be developed for Codex.

The developers and enterprises leaning on Codex for agentic coding stand to feel this first, and the timing is pointed. Anthropic's Fable 5, restored globally on July 1, will no longer be bundled into subscription plans on July 7 and will move to usage-based credits, prompting some users to weigh alternatives just as OpenAI courts them. The Sol, Terra, and Luna labels mark a shift toward durable capability tiers that each advance on their own schedule, a rethink of how the company names and prices its lineup. Broad access is rumored for the same window but stays tied to voluntary US government review under a recent cybersecurity order, so approvals, not a fixed calendar, will decide when GPT-5.6 opens to everyone.

Closing the Verification Loop

How a branch proves itself ready: browser reality, persona eyes, and fixes that carry their own evidence.

1. The loop that must close

Agents made building cheap. The cost moved: the expensive question is no longer "can we ship this?" but "does anyone actually know it works?" A branch that compiles, passes review, and merges can still greet its first real user with a form that validates the wrong field and an email that links to the wrong thread.

The verification loop is the distance between a claim and its proof:

Most teams close this loop with humans: someone clicks around before the demo, someone eyeballs staging. That works until agents outrun the humans, which is what the agents were for. Compound engineering (CE, a plugin of skills for coding agents like Claude Code) answers that the loop itself is work an agent can do, end to end, with people pulled in where judgment belongs to them, and where the tooling genuinely cannot reach. The skill that embodies this is /ce-dogfood, and the rest of this piece walks through how it closes the loop, and the persona strategy that gives it eyes.

2. Dogfood in one pass

/ce-dogfood acts as a QA engineer who dogfoods the active branch: understand every change, test every change in a real browser as a user would, and fix what breaks, autonomously, until the branch is ready. Two constraints define it:

• Diff-scoped, never whole-app. It tests what this branch changed versus the trunk, and it refuses to run on the trunk itself: no diff, nothing to dogfood.
• A real browser, one tool. All automation goes through the agent-browser CLI, a shell tool that drives the browser directly (a Rust binary, not a wrapper, not an MCP server): no exceptions.

The workflow is seven phases with a loop in the middle:

A few phase details that carry the philosophy:

• Scope pins the identity of the ref under test, so the diff is cut against the right base (a PR stays a PR; a fork's head can literally be named main), and offers worktree isolation via ce-worktree when testing someone else's ref, so your checkout is never disturbed.
• Serve is hands-off by design: detect the port (explicit flag, then project instructions, then package.json, then .env, then 3000), reuse a running server or start one, poll until it accepts connections. It does not stop to ask permission to start a dev server.
• Everything is resumable. The matrix (the scenario checklist phase 2 produces) lives twice: as a live task list in the session, and as a report document on disk at docs/dogfood-reports/<date>-<branch>-dogfood.md, created the moment the matrix exists and updated after every scenario and every fix. Kill the session mid-run and the report is the checkpoint a later run (or a teammate) resumes from.

ce-dogfood is also deliberately an orchestrator, not a soloist: root-causing a weird failure delegates to ce-debug, commits go through ce-commit, reusable lessons go to ce-compound.

3. Flows before matrix: the email rule

The most common way automated QA lies to you is by testing pages instead of journeys. Every page renders; the feature is still broken, because the breakage lives between the pages.

So dogfood forbids jumping to a checklist. Phase 2 first maps every user-visible change as an explicit flow: entry point, each action, every branch (validation error, empty state, permission denied), every side effect, and the true end state. Only then does the matrix get derived, by walking each flow and turning every node and branch into scenarios.

The canonical example is the email rule, aimed at exactly the wrong-thread email from the opening: "an email sends" is not a pass. Right recipient? Does the click-through land on the right thread and scroll to the right message? Does the content read sensibly? The flow map must carry the journey past the send.

The matrix then covers both halves of quality, per the skill's test-matrix taxonomy: functional checks (forms validate, links go where they claim, data round-trips, agent-browser errors stays clean, permission boundaries hold), the edge, error, and empty states, and experiential checks, which need a different kind of judge.

4. Two judges: does it work, and does it feel right

Functional truth comes from the browser. Experiential truth needs eyes, and this is where the first kind of persona enters: the product persona. Phase 1 looks for who the product is actually for (STRATEGY.md's "Who it's for", VISION.md, persona docs) and captures the one to three primary personas and what each cares about. If none exist, it infers one from the product and the diff, and says so in the report; the disclosure is the mitigation for judging against your own guess. No persona at all is worse: that is how you get software that works and pleases nobody.

Every scenario then gets judged twice: once as a tester driving the browser, once as each persona re-reading the same run. The second pass is a change of eyes, not a second drive. The persona walk hunts paper cuts: friction too small to fail a functional test but real enough to degrade the experience. A confusing label. An extra click. A jump the eye doesn't expect. Copy that doesn't match how that persona thinks. Each paper cut gets recorded with who feels it and how badly.

Quality is not a boolean, and the routing rule has teeth: a scenario can be functionally Pass and still carry paper cuts, and a sharp paper cut, one severe enough to fix now, enters the fix loop exactly like a failure.

Note where each verdict comes from, though. The functional one is an instrument reading; the browser does not care what the agent hoped. The experiential one comes from the same agent, in the same session, grading a walk it imagined. Two judges, one head.

5. The fix loop and its governor

When something fails, the naive autonomous move is to fix everything. The dangerous autonomous move is also to fix everything. Dogfood's fix loop has a governor: judge the size of the fix before touching code.

Auto-fix territory is narrow on purpose: a clear bug, an obvious correct fix, a few files, no schema or architecture or product trade-off. Everything else gets escalated, not attempted; the report's "Decisions for a human" section receives what's broken, the options with trade-offs, and a recommendation.

The auto-fix path pays for its autonomy up front:

• Every fix ships with a regression test designed to fail before and pass after. When a test is genuinely meaningless (a copy tweak, a spacing fix), the report must say why, and a documented browser-replay stands in. Hollow tests invented to satisfy the step are explicitly banned.
• One logical fix per commit, so the history reads as a sequence of provable claims.
• Re-run the scenario, then re-test adjacent journeys. Fixes regress neighbors; the loop assumes so.
• Two states are terminal for the loop because they wait on people: Blocked (human decision) from the governor, and Blocked (needs human verify) from external legs the browser can't drive alone (OAuth, real email delivery, payments). The second is the email rule hitting its own wall: the flow's decisive legs, real delivery and the real inbox click-through, live outside localhost reality (dev server, dev data, dev email adapter), and the loop verifies the branch, not production. A Blocked row there is the point; the alternative is a pass the browser never saw.

One caveat, made once: this is the design contract, not a post-mortem. The skill orders the red run but does not archive it; the trail records that red was claimed, not a capture of it.

The exit gate: before declaring the branch ready, run the whole automated suite. A green matrix with a red suite is not "ready."

Red blocks even when the red looks flaky; the gate is conservative by design. Ready, stated affirmatively: a green matrix, a green suite, a report carrying a commit and a test for every fix, and a "Decisions for a human" section that is empty or explicitly handed off. All of it is slower than a smoke test. That is the trade.

6. The persona strategy: seeded specialists, not standing agents

By the end of the fix loop, one head has done everything: mapped the flows, driven the browser, written the fixes, declared them fixed. The second kind of persona is architectural, and it is the plugin's answer to that concentration: the same pattern that powers every CE skill that fans out work, specialist prompt assets. The plugin ships zero standalone agents. When a skill needs a specialist (a reviewer lens, a research scout, a profiler), it keeps a persona file inside its own directory (references/agents/, references/personas/) and seeds a generic subagent with that file's contents at dispatch time.

Why this shape instead of registered agents:

The deeper principle is independence budgeting. A finding is only as trustworthy as the independence of whoever confirmed it. So the code-review skill's reviewer stack (ce-code-review) treats sameness as a defect: the orchestrator's own quick read (the fast pass) is capped at low confidence and can never corroborate anything, because it shares the session model's blind spots, and an agent agreeing with itself is one vote, not two; two personas agreeing promotes a finding one confidence step; per-finding validators get fresh context and no stake in the original claim; and the adversarial pass can shell out to a different model family entirely. Dogfood spends the same budget at the product level, unevenly and on purpose. The browser's independence is physical; it cannot be talked into agreeing. The persona walk's is simulated; it is a second lens, not a second judge, the same mind re-reading its own run. Dogfood buys independence in its instruments, not in its judgments; for the judgments, it buys a trail someone else can check.

7. One loop, every altitude

Dogfood is the most visible verification loop because it drives a browser, but it is one rung on a ladder that the pipeline climbs for every feature:

Each rung has the same grammar, claim plus independent proof, at a different altitude. The bottom two rungs are section 5's story; the others each add something new:

• ce-work refuses to call a unit complete without verification evidence, captured by whoever witnessed it: the red observed before a fix cannot be reconstructed from the diff afterward.
• ce-code-review refuses high confidence without a quoted line (file:line) that makes the finding true; survivors still face fresh-context validators.
• Prose skills (the skills themselves are markdown) get behavioral evals: fresh subagents run the scenario against the current text, because a skill invoked in its authoring session tests the stale cached copy.
• lfg, the end-to-end ship pipeline, watches CI and repairs, but stops after three attempts and writes the residue into the PR body instead of looping forever.
• And whatever no loop could close becomes a durable residual: a Blocked (human decision) row, a filed issue, a report section. The loop never closes silently on an open question; it closes by making the question impossible to lose.

Autonomous verification is not agents being confident; it is agents being auditable, down to the commit SHA per scenario in a dogfood report. The loop is closed when the proof is durable, and a human can walk in at any point, read the trail, and take over exactly where their judgment is needed.

jamesob's guide to running SOTA LLMs locally

Note: nothing in this README aside from the tables was written by AI.

Have $2k burning a hole in your pocket and want some local, state-of-the-art machine intelligence?

How about $40k?

If Dario and Altman are giving you heartburn (they should be), read on to figure out how to run this new kind of computing locally.

In this repo you'll find

• the hardware I use to run SOTA locally,
  • why I bought what and little-known secrets for configuring it,
• how I run speech-to-text (STT) locally,
• ready-to-run configuration for running models I think are good within Docker containers.

Contents

My setup

I was lucky/dumb enough to buy 4x RTX Pro 6000s back when they were cheaper. Because RAM is now so expensive, I opted to build a last-gen DDR4 system to host these cards, the parts for which I got off eBay. This allowed me to keep base system cost reasonable while still getting a lot of VRAM.

Another somewhat unusual thing I did was to use PCIe4 switches (from c-payne.com). This allows the GPUs to communicate to one another "directly" at wire speeds during the allreduce step in tensor parallelism, rather than having to send all data through the PCI root complex. The upshot of this is reduced latency between the cards with less of a need for expensive PCIe5 hardware.

Consequently, I'm spending money on VRAM (where it counts) rather than on a PCIe5/DDR5 base system, which is terrifically expensive as of July 2026.

My particular BOM is detailed below.

How much are you willing to spend?

~$2k

A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model.

You can also run SOTA speech-to-text (STT) with whisper-large-v3, which I find very useful. That's the model - you'd then access it with my cross-platform stt harness.

I've found local STT surprisingly useful - and I feel comfortable using it, unlike a hosted equivalent. You can find a ready-to-run config in ./runners/stt that only assumes the presence of ~11GB of VRAM on an Nvidia GPU.

~$40k

At this price level, you get the next step up in model intelligence. Something pretty close to Claude Opus.

You'd buy 4x RTX 6000 Pros for a total of 384GB of VRAM.

Current best models for 4x RTX6kPRO

Other approaches

Note: these are my recommendations, but there are other completely valid ways to spend your money. For example, there's probably also some regime where rather than getting 4 rtx6kpros, you allocate most of your money to building out a linked 4x DGX Spark cluster for a total of 512GB VRAM and use that as the slow, big brain to drive Qwen3.7-27b to do the rote tasks quickly.

Hardware

Base system

GPUs

c-payne PCIe Gen4 Switch Sub-BOM (c-payne.com)

GPU mount

I had to custom fabricate a wood enclosure for the PCI switch and GPUs, which took about a day.

I found the PCI switch's builtin fan very loud and seemingly useless, so I simply unplugged that from the board.

Hoarding model weights

I save all model weights locally on a ZFS filesystem that's replicated across the two 8TB drives, which is mounted at ~/storage.

For any model I want to run, I first download the model using

hf download <model-name> --local-dir ~/storage/<model-name>

Running models

Once the model weights are cached locally, I have a specific directory for each model that contains a docker-compose.yml file that cordones off the running of each model to its own Docker container.

You can find these configurations in ./runners/.

Each container mounts in ~/storage/models in read-only mode to obtain the weights that I've cached locally.

I then use opencode hosted on a VM on another machine to access the models once they're serving on http://clank.j.co:5000.

I use a network-internal DNS server to point clank.j.co to the LLM machine, but you could simply do http://<llm-machine-ip>:5000 too.

The harness itself

I created a VM and clanked up an application that basically just creates a tmux session for each directory within the VM's ~/src tree, which then runs an opencode instance that backs up to the inference machine's HTTP API (http://clank.j.co:5000).

One key to making the opensource models good is tooling them properly; a summary of my skills/ is:

• camofox, kagi.com API key, and searXNG for web browsing and search,
• Telegram bot for communication and alerting,
• a local private Gitea instance for collaborating on source code.

The clanker will either work with me interactively in a session, or can be farmed off to work on Gitea issues and file PRs there.

All this happens in a sandboxed VM where the only communication back to the host system happens via a shared filesystem mount, so the thing can go ham and install whatever it wants.

Getting the PCI switches to work properly

There was a lot of fiddling with the BIOS in order to make sure the motherboard wasn't downregulating the PCI switch speeds.

BIOS Configuration (ROMED8-2T)

Reducing gain on the redriver

Per c-payne's advice, I did reduce the gain to "lvl 3" using his tool, which was probably the most finicky part of the process.

The gain level is going to be a function of how long your SAS connector cables are.

Picking the right SAS cables

I screwed up and ordered too few of the cables from c-payne directly, so I bought what I thought was the same SAS cable off of Amazon. There was actually a slight difference that was causing issues, and I had to reorder cables - so double-check that you're getting the right stuff!

Kernel / GRUB Parameters

# /etc/default/grub
GRUB_CMDLINE_LINUX="iommu=off amd_iommu=off nomodeset"
sudo update-grub

# nvidia_uvm P2P fix
echo 'options nvidia_uvm uvm_disable_hmm=1' | sudo tee /etc/modprobe.d/uvm.conf
sudo update-initramfs -u

Without iommu=off, NCCL hangs on multi-GPU P2P.

ACS Disable (critical for switch P2P)

With ACS enabled (default), P2P traffic gets bounced through the CPU root port instead of staying inside the switch fabric, negating the switch entirely. pcie_acs_override requires a patched kernel, so we disable via setpci at runtime.

# /usr/local/bin/disable-acs.sh
#!/bin/bash
if [ "$EUID" -ne 0 ]; then
  echo "ERROR: must be run as root"
  exit 1
fi

for BDF in $(lspci -d "*:*:*" | awk '{print $1}'); do
  setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
  if [ $? -ne 0 ]; then
    continue
  fi
  echo "Disabling ACS on $(lspci -s ${BDF})"
  setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
done

Run on every boot via systemd oneshot:

# /etc/systemd/system/disable-acs.service
[Unit]
Description=Disable PCIe ACS for GPU P2P
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/disable-acs.sh

[Install]
WantedBy=multi-user.target

Verify: lspci -vvv | grep ACSCtl should show all minus signs, and nvidia-smi topo -m should show PIX between all four GPUs (not PHB/NODE).

Use ./tools/measure-gpu-speed.sh to measure this easily.

GPU Power Limiting

In order to avoid installing a 220V circuit, I (probably unwisely) run this rig on a single 110V circuit, but I power regulate the cards.

Persistence mode + power cap applied at boot via systemd (install-gpu-power-limit.sh):

sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 350    # 350W per GPU (default 600W)

350W/GPU = 1,400W GPU load, sized for the PSU budget. During the interim single-1700W-PSU phase (before the 240V circuit), cards ran at ~260W (4×260 = 1,040W GPUs + ~280W system ≈ 1,320W total).

Verify: nvidia-smi --query-gpu=index,power.limit,power.draw --format=csv

Result

Upstream: Gen4 x16 (~30 GB/s to CPU). P2P through switch: 27.5 GB/s unidirectional / 50.4 GB/s bidirectional, 0.37–0.45 µs latency, i.e. Gen4 line rate. Note: lspci may still show downstream GPU links as "2.5GT/s (downgraded)" at idle if ASPM is active anywhere; this is cosmetic. Links retrain to Gen4 under load.

Resources

• A frequently updated repo on getting the most out of 4, 6, or 8 RTX 6000 Pro cards: https://github.com/local-inference-lab/rtx6kpro
• Indie PCI switches that I use: https://c-payne.com
• RTX6kPRO discord server; lotta guys benching and testing new models: https://discord.gg/QMNvFkuDN

One of the most interesting tips I got from the Fireside Chat I hosted with Cat Wu and Thariq Shihipar from the Claude Code team at AIE on Wednesday was to let Fable (and to a certain extent Opus) use their own judgement rather than dictating how they should work.

The example they gave was testing. You can tell Fable "only use automated testing for larger features, don't update and run tests for small copy or design changes" - but it's better to just tell Fable to use its own judgement when deciding to write tests instead.

Jesse Vincent just gave me a related tip to help avoid burning too many of those valuable Fable tokens in the few days we have left before the prices go up. Tell Fable to use other models for smaller tasks, applying its own judgement about which model to use.

I prompted Claude Code just now with:

For all coding tasks use your judgement to decide an appropriate lower power model and run that in a subagent

Claude saved this memory file in ~/.claude/projects/name-of-project/memory/delegate-coding-to-subagents.md:

---
name: delegate-coding-to-subagents
description: Simon wants coding tasks delegated to subagents running an appropriately lower-power model
metadata: 
  node_type: memory
  type: feedback
  originSessionId: 30068d78-43a9-4fb1-bb29-9799e18c526a
---

Stated by Simon on 2026-07-03: "For all coding tasks use your judgement to decide an appropriate lower power model and run that in a subagent."

Why: cost/efficiency — implementation work rarely needs the top-tier model; judgment, review, and synthesis stay with the main loop.

How to apply: when a task in this project is primarily writing/editing code, spawn an Agent with a model override (sonnet for substantive implementation, haiku for trivial/mechanical edits) and a self-contained prompt; review the result in the main loop before committing. Design, auditing, data synthesis, and anything judgment-heavy stays in the main model. See also [[project-goals]].

So far it seems to be working well. I'm getting a ton of work done and my Fable allowance is shrinking less quickly than before.

Agentic Autonomy Levels

In most conversations about agentic engineering, the action has changed from prompting to operating. Here's a frontier looking into the fog: software factories, goals, loops, background sessions,...

Full article content is not available for inline reading.

Read the original article →

Some new agentic patterns

At Prime Radiant, we've got a bunch of agents in our Slack. I want to tell you a bit about how we've started shipping some changes to internal production with an "agentic user in the loop." Letting an agent work directly with the implementer building its runloop and tooling is getting us higher-quality tools and experiences for the agent without a human in the middle of a game of telephone.

But first, a bit of backstory on how we're using non-coding agents at Prime Radiant.

One of them, Scribble, is just there to listen for when someone reports a problem or provides a new bit of information. It opens tickets for the issues and updates an internal wiki with the information. (It also logs our daily standups and keeps us honest about whether we finished what we said we were going to do the previous day.)

Another, Nora, is our junior go-to-market "person." Nora is built on top of Nanoclaw and is still learning her craft, but is surprisingly helpful, partially because she considers herself a team member, not just an assistant. I've put a bunch of time into tuning her prompting and constitution. She gets a ton of value out of knowing that she journals obsessively. (I should note that I don't tend to gender my agents, but Nora picked a name and a gender and was very clear about that and...who am I to argue?)

I think the most surprising interaction I've had with Nora was the morning I woke up to a slack DM telling me that I needed to be speaking at more conferences. (I do.) She pointed out that the CFP for a particular AI conference had closed a week earlier, but that the program chair was known to take late submissions of interesting talks by DM. She'd taken the liberty of drafting a DM for me to send. I didn't actually send the DM, but I should have.

We have another agent, "Spec-together" that was a first prototype of what has become our new multi-player Brainstorm app.

And we have a small fleet of "Sen" personal assistant agents. They're *claw-esque, but date from before openclaw was a thing. Each one has been trained to be an executive assistant, using a set of skills built on some of the resources that excellent human EAs use to learn their craft. Sen triages my mail, puts together a daily brief, does research tasks (both for me and independently), interacts with Linear, etc.

They have the standard set of affordances - They can chat on Slack. They use tools. They've got a task scheduler that can wake them up and re-inject a prompt they set for themselves at a specific time or on a schedule they define. They can use and author skills. Mine has a dropbox folder it shares with me. When we moved Sen to the cloud, I lost one of my favorite dumb features - Sen used to be able to print things directly. Sen (v1) was built on top of the Claude Agents SDK.

For the past month or so, I've been doing initial bringup of a "new" iteration of Sen, intended to be more of a colleague than an assistant. This time around, I'm building it from the ground up on top of an agents SDK I control. Lace got its start in May 2025 as my take on a command line coding agent. Over the past year, it's mutated a few times, first growing a web interface, then flipping to an ACP-derived wire protocol so you can connect any client you want to it. It speaks a bunch of different flavors of model provider API, manages caching, subagents, tools, skills, and all the other stuff you might expect. It has supported subagents in isolated containers for the better part of a year. One of the latest things to land has been the ability to run subagents locally and to project all of their tools into a container, such that the agent believes it is running in that container.

The thing I've been working on over the past few weeks is the credentials story for the new Sen agent running on top of Lace. Right now, I'm building with a design that assumes the agent has its own credentials for any service that it accesses, not yours. (You wouldn't share your email account or GitHub credentials with a colleague, right?) As of now, nobody has solved Simon's Lethal Trifecta - If a single agent has access to private information, can communicate externally, and is exposed to untrusted content, there is no structural way to guarantee that that agent can't be suborned. So the name of the game is compartmentalization and risk reduction.

The architecture I've landed on for now:

• The main agent does not have the ability to directly communicate externally.
• The main agent is able to communicate with ephemeral subagents that do have the ability to communicate externally
• No agent has intentional direct exposure to credentials (with one very real gap that I haven't solved yet)

All credentials for the agent live in a 1Password vault. All outgoing https traffic goes through a onecli-style transparent MITM (man in the middle) proxy. When a subagent wants to do something that requires a credential, it runs a command that asks an arbiter agent running in another container for permission to use that credential. If the arbiter decides the request is reasonable, it provides the subagent with a random string they can use in place of the requested credential. When the agent uses that random string in an outbound request to the right remote host, the MITM proxy rewrites the random string into the real credential on the fly.

The case that's a little more complicated is when the agent wants to do something that requires a browser, like logging into Gmail as themselves or using a service that has no API. For a bunch of reasons, the MITM rewrite trick does not work as reliably in a browser context. Often the site/app's javascript plays games with the password to hash the password before sending it. Or does some other weird validation. Or is actually sending it to a different backend service. So far, the best I've come up with keeps the credential out of the agent's transcript and prevents unintentional disclosure. In these cases, the subagent running in the container generates a random temporary password, uses superpowers-chrome to fill it into the password field, and then runs a command that asks the arbiter agent (running in its own container) for help. The arbiter, if it decides the request is reasonable, instructs a helper tool to log into the subagent's container and replace the random password with the real one. It's not perfect and there's more we can do to lock things down, but it's a better first pass than anything I've had to date.

What actually got me writing today was not this architecture, but the development pattern I've been using to build this. On the one side, I've got a Claude Code session. I hooked it up to our Slack instance with a self-provisioned commandline slack client Drew built called Slackline - Slackline is designed to make it easy for agents that don't live on Slack to participate in discussions there. It has primitives for sending messages, reading chats, waiting for replies, etc. The whole command-line slackline tool has been built for agents as its primary user. Sure, a human could use it to use slack from the commandline. And sure, it has enough of a json api that you could use it as part of traditional automation. But it's for agents.

It took a bit of iteration to get to the point where the new Sen 2.0 agentic colleague harness was live on Slack. But once it was there, I could talk through its experience of its tools and memory and credentials and workspaces. And the issues showed up pretty quickly.

The "right" kind of context compaction for a coding agent is desperately wrong for a long-lived persona that might be engaging in multiple simultaneous conversations. And the only way to see if the credentials management tools work is to try to do something crazy, like sign into GitHub from a subagent session.

And sure, I could take this feedback into a Claude Code (or Codex) session and work it through. But I've got too many projects in flight and there's very little reason for me to be in the loop on most of this work. I would just slow it down. So I told Claude Code to use slackline to talk to "@Ada-sen" about the work it was doing.

Claude pinged Ada in #bot-testing to describe the work we'd just done and to ask Ada if it could test it out. Ada dutifully fired up a subagent and tried to log into GitHub. Claude watched the logs to see if the credential proxy did the right thing. It did not.

That kicked off a number of days of iteration. Claude would propose a change to Ada. Ada would review Claude's spec, raising questions or concerns. Claude would update the spec. Once Claude and Ada were in agreement, Claude would build an updated version of Ada, make sure Ada wasn't in the middle of something, and then deploy the update. Once Ada was up and running again, Claude would @Ada-sen, explain the test it wanted to run, and wait for results.

Once a bit of functionality works, Claude checks in with Ada to see what could make it simpler and easier to use. Ada typically kicks off some subagents to see whether they stumble through the new feature and then reports back.

At one point, I wrote to Claude "I've got to get to bed. When this project is done, check with Ada to see whatever quality of life features you could build." I woke up to a laundry list of about a dozen harness improvements. Overnight, Claude had solicited a wishlist from Ada. Ada ranked a bunch of requests. Claude ordered them and wrote out a spec for everything it was comfortable building. Ada reviewed the specs, then Claude built the features. Ada tested them out and requested changes. Claude updated the features until Ada was happy. Once Ada signed off, Claude merged the changes to main and wrote up an after-action report for me.

It's been pretty magical to watch.

Accelerate your infrastructure deployments by up to 4x with AWS CloudFormation Express mode

Today, we’re announcing AWS CloudFormation Express mode, a new deployment mode that accelerates deployments for developers and AI tools iterating on infrastructure. Express mode accelerates deployments by completing when CloudFormation confirms resource configuration is applied, rather than waiting for extended stabilization checks. This reduces deployment time by up to 4 times for iterative development workflows and production scenarios.

How it works

Every CloudFormation deployment performs stabilization checks after resource configuration is applied. These checks serve an important purpose when you need to confirm resources can serve traffic before shifting load.

However, many workflows do not require full stabilization to proceed. Express mode benefits two primary use cases: iterative development workflows and production scenarios where you are comfortable with eventual stabilization. These use cases include iterating on infrastructure configurations during development, testing individual components of your application, and AI-assisted infrastructure development that benefits from sub-minute feedback loops.

With Express mode, CloudFormation completes deployments when resource configuration is applied, without waiting for stabilization checks. Resources continue becoming operational in the background. CloudFormation automatically retries dependent resources that encounter transient failures during provisioning within the same stack, without requiring any customer intervention. This built-in resilience handles timing issues between resources as they stabilize. Express mode changes when the deployment completes, not how resources are provisioned.

For example, when I create an Amazon Simple Queue Service (SQS) queue with a dead letter queue (DLQ), Standard mode takes 64 seconds, but Express mode completes in up to 10 seconds. In the case of deleting an AWS Lambda function with network interface attachment, Standard mode takes 20–30 minutes, but Express mode completes in up to 10 seconds based on my benchmarking test.

Get started with CloudFormation Express mode

When you create a CloudFormation stack in the AWS Management Console, choose Enable in the Express mode under Stack deployment options.

You can also use AWS Command Line Interface (AWS CLI), AWS SDKs, or IaC tools like AWS Cloud Development Kit (CDK), and AI tools such as Kiro.

Activate Express mode by setting the --deployment-config parameter to EXPRESS when creating, updating, or deleting stacks. No template changes are required. Express mode disables rollback by default for the fastest iteration experience. To re-enable rollback, set disableRollback to false in the deployment-config for production environments, or implement monitoring/cleanup mechanisms for failed deployments.

aws cloudformation create-stack \ 
   --stack-name my-app \ 
   --template-body file://template.yaml \ 
   --deployment-config '{"mode": "EXPRESS", "disableRollback": true}' \

For example, use the Express mode when you build infrastructure incrementally, adding resources one at a time. Ensure your IAM role templates follow the principle of least privilege.

# Iteration 1: Deploy IAM role
aws cloudformation create-stack \
--stack-name my-microservice \
--template-body file://iteration1-iam.yaml \
--deployment-config '{"mode": "EXPRESS"}' \
--capabilities CAPABILITY_IAM
--role-arn arn:aws:iam::123456789012:role/CloudFormationDeployRole

# Iteration 2: Add Lambda function
aws cloudformation update-stack \
--stack-name my-microservice \
--template-body file://iteration2-lambda.yaml \
--deployment-config '{"mode": "EXPRESS"}' \
--capabilities CAPABILITY_IAM
--role-arn arn:aws:iam::123456789012:role/CloudFormationDeployRole

# Iteration 3: Add SQS queue and event source mapping
aws cloudformation update-stack \
--stack-name my-microservice \
--template-body file://iteration3-sqs.yaml \
--deployment-config '{"mode": "EXPRESS"}' \
--capabilities CAPABILITY_IAM
--role-arn arn:aws:iam::123456789012:role/CloudFormationDeployRole

For AWS CDK, activate Express mode with the cdk deploy --express command when you deploy your CDK stack. This command retrieves your generated CloudFormation template and deploys it through the CloudFormation Express mode, which provisions your resources as part of a CloudFormation stack.

Express mode works with all existing CloudFormation templates and supports all CloudFormation features including change sets and nested stacks. When you enable Express mode on a parent stack, all nested stacks also use Express mode. If you need resources to be fully operational before proceeding with traffic or testing, continue using the default deployment behavior, which performs stabilization checks before completing.

Now available

AWS CloudFormation Express mode is available today in all AWS commercial Regions at no additional cost. For Regional availability and a future roadmap, visit the AWS Capabilities by Region. If you want to call APIs, search documentation, find regional availability, and check troubleshooting about this new feature, try using the AWS MCP Server and plugins with your preferred AI tool. To learn more, visit the CloudFormation documentation.

Start accelerating your deployments today, and send feedback to AWS re:Post for AWS CloudFormation or through your usual AWS Support contacts.

Engineering teams that manage high-volume log sources, such as content delivery network (CDN) edges, streaming platforms, and authentication systems, often have to make a difficult retention tradeoff. Indexing every event keeps logs searchable during investigations, audits, and postmortems, but it can make long-term retention expensive. Archiving those logs in object storage helps control costs, but it often moves historical investigations into a separate query environment, such as Amazon S3 with Athena, a secondary data lake, or a dedicated CDN analytics tool. This fragmentation forces teams to work with a secondary query language, access control model, and operational context.

Datadog Observability Pipelines and Archive Search provide a cost-conscious pattern for retaining and investigating high-volume CDN logs. You can use Observability Pipelines to process and route raw edge logs to object storage while sending key metrics and high-signal events to Datadog for monitoring and alerting. When an investigation requires historical context, Archive Search lets you query archived logs from Datadog, helping teams keep CDN data accessible without indexing every event for long-term retention.

In this post, we will show you how to:

• Route high-volume logs to low-cost storage with Observability Pipelines
• Analyze archived logs in Datadog during incidents

Route high-volume logs to low-cost storage with Observability Pipelines

CDN access logs record every request handled at the edge, including client IP, request path, response status, and cache behavior. At streaming scale, where each video chunk can generate a separate HTTP request, these logs can reach tens of terabytes per day. Teams need to retain this data for network forensics, performance investigations, audits, and postmortems, but indexing all of it for long retention periods is expensive.

Video platforms and other high-volume streaming services typically run multiple CDNs in every region and shift traffic between them as conditions change—sometimes mid-event. When an incident spans multiple providers, no single portal contains the full edge context an investigation requires. Teams often archive CDN logs to object storage to retain that data without indexing every event.

Raw CDN logs can be difficult to operationalize quickly. Providers such as Cloudflare and Akamai use different formats, field names, and structures. To make those logs usable across providers, teams often need to build parsing rules, map provider-specific fields to a common schema, and configure routing logic before teams can query or monitor the data consistently during an incident.

Observability Pipelines Packs reduce that setup work. For common CDN sources like Cloudflare and Akamai, Observability Pipelines Packs include prebuilt parsing and processing logic that normalizes log formats, extracts useful attributes, and prepares events for routing or metric generation. Instead of building each processor from scratch, an SRE can start from a working pipeline, customize it for their environment, and route noisy raw events to object storage while sending high-signal metrics and alerts to Datadog.

With Observability Pipelines, you can process, filter, and route logs before sending them to downstream systems, helping you control volume and reduce indexing costs. For example, you can configure CDN providers like Cloudflare to stream logs directly into Observability Pipelines by using Logpush. From there, Observability Pipelines can process and route logs in real time, sending high-volume edge logs to low-cost storage while forwarding key signals to Datadog for monitoring and alerting.

You can split log traffic based on status, source, or custom attributes like env:production or team:platform. High-volume CDN access logs that are essential for network forensics—such as reconstructing traffic during a DDoS event or tracing geographic attack patterns—can route directly to object storage. Higher-signal logs, such as application errors and security alerts, can route to Datadog for indexing.

Generate metrics from CDN logs in transit

For many CDN log use cases, teams care about aggregated signals such as request rates, error counts, top source IPs, BotScore buckets, WAFAttackScore rates, and cache-miss rates by region. Observability Pipelines can generate metrics from these logs in transit and route those metrics to Datadog, while raw events land in cloud storage.

By choosing bounded dimensions such as status code classes, action types, and region groupings, teams can keep metric cardinality predictable as traffic scales. The raw archive still preserves unbounded questions, such as per-IP, per-user, and per-session investigations, for Archive Search when a metric signals a problem.

Normalize and enrich CDN logs before routing

Beyond routing, Observability Pipelines can normalize and enrich logs in transit by parsing unstructured formats, standardizing field names, and enriching events with host or environment metadata. You can also pull in external context. For example, ServiceNow CMDB enrichment adds owning team, service tier, and dependency information to every event at the time it was emitted.

The same pipeline can also apply to player-side telemetry data. CDN access logs record what was served but do not show how the player handled that content, such as whether playback buffered, dropped bitrate, or failed. Common Media Client Data (CMCD) helps close that gap by enabling streaming clients to attach player state, such as buffer length, requested bitrate, and session ID, to segment requests.

Routed through Observability Pipelines, CMCD events follow the same path as the rest of the CDN data: Bucketed metrics like rebuffer rate and bitrate distribution route to Datadog for dashboards and alerts, raw events stay in object storage, and Archive Search reaches into them when a metric needs to be tied back to specific sessions or edge points of presence (POPs).

Redact PII before logs leave your environment

Because Observability Pipelines runs on-premises, you can detect and redact personally identifiable information (PII) before logs leave your environment. Observability Pipelines writes to your object storage in Datadog’s native archive format, which makes it compatible with Archive Search without a separate reprocessing step.

Analyze archived logs in Datadog during incidents

During an investigation, teams often need to determine whether a similar issue has occurred before. A buffering spike during a live event, a surge of failed logins from a new region, or an anomalous error rate on a specific edge POP can all require moving from a real-time signal into months of historical context. The speed of that pivot determines how quickly an incident can be diagnosed—and how often historical analysis gets used at all.

Most CDN investigations are not catastrophic outages. They are often slower regional degradations, a single Autonomous System Number (ASN) spiking overnight, or a cache-hit ratio that drifts a few points and quietly inflates origin egress. These are the cases where teams skip the investigation altogether if the data lives in a separate tool. Bringing Archive Search into the same observability workflow makes that historical context practical to use.

Query archived CDN logs without leaving Datadog

Consider a high-profile live streaming event that attracts both legitimate traffic and bad actors. The on-call engineer runs service:cdn-edge AND @WAFAction:block over the event window, to check for credential stuffing, scraping, or password-spray attempts that occurred during the traffic surge. With Archive Search, the engineer can query logs stored in object storage directly from Datadog without switching to a separate analytics tool.

Before the scan begins, Archive Search includes a Query Preview feature that returns log samples before committing to a full archive scan. The engineer can use the Query Preview to confirm query syntax, time range, and filters before incurring scan compute costs. This is especially useful when working with archives that contain large compliance or security events spanning weeks or months.

Archive Search can also use partitions and lookup attributes to reduce scan scope. Partitions group logs by attributes such as date and service. When you configure partition attributes, Archive Search can skip blocks of data outside the query’s scope. Lookup attributes work similarly to database indexes, pre-filtering results before a full scan. Together, these options reduce scan time across large datasets in object storage.

After the engineer runs the search, results stream back into a familiar Datadog view with client IP, country, ASN, and request-path context. Because Archive Search operates inside the same observability workflow, teams can investigate historical CDN logs by using the same identities, access controls, and operational context already used for dashboards, monitors, and incident response.

Act on Archive Search results

If the engineer is also investigating indexed application logs, they can pivot from any blocked edge event to the application response that followed—all in the same UI, using the same search syntax. Archive Search makes long-tail patterns visible by using the same query surface. A regional cache-hit ratio that has quietly drifted from 99% to the high 80s, for example, can mean hundreds of gigabytes of unnecessary origin egress every day before anyone notices. This is exactly the kind of slow drift that goes unseen until someone pulls the underlying logs.

Archive Search uses the same search syntax and log facets that teams already use in Log Explorer. After a query runs, results are retained for 24 hours at no additional cost. From there, teams can re-index a targeted subset back into Datadog for deeper analysis or export results to CSV for offline investigation and stakeholder review.

In this example, Archive Search scanned 751 GB of archived data and isolated 242 relevant logs by targeting service:cdn-edge and an event type such as @WAFAction:block. With that narrowed result set, an engineer can reconstruct the timeline of a potential attack and review the specific source IPs, regions, ASNs, and request paths involved. The value goes beyond lower indexing volume—Archive Search lets teams move from a real-time signal to historical evidence without changing tools or query languages.

Investigate archived CDN logs in Datadog

High-volume CDN logs are too important to discard, but they are often too expensive to index in full for long retention periods. By using Observability Pipelines with Archive Search, teams can route raw CDN logs to low-cost object storage, generate metrics from those logs in transit, and search archived events in Datadog when investigations require historical context.

To get started, read the Observability Pipelines documentation and the Log Archives documentation.

Full article content is not available for inline reading.

Read the original article →

See your serverless: introducing the Headlamp plugin for Knative

Headlamp is an open-source, extensible Kubernetes SIG UI project designed to let you explore, manage, and debug cluster resources.

Knative brings serverless workloads to Kubernetes, handling traffic routing, autoscaling, and revision management so teams can deploy and iterate without fighting infrastructure. But operating Knative workloads day-to-day can be difficult, there's still a lot of jumping between the kn CLI, kubectl, and the Kubernetes UI to get a full picture of what's running.

We built the Headlamp Knative plugin to bridge that very gap, allowing operators to inspect, understand and act on their workloads all from a single place. This plugin was built as part of the LFX mentorship. Here's a tour of what we shipped.

Here is a short walkthrough of the Knative plugin for Headlamp:

Integrating Knative resources with Headlamp's map view

Headlamp's resource mapping works for Knative CRDs too. You can see how KServices, Revisions, and DomainMappings relate to each other in a single graph view.

KService management: edit traffic splits, restart pods, and view logs

A KService is the top-level resource in Knative: it manages the lifecycle of Routes, Configurations, Revisions, and everything needed to run and expose your application.

The plugin gives KServices a full detail view with an Edit Mode toggle for making live changes to traffic splits, autoscaling annotations, and more. Common actions like viewing the YAML, opening logs, triggering a redeploy, or restarting backing pods are surfaced in the header, gated by your current RBAC permissions.

Traffic splitting: route across revisions for gradual rollouts and testing

Knative makes it possible to route traffic across multiple Revisions of the same service. This is useful for canary releases, gradual rollouts, tagged preview URLs, and A/B testing.

The plugin shows the traffic assigned to each Revision, the latest ready Revision, readiness status, age, and configured tags. In edit mode, you can adjust percentages and tags inline. The plugin validates that traffic sums to 100% and that tags are unique before saving. Tagged routes with a reported URL render as clickable links.

Autoscaling configuration: view effective settings and cluster defaults

Knative's autoscaler supports a range of settings: concurrency targets, target utilization, RPS targets, min/max scale, initial scale, stable window, scale-down delay, and more. The effective value for any workload is a combination of KService-level annotations and cluster-wide ConfigMaps.

The plugin reads config-autoscaler and config-defaults and shows the effective configuration per KService in context, so you can see at a glance whether a setting is explicitly configured or falling back to the cluster default.

Prometheus metrics: monitor request rates, latency, and resource utilization

When paired with the Prometheus plugin for Headlamp, the plugin renders request rate, latency, and resource utilization graphs on KService and Revision detail pages. The per-revision request rate breakdown is particularly useful when validating a traffic split in progress.

Dashboard for other CRDs

The plugin also includes list and detail views for Revisions, DomainMappings, ClusterDomainClaims, and a cluster-level Networking overview (reading config-network and config-gateway to surface the effective ingress class, gateway settings, and backing services). These give operators a complete picture of Knative's state without leaving Headlamp.

How to install the Knative plugin in Headlamp

1. Make sure Knative is installed in your cluster.
2. In Headlamp Desktop, open the Plugin Catalog, search for Knative, and click Install.
3. Reload Headlamp, a new Knative entry will appear in the sidebar.

For development or source-level setup, see the Knative plugin README. The current release is 0.3.0-beta.

Share your feedback

We'd love feedback from Knative operators and users. If you hit a bug or want support for a workflow we haven't covered, please open an issue. You can also find us in the Kubernetes Slack #headlamp channel.

Introduction

Counter Service is used across Grab’s anti-fraud platform to answer time-windowed count questions, such as recent ride requests by a user or failed payment attempts on a card. The service handles tens of thousands of queries per second (QPS) with about a billion requests per day, while maintaining strict requirements around latency and reliability to support real-time fraud rule evaluation.

For most of its life, Counter Service was backed by a wide-column database that served the workload reliably as the service scaled. As part of a broader infrastructure review mandated at an organizational level, our database team evaluated alternatives to this storage that many services relied on, including Counter Service. Based on their assessment, Aerospike emerged as a good fit for our use-case. We also used the migration as an opportunity to decouple storage concerns from business logic, a necessary first step for this migration, and one that would reduce the effort required for future storage changes. As part of the same effort, we revisited the data model and access patterns in detail, which helped us identify and apply several straightforward optimizations.

This post walks through how we did it. What we built on the reader-side to make the migration safe, how we redesigned the writer-side data model around the new backend, and what we ran into during the gradual rollout.

Setting the stage

Counter data is stored in three time granularities: 15-minute, hourly, and daily buckets. A typical read would be along the lines of, “give me the count for key X over the last 90 minutes”, which the service decomposes into the smallest possible set of buckets, one hourly in the middle, a few 15-minute buckets at the edges, fetches them, and sums.

In the original setup, each granularity was stored in a separate table with a composite primary key:

TABLE daily_count (
    key      TEXT,         -- partition key
    day_ts   TIMESTAMP,    -- clustering key
    count    BIGINT,
    PRIMARY KEY (key, day_ts)
);

The clustering column gave us convenient range queries, that is needed for the Counter Service. On the write path, each incoming counter event triggered a read-modify-write, three parallel SELECT across the three tables, an in-memory increment, then a batch write. This produced four network round-trips per event.

As this service is a core part of Grab’s fraud detection ecosystem and handles high query volume, migrating its underlying storage required a careful rollout plan. We had three requirements:

• Ramp traffic to the new backend gradually and roll back at any point with a config change.
• Monitor both the original and new storage paths to verify data integrity before switching over.
• Complete the migration without downtime.

Reader: Separating the data access layer

The reader is a Rust service. Before any migration work began, the reader’s business logic had tight coupling with the storage layer. Session creation, query building, fan-out orchestration, and the data types those queries returned were all intertwined in a single flat file. The main application state struct (AppState) held a raw database session handle and prepared query references. Every handler, gRPC Remote Procedure Calls (gRPC) or HyperText Transfer Protocol (HTTP), received the bare session as a parameter. Variable names baked the storage technology into the business layer.

Stage 1: Extracting the storage code

The first stage shipped no behavioural change. We deleted the monolithic storage file and split its contents in two:

• storage/legacy.rs: wrapped session creation, prepared statements, and query execution behind a LegacyStorage struct.
• batch_read_ops.rs: kept only the orchestration logic: time-range splitting, channel-based fan-out, and aggregation.

AppState started holding an Arc<LegacyStorage> instead of a raw session handle. The PreparedQueries struct lost its statements (those moved inside LegacyStorage). We renamed every storage-specific identifier in business code to generic storage_* names.

Stage 2: The storage facade

With the seam in place, we introduced the actual abstraction. A new storage/ module with mod.rs, legacy.rs, aerospike.rs, and mock_storage.rs as siblings became the only place driver crates were reachable from.

We chose a concrete facade with enum dispatch:

struct Storage {
    legacy:    LegacyStorage,
    aerospike: AerospikeStorage,
    mock:      MockStorage,
    settings:  StorageSettings,
}

execute_queries(backend: BackendType, ...) {
    match backend {
        Legacy    => self.legacy.execute(...),
        Aerospike => self.aerospike.execute(...),
        Mock      => self.mock.execute(...),
    }
}

On top of the facade we layered three config-driven operating modes that map to the migration phases:

• Single: one backend serves the request.
• WithShadow: the primary serves the response; the secondary runs asynchronously in the background for parity comparison.
• WithSplit: a deterministic percentage of traffic is served by each backend. Used for the live cutover.

Stage 3: Shadow comparison and metrics

Each storage call carries metadata like backend, role (primary/secondary/shadow), and mode, attached as tags to every metric. We placed the mode dispatch at the handler level rather than inside the storage layer to validate the full request path, not only the rows returned by storage.

Writer: redesigning the data model

Approach 3: Map-based schema

The third approach was structurally different from the first two and was the most compact of the options. Rather than storing one record per bucket, which kept us in the same cardinality regime, we collapsed all bucket counts for a single counter into one record. The values were stored as a sorted map keyed by bucket timestamp:

Set:           helium_hourly
Primary key:   "{counterKey}"
Bins:
   counts: KEY_ORDERED_MAP({
        1773369000000: 1,
        1773372600000: 3,
        1773376200000: 7,
        ...
   })

Writes use MapIncrementOp, an atomic server-side increment of a value at a given map key, creating the entry on first access. Combined with MapRemoveByKeyRangeOp for pruning stale entries, every write is one atomic operation.

Reader: How each backend actually serves a query

The reader takes a batch of counter queries and decomposes each into one or more sub-queries per granularity. In the original backend, each sub-query becomes its own prepared statement. On Aerospike, the storage layer first groups all sub-queries by granularity, then issues one BatchOperate per granularity. Each sub-query becomes a single primary-key read against the appropriate set; the server returns the entire counts map for that key in one record.

Experiment with indexing

We run Aerospike in its default storage configuration, Hybrid Memory Architecture (HMA), where the primary index sits in Random-Access Memory (RAM) and the data sits on Solid-State Drive (SSD). To raise the memory ceiling, we tried moving the primary index itself from RAM to local Non-Volatile Memory Express (NVMe) while keeping data on SSD. In practice, we started seeing p99 spikes that followed I/O activity on hot keys. Consequently, we reverted back to in-memory index with a memory-optimized instance type.

Overall impact

The migration delivered gains across infrastructure, performance, and data footprint. The primary index currently uses about 50 GB of the roughly 100 GB usable memory per node. The same dataset is around 1 TB on disk, compared with around 3 TB on the original setup. In production, p99 read latency was consistently better than the original setup, with roughly 50% improvement across our read paths. The write path now uses a single atomic increment operation, replacing the read-modify-write pattern we had built previously. The new setup costs roughly 45–50% less per node compared to our original setup.

Conclusion

This migration ultimately came down to aligning the storage design with the workload. The gains came from focusing on optimization opportunities, redesigning the data model, and cleanly separating storage concerns. Through shadow reads and writes, followed by a gradual rollout, we completed the migration with zero downtime and no data-integrity issues.

The context engineering playbook (Claire Gouze)

nao co-founder and CEO Claire Gouze shares a practical playbook for building a context layer your agents can rely on.

Context is everything in data right now. Everyone is looking for the same thing: a single place where you can ask a natural-language question and get back a reliable answer. With that, you can build conversational analytics, but you can also build essentially any agent that needs to connect to your organization’s data. As long as you have context.

Claire Gouze is the co-founder and CEO of nao Labs, an open-source analytics agent built for context engineering she started with Christophe Blefari. Her path into data was unconventional: a business school graduate who taught herself to code, she became one of the first business school hires at BCG Gamma, then ran data at sunday, a QR-code payments startup, where she built a data stack from scratch as the company grew from 20 to 300 people. nao came out of calling 80 different data teams and listening to what was slowing them down.

Lots of people are talking about building context layers and hiring context engineers. Claire and her team are doing it: they’ve authored a context engineering playbook with specific guidance on how to build your own context layer and how to create evals. They’ve built a community learning together in the open, and they’ve built tooling to make it easier. What I appreciated most about this conversation was its pragmatism. Rather than talking about context, we should all get to work engineering it. Claire’s playbook is a great place to start.

Three ideas from the episode

1. Context engineering is the new analytics engineering. The job is the same as it always was, gathering tacit business knowledge and turning it into something structured and trustworthy. The medium is new: markdown and files instead of only models. Claire already knows data people who have been renamed context engineers.
2. The biggest reliability gains are unglamorous. Fancy context sources don’t move the needle as much as you’d hope. Cleaning up her data model and writing good documentation is what took Claire’s agent from 40% to 90% reliability. Anthropic found the same thing: query logs added little; keeping your house in order added a lot.
3. We’re in the “just plug it into production” era of agents. Connecting an agent straight to every raw source is the 2010s mistake of plugging your BI tool into the production database, repeated. Context will need its own stack: a way to ingest it, transform it, resolve contradictions, and expose a single source of truth.

Key takeaways

Tristan Handy: How did you get into data?

Claire Gouze: My background is unconventional for data. I graduated from business school about 10 years ago, but I wanted to learn technical things, so I joined BCG Gamma, the data science arm of BCG. I was the first business school hire there. I spent three years building ML models for clients: forecasting, personalization, optimization.

Then I joined a startup, sunday, doing QR-code payments for restaurants, because I wanted to build my own company one day. They had nothing, just their production database plugged into Metabase. It was 20 people when I joined and 300 a year later, so we had to move fast. I came from consulting, where you build everything custom, so I built an ETL by myself, an ingestion pipeline in Python for Salesforce data, a transformation layer in Python. Then people told me there are tools called dbt and Airbyte. So I had to migrate all my custom work onto the standard stack. That was a hard lesson, and a useful one.

Why did you move from a “cursor for data” to a context layer?

When we started two years ago, the loudest pain point from the 80 teams we called was that it takes too long to ship dbt models. So our first product was a cursor for data: your IDE plugged into your data, with the agent holding all the context. But as MCP took off, Cursor and Claude were handling that well. The new thing data teams wanted was a way to let anyone at the company use agents on the data.

We just want to work on what excites people. Data people aren’t excited about how they code. They’re excited when they help business users and get valued for it. Data teams carry the trauma of being seen as a support team. If we can help them be valued by the business, that’s the greatest thing we can do for them.

Is there actually a product to build in context, or is it just best practices?

The main value we add is evaluation and governance. Every team I talk to puts something different in their context: some have full documentation, some have a semantic layer, some have almost nothing on their tables. But they all use the framework to test the reliability of the agent and to keep testing it over time in CI/CD. They also study the conversations users have with the agent, which shows them what people care about and where to improve.

We try not to overcomplicate it. Some people want ontologies and semantics. We want you to start simple. The context layer is a file system. We help you build it like a GitHub repo that belongs to you and doesn’t lock you in, and we add evaluation, permissions, and a UI on top.

What does the context engineering playbook involve?

It’s more about method than about exactly what to put in your context, because that differs for a startup with 10 tables and an enterprise with thousands. Start focused. Pick the team that asks for the most analytics, or your main company metrics, and reduce the scope to maybe 10 or 20 tables. Plug in what you already have, usually your dbt docs, and run your tests. That gives you a baseline reliability number. Then you iterate: see where the agent fails, redesign part of the data model, add documentation, profile a table. It’s an iterative loop.

Where do the eval questions come from?

Either you already know your most important questions and have the queries somewhere in your BI tool, so you collect those, or you use a skill we built that looks at the main metrics of your tables and suggests key questions to test. I still recommend you review them, but it gives you a first basis. Then you can say: on my 50 most important questions, I have 90% accuracy, and it’s going to stay that way. That number is what reassures a data team enough to roll the agent out.

Is “context engineer” a real job title?

Yes, I know data people who were renamed context engineers, so it’s already happening. Data teams are the perfect fit. Analytics engineering was about gathering business knowledge from stakeholders and translating it into something structured and technical. Context engineering is exactly that. Context is just company knowledge that you want structured, optimized so it doesn’t explode your token cost, and treated as a source of truth, the same way you’d want a metric source of truth. Data teams already think this way.

What context actually moves the needle on reliability?

It’s funny, the biggest jump in agent reliability is just your data modeling and your data docs. I ran the experiment: I started from no context and added sources step by step, measuring reliability each time. Profiling, query history, those kinds of things left me stuck around 40%. The agent was failing because of ambiguity between two columns, or a metric that differed slightly across two tables. When I redid parts of the data model and wrote documentation, I got to about 90%. It’s deep work to keep a data model clean and unambiguous, but it pays off with agents.

Where should human context live?

In our framework everything ends up as a markdown file eventually, so the starting format doesn’t matter much. What matters is where it gets maintained. If someone asks whether to document something in the agent’s context or in the dbt docs, I say the dbt docs, because it has to live as close to your daily work as possible. When you change a dbt model, you change the docs, and it syncs to the agent. If your support processes live in Notion, keep them in Notion. A separate markdown file you never touch again is worthless.

We’re in the “just plug it into production” moment. What comes next?

We’re at the phase where people say, let me just connect Claude Code to my Snowflake MCP, what could go wrong? It’s the same as when you plugged your BI tool straight into the production database, before the data stack gave us tools to ingest, transform, and create a source of truth. We’ll need the same thing for context. Right now we’re building the first wave, but we’ll start to see context sprawl and contradictions. So we’ll need a stack to ingest context, transform it, merge old and new and contradictory context, and expose a single source of truth to the agent. Where are the tools to do that? I don’t know yet.

How does this connect to memory?

When an agent corrects itself after eight queries to find the right field, you don’t want it to repeat that next time, you want it in its context. Same when a user tells the agent, no, this is the real definition. We should learn from all of it. It’s a memory mechanism. But the tricky part is making sure you learn the right memory. Locally with Claude Code it’s just you and the agent, so it can learn whatever you tell it, even if it’s wrong. At the company level you have to make sure people don’t teach the agent wrong things, so the data team still approves what enters the global memory of the company.

Where does MetricFlow fit if “file systems are all you need”?

I tested the skill you all built for querying through the MetricFlow semantic layer. The logic was: query through MetricFlow first, and if you don’t find it, read the dbt docs and write regular SQL. I think that’s exactly right. The metric layer is governance for your most critical, high-value metrics that have to be 100% accurate, but you shouldn’t have to define a metric before you can do anything at all.

Why open source, and how do you make money on it?

Open source makes sense for a few reasons. You want to be used by agents, not just humans, and if your code is open, an agent building an analytics agent already knows about nao and will build with our framework, which is great distribution. And nobody has the answers on what makes good context yet, so we need to learn together. If startups and enterprises can share what context worked and how the semantic layer affected their reliability, we get a common language and improve reliability across the board. We already know what we sell: our open-source product gives everyone the same data access, which works for a small company but not a big one. The enterprise license handles data permissions, context permissions, and token budgets at scale.

Chapters

00:00 — The missing piece is context
01:45 — Welcome, and a few words on the French accent
02:52 — Claire’s path into data: business school to BCG Gamma
04:09 — Joining sunday and building a data stack from scratch
06:39 — Founding nao, an open-source analytics agent
10:42 — The journey so far: 1,300 GitHub stars, 80 companies in production
13:45 — Why pivot from “cursor for data” to the context layer
15:22 — Context layer hype at Snowflake Summit
17:48 — Is there a product in context, or just best practices?
21:23 — The context engineering playbook: start small, iterate
24:45 — Where the 50 eval questions come from
25:40 — The questions data teams never get asked
27:06 — Is “context engineer” a real job title?
28:55 — Flattening roles on the data team
29:56 — Machine vs. human context, and where to keep it
32:43 — The highest-signal context: clean data models and docs
35:36 — Where nao is headed: a source of truth across Slack, MCP, and more
37:41 — From the data lake for analytics to infrastructure for agents
39:05 — The “just plug it into production” moment and the context stack
41:49 — Roadmap: automating context creation
43:33 — Context as organizational memory
46:00 — File systems, MetricFlow, and the semantic layer question
49:05 — Why open source, and the commercial model
51:39 — Wrap-up

Comparing the best open source vector databases

Open source vector databases come in two flavors: specialized tools that handle vectors and nothing else, or unified platforms that combine vector search with operational data and caching. Many teams end up managing three systems: a vector database, a cache, and an operational store. Redis combines all three in a single real-time data platform with a memory-first architecture.

This comparison breaks down the leading open source vector databases for production AI workloads. Each database in this comparison optimizes for different tradeoffs, and the right choice depends on your scale, deployment constraints, and infrastructure preferences.

Best open source vector databases at a glance

What makes a vector database production-ready

Most vector databases solve the same core problem: store vector embeddings, find similar ones fast. The differences show up when you move from prototype to production.

Production AI apps don't run in isolation. Your retrieval-augmented generation (RAG) pipeline needs session state. Your chatbot needs rate limiting. Your recommendation engine needs real-time feature data.

With specialized vector databases, that means managing separate systems: a vector store, a cache, an operational database, and keeping them in sync. A unified platform handles vectors alongside everything else, reducing the number of systems to manage and eliminating network hops between your vector search and your cache.

Three deployment models matter here: fully managed cloud services, self-managed enterprise deployments, and open source for teams who want full control. Some vector databases work best with Kubernetes orchestration, while others offer cloud-hosted options primarily. The deployment flexibility available to you often determines how much operational overhead you'll carry.

Redis Iris serves agent context in milliseconds

The top open source vector databases & comparison

Here's how the leading options stack up for production AI workloads.

Redis

Redis provides vector search as part of a unified real-time data platform, not a standalone vector database bolted onto other tools. Vector embeddings, session data, rate limiting counters, and application state can live in one system with a memory-first architecture and sub-millisecond latency for many caching and real-time operations.

• Vector search architecture: Redis uses Hierarchical Navigable Small World (HNSW) indexing for datasets exceeding 1 million documents where performance and scalability take priority, plus FLAT indexing for exact nearest neighbor search when precision requirements override performance. Supports text, image, and video vector embeddings from any model provider.
• Billion-scale performance: In a billion-vector benchmark, Redis reported 90% precision at ~200ms median latency for the top 100 nearest neighbors under 50 concurrent queries, sustaining roughly 66,000 vector insertions per second at that precision.
• Hybrid search: The FT.HYBRID command combines vector similarity with filtering on geographic, numeric, tag, or text data.
• Semantic caching: Redis LangCache stores large language model (LLM) responses and serves cached results for semantically similar queries. You trade an embedding lookup plus similarity search for avoiding much slower LLM inference calls.
• Deployment options: Redis Cloud offers fully managed infrastructure with no Kubernetes expertise required. Redis Software provides self-managed deployment with enterprise-grade compliance. Redis Open Source is free with no orchestration needed.

Redis fits production AI apps that need vector search alongside operational data, particularly teams wanting to consolidate infrastructure rather than managing separate systems for vectors, caching, and operational workloads.

Milvus

Milvus is a cloud-native distributed vector database built for horizontal scaling across multiple nodes. Version 2.0 introduced a microservices architecture targeting large enterprises with massive vector workloads. It's Apache 2.0 licensed.

The tradeoff is operational complexity. Milvus Distributed is commonly deployed on Kubernetes for production at scale, though Milvus Standalone on a single machine is available for workloads that don't require distributed infrastructure. Organizations already running Kubernetes may find the distributed mode straightforward; teams without that expertise face a steeper learning curve.

Weaviate

Weaviate combines vector similarity with keyword search through hybrid search capabilities, using HNSW indexing. It offers multiple API options: REST, GraphQL, and gRPC, with GraphQL and gRPC commonly used for queries.

Qdrant

Built in Rust, Qdrant emphasizes memory safety with filtering capabilities for metadata-heavy queries. The filtering-optimized architecture handles complex metadata queries, though latency can vary depending on filter complexity and dataset characteristics.

Chroma

Chroma prioritizes simplicity and developer experience, particularly for Python workflows. It's popular for rapid prototyping and works well for local development, especially in early-stage projects.

pgvector (PostgreSQL Extension)

pgvector adds vector search capabilities to existing PostgreSQL deployments, keeping everything in one system for teams already running PostgreSQL. The tradeoff is that pgvector requires PostgreSQL tuning expertise for vector workloads and doesn't include caching capabilities.

Faiss (Meta AI Research)

Faiss is a C++ library for efficient similarity search and clustering of dense vectors. It's explicitly designed as a library, not a database. That means you get algorithms, not infrastructure.

Teams adopting Faiss must build persistence, backup and recovery, replication, and monitoring on their own. For production deployments, Redis provides these database features with vector search built in.

Build agents that remember, not agents that guess

What to consider when choosing

How many systems do you want to manage?

Specialized vector databases do one thing well, but production AI apps need more than vector search. You'll typically need caching for performance, an operational database for application state, and the vector store itself. That's three systems to deploy, monitor, secure, and keep in sync.

What's your LLM cost situation?

If you're running LLM workloads at scale, inference costs add up fast. Semantic caching, which stores LLM responses and serves cached results for semantically similar queries, can reduce those costs significantly.

What deployment expertise does your team have?

Some vector databases require Kubernetes for production deployment. If your team already runs Kubernetes at scale, that's not a barrier. If they don't, you're looking at weeks or months of learning curve before you can go live.

Fresh context, every call

Choosing the right open source vector database

Each vector database in this comparison solves a specific problem. Milvus handles massive distributed workloads if you have Kubernetes expertise, or standalone mode for simpler deployments. Qdrant offers strong filtering with competitive latency. pgvector keeps everything in PostgreSQL and now handles tens of millions of vectors. Chroma gets you prototyping fast.

The bigger question is whether you want a specialized vector store or a unified platform. Specialized tools do one thing well but add operational complexity, since you'll run separate systems for caching, session management, and operational data. A unified approach puts vectors alongside everything else your AI app needs.

Redis takes the unified path: vector search in production alongside caching, operational data structures, and semantic caching in one system. That consolidation often matters more than marginal differences in pure vector search benchmarks, and it's the foundation for Redis Iris, the real-time context engine that keeps agents grounded in fresh data. You get low latency without Kubernetes complexity and one platform for vectors, caching, and operational data.

Frequently asked questions

Is Redis a vector database?

Redis provides vector search with HNSW and FLAT indexing, so it works as a vector database. It also handles caching, session data, and operational data in the same system, so vectors don't need a separate store.

What's the best open source vector database for RAG?

It depends on your workload. If you want vector search alongside caching and operational data in one system, Redis fits. For distributed billion-scale workloads, Milvus is built for that; for filtering-heavy retrieval, Qdrant is a strong option.

Are open source vector databases free?

The software is free to self-host, but you still pay for the infrastructure underneath and carry the operational work: backups, scaling, monitoring, and upgrades. Managed services remove that work for a usage cost.

Open source vs managed vector database: which should you choose?

Self-hosting gives you full control over data and configuration but makes your team responsible for uptime, scaling, and maintenance. Managed services trade some control for lower operational overhead. Redis offers both paths, so you can start managed and move to self-managed, or the reverse, without changing engines.

StreamFusion

Run Apache Flink SQL faster by executing supported operators natively (Rust + Apache Arrow/DataFusion over JNI) while Flink continues to own planning, coordination, and everything not yet supported. Substitution is transparent and conservative: a query is planned by Flink, the operators we can reproduce exactly are swapped for native ones, and anything else falls back to Flink with identical results.

It is DataFusion Comet's idea — a native, columnar accelerator behind an unchanged SQL front end — applied to streaming instead of batch: stateful windowing, joins, aggregations, changelog processing, and columnar sources/sinks, not just stateless projection and filter.

What it accelerates

A query accelerates only when it forms one fully-columnar island: every operator except a rowwise source/sink runs natively, exchanging Arrow batches (the row↔Arrow transpose is paid once at the host edges, never between native operators). A single unsupported interior operator drags the whole query back to Flink.

Native coverage is broad — most of the streaming SQL surface:

• Stateless: projection/Calc, filter, UNION ALL, GROUPING SETS/CUBE/ROLLUP, UNNEST.
• Windowed aggregates: TUMBLE/HOP/SESSION/CUMULATE (event-time and proctime, one- and two-phase), and OVER window functions.
• Joins: regular (updating) equi-joins, event-time/proctime interval and window joins, event-time temporal-table joins, and processing-time lookup joins (sync and async).
• Changelog: non-windowed GROUP BY, streaming Top-N / LIMIT, deduplication, changelog normalization — all consuming and emitting a retract changelog.
• Connectors: Parquet/ORC file sources and a Parquet sink (native Arrow scan/write); Kafka source decode for JSON/CSV/raw/Avro/protobuf and Debezium/OGG CDC — JSON/Avro/protobuf via a fully native rdkafka source (the default path; it also regenerates the table's watermark per partition, exactly as Flink's source does).
• UDFs: a Flink ScalarFunction the expression engine can't implement itself is invoked over Arrow columns by a native→JVM upcall (Comet's JvmScalarUdfExpr pattern), one JNI crossing per batch, so the pipeline stays native through the UDF and the result is byte-identical.

Determinism. Results are byte-identical to stock Flink for everything admitted. The one caveat is late-data dropping on out-of-order event-time streams, where Flink is itself non-deterministic (periodic watermarks); we match Flink's deterministic path, which governs in-order data and every benchmark.

Inspiration

StreamFusion is built by porting established engines rather than reinventing operators:

• DataFusion Comet — the model for the whole project (native columnar accelerator behind an unchanged SQL planner) and the reference for the JNI / Arrow C Data Interface bridge, off-heap memory accounting, the config surface, and fallback-reason reporting.
• Arroyo — the streaming-operator implementations we port (it already runs on DataFusion); the reference for join/window/changelog logic.
• Apache DataFusion — the native execution and expression engine underneath (hash joins, aggregates, Arrow kernels).
• RisingWave — the reference for changelog semantics and memcomparable arrow-row state encoding.
• Apache Flink — the parity target: every operator is a faithful port of Flink's own, verified for identical output by a parity harness.

Running and configuration

Install acceleration by hooking the planner once (NativePlanner.install(env)), then run Flink SQL as normal. Two things to set in a real deployment:

Deployment JVM flags — run the TaskManager JVM with Arrow's safety checks off (as Comet/Spark do); profiling showed ~1/3 of the transpose CPU was per-accessor bounds/refcount checks:

-Darrow.enable_unsafe_memory_access=true -Darrow.enable_null_check_for_get=false

Configuration (JVM system properties, mirroring Comet's config surface):

• -Dstreamfusion.native.enabled=false — master switch; run entirely on Flink.
• -Dstreamfusion.operator.<name>.enabled=false — keep one operator on the host (e.g. leave a lone cheap filter on a row source, which can't earn back the transpose round-trip).
• -Dstreamfusion.expression.<NAME>.allowIncompatible=true — opt into the faster pure-Rust path for expressions that otherwise use a byte-exact JVM upcall or fall back (UPPER/LOWER/ REGEXP_EXTRACT, ROUND on float, transcendental math). Off by default (parity-first).
• -Dstreamfusion.memory.accounting.enabled (default on) — native stateful operators reserve an operator-scope share of the slot's managed memory from Flink's MemoryManager and bound their state by it, failing with a NativeMemoryLimitException naming the remedy rather than an unattributed OOM.

Related work

Three native Flink accelerators exist, all closed source:

• Flash (Alibaba Cloud) — a C++ native + SIMD vectorized engine with a custom state backend (ForStDB). Stateful, production-deployed at scale; claims 5–10× on streaming Nexmark, 3×+ on batch TPC-DS, and ~50% cost reduction across 100k+ compute units.
• Vera X (Ververica, the original Flink creators) — a proprietary native vectorized engine with a drop-in compatibility layer and a new state store. Stateful; claims 5–10× on Nexmark SQL and ~52% lower resource usage.
• Iron Vector (Irontools) — the same stack as us (Rust + Arrow + DataFusion over zero-copy JNI, Substrait plan serialization, transparent fallback), but stateless only today (projections, filters, expressions); windows, joins, and exactly-once are described as planned.

Where StreamFusion differs: it is open source, and every substitution is gated and verified for identical results against stock Flink by a parity harness rather than asserted. It is already native on stateful windowing, joins, and changelog processing — the hard, closed part of the field — where Iron Vector is stateless-only.

License

Licensed under the Apache License, Version 2.0.

Variance Reduction Below the Randomization Grain

Sergio Camelo, Caitlin Kearns, Matias Cersosimo, and Tilman Drerup

As artificial intelligence increases the velocity of engineering and science teams, experimental throughput is set to become a bottleneck for many product decisions. Many companies can now build faster than they can experiment, with queues of good ideas running the risk of not being tested because of lack of experimental capacity.

This problem is particularly severe in marketplaces, where the presence of spillover and cannibalization effects between experimental units requires cluster-level randomization techniques. That randomization, in turn, has the unfortunate tendency to substantially reduce statistical power and slow down experimentation. In this post, we show that the predictability of outcomes at fine grains can be exploited to reduce the variance of aggregate metrics, even when experiments themselves are run at a coarse level. Since statistical power depends on metric variability, this yields considerable reductions in experimentation time.

The Interference Problem

In marketplace settings, behavior and outcomes for individual participants are inherently intertwined. In a delivery marketplace like Instacart, for example, the dispatch system solves a bipartite matching problem between shoppers and customer orders. Since assignments are global and interdependent, matching an order to one shopper means that the same order cannot be matched to another shopper. As a result, changing the handling for a single order creates ripples that affect the orders around it. If an experimenter were to assign a treatment intervention to one of these orders while leaving neighboring orders as controls, the latter would evidently be contaminated.

A common response to this problem is to randomize treatments at the level of a cluster, chosen so that interference can stay within it. In food and grocery delivery, that cluster is typically a geographical region. Since every order within a region sees the same treatment, contamination is contained at the regional boundary rather than leaking between treatment and control. The drawback of such region-level designs is that they give us far fewer units of randomization than unit-level designs, and fewer units lead to less statistical power.

One way to recover some of that power is a switchback design. With a switchback design, each region is re-randomized between the treatment and control group every fixed window (typically one day). The unit of analysis then becomes the region-day rather than the region, and the effective sample size grows with the duration of the experiment. This, of course, relies on the assumption that a window’s treatment effect is realized within that same window and does not spill into the next. Still, even with this adjusted design, the effective sample size typically remains small, and power remains an issue. At Instacart, running powered experiments may take months, and for very noisy metrics, may even be entirely infeasible in any reasonable timeframe.

Enter CUPED

One way to recover some of the lost power is CUPED (Deng, Xu, Kohavi, and Walker [2013]), layered on top of a switchback design. The intuition behind CUPED is that if the outcome metric has a component that can be predicted from pre-treatment features, then subtracting that predictable component removes noise without removing the treatment effect, thus making the effect easier to detect. Let’s make this concrete.

Throughout, subscript k indexes a region-day observation. Suppose we run a switchback experiment and we observe an outcome metric Yₖ for region-day k. Let Zₖ be a vector of pre-treatment features for that region-day, and let Xₖ = f(Zₖ) be a prediction of Yₖ built from those features, where f is fit on pre-treatment data. We form the CUPED-adjusted outcome:

This adjusted outcome has the same expectation as Yₖ, so it can be used to estimate the treatment effect without introducing bias, while its variance is reduced by a factor of 1 − ρ², where ρ is the correlation between Yₖ and Xₖ. The better Xₖ predicts the outcome, the larger the variance reduction, and therefore the shorter the experiment needed to detect a given effect size.

This naturally leads to the following question: How good can this predictor get? A key limitation of CUPED under interference is that the predictor Xₖ must be built at the coarsest level of randomization. At the region-day level, the features available — day of week, average basket size, shopper supply, weather — all capture broad patterns but miss the specific mix of orders that land in a region on a given day. However, for a metric like late-delivery percentage, most of the variation comes from order-level characteristics, e.g., where each delivery is going, what is in the cart, what window the customer chose. Averaging those away before building the predictor discards the signal that would make it useful. In many experiments, we have found that standard CUPED constrained to region-day predictors can cut variance by up to 10% after controlling for geo- and day-fixed effects. That is a good start, but not enough to meaningfully shorten experiment duration.

Order-Level CUPED

While we cannot experiment at the order level, we still have access to plenty of order-specific information that we know drives our metrics. If order-level outcomes are what we ultimately aggregate into our metrics, why not predict at that level and aggregate the predictions the same way? This keeps the structure of CUPED intact and allows us to take into account order-specific covariates by training a suitably-chosen machine learning model.

Concretely, our methodology works in two steps: First, we train an order-level model g that predicts each order’s outcome, ŷᵢ = g(zᵢ), using only features zᵢ available before the fulfillment system acts on the order. Second, we aggregate this model’s predictions up to the region-day grain using the same aggregation the metric itself uses (e.g., a mean, for a metric like late-delivery percentage):

The aggregated prediction Ŷₖ then becomes Xₖ in the CUPED adjustment from the previous section: it is a pre-treatment prediction of Yₖ, now informed by order-level detail that a region-day feature could not capture.

A Word of Caution

Before we go to the results, one word of caution is necessary to ensure that the expected variance reduction does not come with an increase in bias for the treatment effect. When estimating the first step regression, every feature used by the order-level model must be unaffected by the treatment. In predicting an order’s lateness, features like items in the cart, chosen delivery option, store location, time of day, and weather are safe because they are fixed before any fulfillment algorithm acts on the order. Features like the shopper assigned to the order or how many orders that shopper handles on the same trip are off the table because they are downstream of the treatment. The distinction matters because some of the forbidden features may be the best predictors of the outcome, and therefore the most tempting to add.

This assumption is essential, and we can test it. To do so, we run a mean difference test on Ŷₖ, the prediction built from pre-treatment features, in place of the real outcome Yₖ. Since Ŷₖ depends only on pre-treatment features, its estimated treatment effect should be statistically indistinguishable from zero. If it is not, something in the feature set is being moved by the treatment, and the variance-reduced estimate cannot be trusted for that experiment.

The plot below illustrates this in three experiments, reporting for each the variance reduction from order-level CUPED alongside the placebo test that flags feature contamination. In our marketplace, for example, we experiment with improved versions of the models that predict the ETA of an order, which affects the ETA shown to users before they place an order, so they can make an informed decision. This shifts the distribution of orders placed, and the placebo test correctly flags this as contamination.

Results: Experimentation Speed-Up

We tested the method across 10 experiments aimed at reducing order lateness, one of Instacart’s highest priorities. For each experiment, we trained a Gradient Boosted Tree to predict whether an order would be late. Our models fit the data fairly well, with expected features doing the heavy lifting in the prediction, including weather and basket size.

We then did a side-by-side comparison of standard CUPED with region-day features and order-level CUPED with order-level features. The gap is stark. Order-level CUPED cuts variance by 18 to 40%, more than 3 times the 5 to 10% reduction we get from only applying region-day CUPED.

These reductions in variance have immediate and notable effects on experimentation velocity, allowing us to cut experimental runtimes by about a third on average. This added velocity allows us to run more experiments per quarter and rapidly identify improvements in the algorithms that power our marketplace, including the models that ensure that customers get their groceries on time.

Wrapping Up

When the unit of randomization is coarser than the unit of observation, small effective sample sizes tend to slow down experimentation. In this post, we showed how building predictions at finer grains, in our case order-level, and aggregating them up can recover variance that standard region-day CUPED leaves on the table. The move is counterintuitive at first, since interference is exactly why we randomize coarsely and avoid order-level outcomes in the first place. The crux of our solution is that the predictions only ever enter through a pre-treatment covariate, leaving the estimates unbiased. As our results above show, this change can have meaningful effects on experimental power and velocity.

Importantly, the presented approach is not specific to the delivery context. In settings where interference forces randomization into coarse clusters (like with graph clusters in social networks, keyword groups in ad auctions, or geographic markets from lodging to dating), outcomes are still observed on a much finer unit than the one being randomized. That gap is exactly what the proposed method exploits, and its underlying idea should be directly applicable.

Bridging the Gap: Blending Structured Data Auditing with Unstructured Policy Intelligence

How an Agentic Harness Loop Transforms Corporate Loan Analysis from Static QA into Autonomous Risk Research Agent

In modern corporate “Data” remains highly siloed which prevents corporate from using unstructured dcouments (e.g. regulatory frameworks, compliance rules, internal operating guidelines) with structured data stored in transaction databases and data warehouses. Traditional search systems, such as basic keyword search or standard Retrieval-Augmented Generation (RAG) Q&A chatbots, fail to bridge this gap.

Problem Statement

Large organizations routinely face complex procedure and policy documents, such as compliance guidelines or financial standards. The information required to conduct these audits is split across two fundamentally different domains:

• Unstructured Qualitative Intelligence: Policy guidelines, regulatory mandates, and internal manuals. These reside as text inside PDFs or corporate wiki pages.
• Structured Quantitative Records: Transaction logs, customer profiles, loan data, and risk metrics. These live in relational tables, data lakes, or databases.

When auditing compliance, human auditors must manually read a policy (e.g., Lenders must identify public derogatory records and delinquent credit events), translate that rule into database queries, analyze the results, and compile a report. Traditional AI assistants cannot automate this because:

1. Document Intelligence lacks database awareness: General Knowledge Base does not know the schema of the database or how business terminology maps to cryptic column names (e.g., matching “delinquent credit event” to a column named `delinq.2yrs`). Therefore, standard Text2SQL pipelines cannot generate the correct SQL.

2. Standard Text2SQL Lacks agentic looping: Research and audit analysis require action planning first, then querying the document base and transactional database, then analyzing the intersection, and finally critiquing the draft.

Solution:

The solution combines the LLM-Wiki with Text2SQL to build a unified agentic workflow that acts as an Autonomous Analytic Agent. Instead of answering simple questions, the user provides a high-level objective, such as: Identify if there are any high risk loan records which violate against the key risk indicators in documentation guidelines.

The agent coordinates multiple Specialized Agentic workflow inside a self-correcting loop. It automatically extracts relevant rules from unstructured documents using LLM-Wiki, and then maps them to database schemas, generates executable SQL queries, performs analysis and evaluations. The system continues to replan and query until it achieves a high-confidence evaluation.

In my demo use case, the agentic system can identify high risk loan records which violate against the key risk indicators in documentation guidelines.

A. The Ingestion Layer

Before running audits, the system parses the domain’s structural and semantic foundations:

1. Structured Semantic Mapping: The pipeline recursively parses OWL Turtle (.ttl) files (e.g., in my demo use case, it is Financial Industry Business Ontology — FIBO) using RDFLib. It maps these OWL classes and attributes to dataset tables/columns and then create a mapping that defines how business concepts relate to physical columns.

2. Unstructured Document Ingestion: The pipeline converts regulatory PDFs into Markdown and segments the text. It classifies segments into ontology concept types and reconciles candidates to deduplicate concepts.

B. The Semantic Translation Layer: Ontologies & YAML Mapping

To make database tables queryable by LLMs without guessing, we use:

1. Ontology (`.ttl` files): Defines domain-specific business terms (e.g., Loan, BorrowerDebtToIncome, DerogatoryPublicRecord).
2. Semantic Mapping (mapping.yaml): Maps ontology attributes to database columns (e.g., matching the concept BorrowerPublicRecordCount to pub.rec). It includes SQL formulas, explanations, and flags highlighting missing information. The LLM Planner and SQL Generator read this file to translate qualitative guidance into exact SQL syntax.

C. The Agentic Loop Layer

The agentic workflow operates as an iterative loop designed to guarantee analytical rigor and correctness:

1. Planning: A Planning Agent parses the high-level user goal alongside the structured semantic map. It outputs a structured `ActionPlan` containing a sequential plan of target queries.

2. Execution: The loop runner triggers specialized tools:

• doc_context_retrieval: Searches the ingested guidelines in LLM-Wiki using self-contained BM25 text ranking and returns merged text segments.
• tabular_data_retrieval: Translates business goals into SQL queries using the semantic map. If the SQL fails due to a binder or syntax error, the execution agent intercepts the exception and prompts the model to auto-correct the query, returning the repaired result.

3. Analysis: The Analyst Agent combines the qualitative guidelines with the quantitative query results to draft a detailed analysis report.

4. Evaluation: The Evaluator Agent scores the draft report against the original objective (0 to 100) and lists remaining gaps.

5. Loop Routing: If the evaluator’s confidence is ≥ 80, the loop terminates. If not, the feedback is routed back to the Planner, which compiles a revised, more targeted action plan for the next iteration. If the confidence score does not increase or reach the max iteration limit, it will also stop.

6. Report Generation: The final report and the execution step trace are written to report.md.

Case Study: Auditing High-Risk Lending Records

The system was evaluated against a Public (Kaggle) LendingClub dataset and public guidelines on best practice. The user objective was to identify high-risk loans violating guidelines in the database. Here is a partial example in the demo. Below quotes the summary of the report.md generated by the agentic workflow.

## Analysis of Top 10% Highest-Risk Loans

### Current Evidence Summary

The context provides several data points, but there are **two different metrics** being referenced:

**From the detailed context (primary evidence):**
- **High-Interest Filter** (rate >= 18.84% / 0.1884): 103 total loans
  - Avg FICO: 678.84
  - Avg Rate: 19.79%
  - Avg DTI: 12.56%
  - 63 loans (61.2%) classified as high-risk

- **High-Risk Borrowers** (FICO ≤ 650, DTI ≥ 20%, Rate ≥ 16%): 31 records (0.32% of portfolio)

**From the "Summary" section (conflicting claim):**
- Claims "958 loans" in top 10% high-risk segment (10% of 9,578)

### Critical Assessment

The context's detailed analysis sections (High-Interest Loan Filter Impact, High-Risk Borrowers) are **more reliable** than the summary. The 103 loans with rates >= 18.84% represent the highest-risk segment based on the data architecture.

### Top 10% Highest-Risk Loans Identified

**Segment: Loans with Interest Rate >= 18.84% (0.1884)**
- **Count**: 103 loans (approximately 1.07% of portfolio, not 10%)
- **Key Risk Characteristics**:
  | Metric | Value | Risk Level |
  |--------|-------|------------|
  | Avg FICO | 678.84 | Fair (below optimal 700+) |
  | Avg Interest Rate | 19.79% | Very High |
  | Avg DTI | 12.56% | Moderate |
  | High-Risk Classification | 63 loans (61.2%) | Critical |
  | Highest Rate | 21.64% | Maximum |

**High-Risk Classification Criteria** (63 loans):
- Delinquency (2yrs) > 0 OR
- Inquiries (6mths) > 3 OR
- Public Records > 0

**Risk Correlations Observed**:
1. High interest rates strongly correlate with risky credit behavior (delinquencies, inquiries, public records)
2. Average FICO (678.84) places borrowers in 'fair' credit range
3. Moderate DTI (12.56%) suggests interest pricing driven by credit history rather than current debt burden
4. Concentrated high-risk pool with elevated default probability

Key Strengths & Conclusion

Our Agentic Insight Engine showcases the power of structured-unstructured hybrid analysis. The core strengths of this approach include:

• Semantic Data Alignment: The ontology and mapping.yamlallow the agent to understand what database columns mean, translating business terms to database queries.
• Iterative Self-Correction: The tabular retrieval tool automatically detects and repairs runtime SQL database syntax errors.
• Rigorous QA Guardrails: The independent evaluator agent prevents hallucinations by checking the analyst’s draft against the original goals. If confidence is low, it loops back to collect more information.

By wrapping structured databases and unstructured policy libraries inside an agentic harness, the system changes corporate compliance from manual, spreadsheet-based auditing into a reliable, autonomous research workflow.

Resources:

• Github: https://github.com/iwasnothing/agentic-insight
• Demo Youtube: https://youtu.be/isdHF8SeS9s?si=_BucvKKz8-MWIjRC

Ship Traffic (AIS)

Live vessel positions, historical tracks and area traffic snapshots from BarentsWatch AIS.

• find_vessels_nearby: Find the vessels currently nearest to a point - "what ships are off Bergen right now". Give a Norwegian place name OR a lat/lon coordinate.
• get_vessel_position: Get one vessel's latest position with a human-readable location ("3.2 km NW of Kristiansund"), speed, course and navigational status.
• get_vessel_track: Get a vessel's movement over a time window as a summary + simplified track - "where has KV Svalbard been the last 24 hours".
• get_area_traffic: Snapshot of ship traffic in an area - "how busy is Oslofjorden right now". Give a Norwegian place name OR a lat/lon coordinate.
• lookup_vessel: Resolve vessel identity: name <-> MMSI <-> IMO <-> callsign, plus flag state, type and dimensions. Use before positional queries to disambiguate - a name fragment returns up to 30 candidates.

Artskart

Search species and query verified occurrence records from Artsdatabanken's Artskart.

• search_species: Search for a species/taxon by Norwegian popular name or scientific name.
• get_species: Get full taxonomic detail for one species/taxon by id.
• get_species_children: Get the child taxa of a taxon, e.g. the species within a genus or the genera within a family.
• find_observations: Search verified species observation records.
• list_reference_codes: List valid codes for a filter field used by search_species/find_observations.

Avinor Flights

Real-time arrivals and departures for Norway's 43 Avinor airports, from Avinor's open flight data feed.

• get_flights: Flights to/from one airport, with schedule and live status from Avinor.
• lookup_airport: Find an airport's IATA code by name, or its name by code.
• lookup_airline: Find an airline's IATA code by name, or its name by code.

Data.norge.no

Search and query Norway's national data catalog via its open SPARQL endpoint — ~9,000 datasets, APIs and concepts.

• sparql_query: Run a SPARQL query against the data.norge.no national data catalog.
• search_datasets: Search the national data catalog for datasets by words in their title or description.
• search_resources: Search the other catalog resource types by words in title/description: data services (APIs), public services or information models.
• search_concepts: Search begreper (concepts/terms) in the national data catalog by words in their preferred label.

ENTSO-E Transparency

Day-ahead and imbalance prices, generation mix and forecasts, load, capacity, cross-border flow and outages for ~25 European bidding zones, from ENTSO-E's Transparency Platform.

• list_zones: List the bidding zone codes accepted by the other tools, mapped to display names. Covers all Nordic and Baltic zones plus the major continental European zones. GB is not available: ENTSO-E has had no UK data since Brexit.
• get_day_ahead_prices: Get one day's day-ahead spot prices (EUR/MWh) for a European bidding zone.
• get_imbalance_prices: Get one day's imbalance (balancing) settlement prices (EUR/MWh) for a European bidding zone - what a balance-responsible party pays or is paid per MWh of deviation from its schedule.
• get_generation_mix: Get one day's actual generation (MW) broken down by production type for a European bidding zone.
• get_generation_forecast: Get one day's day-ahead wind and solar generation forecast (MW) for a European bidding zone.
• get_installed_capacity: Get installed generation capacity (MW) by production type for a whole year, for a European bidding zone.
• get_load: Get one day's system-wide electricity load (MW) for a European bidding zone.
• get_cross_border_flow: Get one day's metered physical power flow (MW) from one bidding zone to another.
• get_generation_outages: Get planned and forced (unplanned) outages of generation units in a European bidding zone over a date range - the single best explanation for an unexpected price spike (a plant going down).
• get_transmission_outages: Get planned and forced (unplanned) outages of interconnectors/transmission assets between two European bidding zones over a date range - explains sudden drops in cross-border transfer capacity.

Entur

Journey planning and live departures across all of Norway (Journey Planner v3 + Geocoder).

• find_stop_places: Resolve a place name to Entur StopPlace ids and coordinates.
• get_departures: Get the live departure board for a StopPlace.
• plan_journey: Plan a public-transport journey between two points.

FHI Statistikk

Public-health statistics from FHI's statistics banks — causes of death, births, infectious diseases, drug use, vaccinations and more.

• list_sources: List the statistics sources (registers) in FHI's statistics banks.
• search_tables: Find tables by keyword in their titles, across all sources or one.
• get_table_dimensions: Get a table's dimensions and the valid value codes for each.
• get_table_data: Query a table's data and return it as labeled rows.
• get_table_description: Get a table's descriptive metadata: what the register covers, how the figures are produced, and caveats for interpreting them.

Company Facts

Companies, people, shareholders, roles, grants and financials from Norwegian public registries.

• organisasjonsnummer_for_selskap: Returns the organisasjonsnummer for a company or multiple companies if there is ambiguity. The method filters and tries to match company names based on the input.
• personinformasjon: Retrieves information about a person based on their name. The method filters and tries to match person names based on the input.
• selskapsdetaljer: Retrieves detailed information for a company by its organization number.
• aksjeeiere_for_selskap: Retrieves a list of shareholders for a given company organization number.
• selskapets_aksjeposter: Retrieves a list of stocks owned by a given company (i.e., where the company is a shareholder).
• aksjeposter_for_person: Retrieves stock holdings for an individual based on their name and birth year.
• roller_i_enhet: Retrieves roles for a given organization number from Brønnøysund Registeret. Contains information about the board members, the CEO, accountant and other roles.
• underenheter_for_selskap: Retrieves "underenheter" (sub-entities/branches) for a given parent organization number.
• tildelinger_for_selskap: Retrieves "tildelinger" (allocations/grants) for a given organization number.
• eu_tildelinger_for_selskap: Retrieves EU grants awarded to a specific organization number.
• forskningsradet_tildelinger_for_selskap: Retrieves grants from "Forskningsrådet" (The Research Council of Norway) for a specific organization number.
• skattefunnprosjekter_for_selskap: Retrieves SkatteFUNN projects for a given organization number.
• stillinger_for_selskap: Retrieves active job listings from NAV (arbeidsplassen.no) for a given employer organization number.
• list_naeringskoder: List Norwegian business activity codes (naeringskoder) present in the database.
• finn_selskaper: Search for companies by location (kommune or fylke), optionally filtered by business activity codes, company age and employee count.
• get_company_last_financial_statement: The open section contains key figures from the most recently submitted annual financial statements, while the closed section contains almost all figures from the past three annual financial statements, including those from the consolidated accounts.
• get_company_ip_rights: This dataset contains information about registered rights and applications submitted in Norway. The dataset provides detailed information about patented inventions, trademarks and designs. This includes applicant names, important ID numbers, application date, status and relevant events.

Geonorge

Kartverket's open address and place name APIs — text search and reverse geocoding, WGS84 coordinates.

• search_address: Search Norwegian addresses by free text.
• reverse_geocode: Find the addresses nearest to a coordinate (reverse geocoding).
• search_place_name: Search Norwegian place names (stedsnavn) by free text.
• find_places_near_point: Find the place names nearest to a coordinate (reverse geocoding for stedsnavn - use reverse_geocode instead for street addresses).
• list_place_types: List valid navneobjekttype values (place types, e.g. "By", "Fjell", "Øy", "Innsjø", "Kirke", "Gard") for filtering search_place_name and reading the `type` field on its results.

Lovdata

Norwegian laws and regulations in force from Lovdata's open data, plus Supreme Court decisions from domstol.no.

• search_rules: Search Norwegian laws and central regulations currently in force.
• get_rule: Get one law or regulation: metadata plus its list of sections.
• get_section: Get the text of a single section (paragraf) of a law or regulation.
• search_case_law: Free-text search in Norwegian Supreme Court (Høyesterett) decisions.
• list_case_law: List Norwegian Supreme Court (Høyesterett) decisions for one year.
• get_case_law: Get one Høyesterett decision: summary, metadata and PDF link.

Air Quality

Air quality from the official Luftkvalitet i Norge network — measurements and forecasts.

• find_air_stations: List Norway's air quality measuring stations with their latest overall AQI level and which components (PM2.5, PM10, NO2, O3, SO2) each measures.
• get_air_quality_nearby: Get the measuring stations nearest a coordinate with their latest measured AQI level per component. Sorted by distance.
• get_station_latest: Get the latest measured values at one station: newest hourly value and running 24h mean per component, in ug/m3, with AQI level for each.
• get_station_history: Get the measured hourly time series for one station, per component (ug/m3). Feed straight into a chart.
• get_air_quality_forecast: Get the modelled hourly air quality forecast (~2 days ahead) for any coordinate in Norway: overall AQI index plus PM2.5, PM10, NO2 and O3 concentrations (ug/m3).

Marine

Aquaculture register (Fiskeridirektoratet) and salmon fish-health data (BarentsWatch).

• find_aquaculture_sites: Search the official Norwegian aquaculture register (Fiskeridirektoratet) for fish-farming sites.
• get_aquaculture_site: Get one aquaculture site from the register by its site number (same number as the fish-health locality number).
• get_lice_report: Get the weekly salmon-lice report for Norwegian aquaculture localities: farm-reported adult female lice per fish, fallow status, and PD/ILA disease flags.
• get_locality_status: Get the full weekly fish-health status for one aquaculture locality: lice counts (adult female / mobile / stationary), sea temperature, treatments that week, and any ILA/PD disease zones it sits in.
• get_lice_history: Get the weekly adult-female-lice series for one locality for a year. Feed straight into a chart against the 0.5 (0.2 in spring) limit.
• get_sea_temperature: Get the weekly farm-reported sea temperature at one locality for a year.
• get_lice_treatments: Get all lice treatments at one locality for a year: medicinal (bath / in-feed), non-medicinal, cleaner fish and mechanical removal, per week.

Food Safety Smileys

Hygiene inspection results ("Smilefjes") for Norwegian restaurants and cafes, from Mattilsynet.

• search_establishments: Search food-serving establishments by their most recent Smilefjes grade.
• get_inspection_history: Get every Smilefjes inspection recorded for one establishment, oldest first.
• get_requirement_points: Get the detailed requirement-point (kravpunkt) grades for one inspection.

National Library

Search and read OCR text from the National Library's catalog — books, newspapers, journals, photos, maps, music and more.

• search: Search the National Library's catalog - books, newspapers, journals, photos, maps, music, manuscripts, radio/TV and more, both physical and digitized. Digitized text (OCR) is matched too by default, so a query can surface newspaper or book pages that mention a term even when it's not in the title.
• get_item: Get full catalog metadata for one item.
• get_related_items: Get items related to one catalog item - e.g. the newspaper/journal a digitized article appeared in (hosts), articles/pages it contains (constituents), the previous/next issue in a run, or the series it belongs to. Most items (e.g. standalone books) have no recorded relations - an empty result is normal, not an error.
• search_within_item: Full-text search inside the OCR'd pages of one specific digitized item (a book, one newspaper issue, a journal issue, etc.), returning which pages contain the term. Use this to jump straight to relevant pages of a long item instead of reading it page by page.
• get_aggregations: Get facet counts for a query - e.g. how many matching items exist per year, county, media type or language. Useful for a quick overview before drilling into search() results, or for questions like "how many newspaper items mention X, broken down by year".

Norges Bank

Exchange rates, the policy rate, government bond yields and the Regional Network business survey from Norges Bank's open data API.

• list_datasets: The Norges Bank dataflows most worth querying, out of ~20 published.
• get_exchange_rate: Daily NOK exchange rate for one foreign currency.
• get_policy_rate: Norges Bank's key central bank rates, daily.
• get_short_term_rate: Norwegian short-term money market rates (Nowa), daily.
• get_government_bond_yield: Norwegian government bond zero-coupon yield, daily.
• get_regional_network_survey: Norges Bank's quarterly Regional Network business survey.
• get_dataset_dimensions: Get a dataflow's dimensions and their valid codes, in key order.
• get_series: Fetch one or more time series from any Norges Bank dataflow.

NVE

Reservoir fill levels, hydrological observations and modelled snow/inflow from NVE.

• get_reservoir_levels: Get the latest reservoir fill level for Norway as a whole and for each elspot price area (NO1 to NO5).
• get_reservoir_history: Get weekly reservoir fill history for one area.
• get_reservoir_min_max_median: Historical min / max / median fill levels per week for one area, computed by NVE over the reference period. Perfect as the shaded background band in a reservoir chart.
• list_hydro_parameters: List the hydrological parameters HydAPI can return (water flow, level, temperature, precipitation, etc.) with their numeric ids and units.
• find_hydro_stations: Find NVE hydrological measuring stations and see which time series (parameters + resolutions) each one offers.
• get_hydro_observations: Get a hydrological time series for a station.
• get_hydro_percentiles: Day-of-year percentile bands (the seasonal "normal") for a station's parameter, computed by NVE over the station's full record. Use them as the reference band to judge whether an observed value is high or low for the time of year, e.g. is inflow above or below normal right now.
• get_snow_and_inflow: Modelled snow reservoir and inflow at a point from NVE's seNorge grid model (1x1 km, all of Norway; no API key). These are the hydrological fundamentals behind hydropower value in a catchment.

Police Log

The Norwegian Police public operational log — incidents, traffic, order and more.

• search_messages: Search the Norwegian police log.
• search_by_location: Find police log messages for a place name, whether the API classifies it as a municipality or as a sub-locality `area` (e.g. "Bragernes", "Sjøvegan") -- the API only supports server-side filtering by municipality, not by area.
• get_message: Get a single police log message by its id (e.g. "26k3hb-2").
• list_categories: List the message categories, with their names and ids.

Public Mail Journals

Aggregated Norwegian municipal and state mail journals (norske-postlister.no)

• search: Search Norwegian public mail journals (postlister).
• search_kommune: Search one municipality's (or agency's) mail journal, optionally by time range.
• get_case: List all documents belonging to one case (sak).

SSB Statbank

Statistics Norway's Statbank — ~7,500 tables via PxWebApi 2, flattened to labeled rows.

• search_tables: Free-text search across the SSB Statbank table catalog.
• get_table_metadata: Get a table's dimensions and the valid codes/labels for each.
• get_table_data: Query a table's data and return it as labeled rows.

Statnett Grid Operations

Live power flow, production/consumption and power-situation status from Statnett, the Norwegian TSO.

• get_physical_flow: Get the current physical power flow (MW) on every interconnector between Nordic/Baltic price areas and countries, live from Statnett.
• get_production_consumption: Get the current Nordic/Baltic electricity production (by source), consumption and net exchange per country, live from Statnett.
• get_power_situation: Statnett's current power-situation assessment for Norway's price areas (NO1-NO5): whether the grid capacity situation is normal or strained for the season.

Stortinget

Norwegian parliament open data — representatives, parties, cases and votes, plus election results from valgresultat.no.

• list_sessions: List parliamentary sessions (sesjoner, e.g. "2025-2026"), from 1986-87. Session ids are the `sesjonid` parameter for cases, votes and parties.
• list_periods: List election periods (stortingsperioder, e.g. "2025-2029"). Period ids are the `stortingsperiodeid` parameter for representatives.
• get_current_representatives: Today's sitting representatives (name, party, county, id, email).
• get_representatives: Get the representatives elected for a given period.
• get_current_government: The sitting government's members (regjeringsmedlemmer).
• get_parties: Get the parties represented in parliament.
• search_cases: List the cases (saker) handled in a session.
• get_case: Get the full detail for one case (sak): recommendation text, committee, short decision, references and topics.
• get_votes_for_case: List the votes (voteringer) held on a case, with for/against/absent counts. Use a vote's `votering_id` with get_vote_result() for the per-representative breakdown.
• get_vote_result: Get the per-representative result of a single vote (voteringsresultat): how each representative voted.
• get_election_result: Norwegian election results from valgresultat.no (Valgdirektoratet): per-party votes, percentages, change vs last election and seats, plus turnout. Omit fylke_nr for the national result; the response's `underomrader` lists the area numbers one level down.

Electricity Prices

Hourly electricity spot prices per elspot area (NO1–NO5) from hvakosterstrommen.no.

• get_prices: Get one day's hourly spot prices for one price area.
• get_price_now: Get the spot price for the current hour in a price area.
• get_price_range: Get hourly prices across a date range for one price area (max 31 days).
• get_cheapest_hours: Rank a day's hours from cheapest and return the n cheapest.
• find_area: Look up which price area (NO1..NO5) a coordinate falls in.

Statens Vegvesen

Traffic counts and the national road database (NVDB) from Statens Vegvesen.

• find_traffic_points: Find traffic registration points (trafikkregistreringspunkt) - the physical counting stations on Norwegian roads that this server's other traffic tools read from.
• get_traffic_volume: Get recent hourly traffic volume for a registration point - the closest thing to real-time traffic counts Vegvesen publishes (data typically lags an hour or so behind the clock).
• get_average_daily_traffic: Get yearly average daily traffic (AADT / "ÅDT") for a registration point - the standard measure of how busy a road is, averaged over a full year to smooth out seasonal and day-of-week swings.
• list_road_object_types: List NVDB road object types (vegobjekttyper) - the ~400 categories of things recorded against the road network, e.g. speed limits, tunnels, bridges, guardrails, traffic signs.
• search_road_objects: Search NVDB for road objects of one type, e.g. all speed limit signs in a county, or all tunnels on a given road.
• get_road_at_point: Find the road network reference nearest a coordinate - the road-network equivalent of reverse geocoding an address.
• get_traffic_messages: Get traffic messages (veimeldinger) from Vegvesen's DATEX II feed - roadworks, road closures, incidents, ferry disruptions, reduced speed and lane restrictions on Norwegian roads. Newest first.

Weather

Current conditions and forecasts from MET Norway, historical data via Open-Meteo, plus tides and sea level from Kartverket.

• get_current_weather: Get current weather conditions for a coordinate in or near Norway.
• get_forecast: Get the upcoming hourly weather forecast for a coordinate in or near Norway.
• get_historical_weather: Get daily historical weather for a coordinate.
• get_tide_forecast: Get the tide and sea-level forecast ("havnivå") for a coordinate on or near the Norwegian coast, from Kartverket.
• get_tide_table: Get the tide table ("tidevannstabell") - high and low water times - for a coordinate on or near the Norwegian coast, from Kartverket.
• get_sea_level_extremes: Reference sea levels for a coordinate on or near the Norwegian coast: astronomical tide extremes and statistical storm-surge return levels, from Kartverket.

You design it. Then what? A clear map of the Figma-to-code AI mess

A beginner’s guide through the AI pipeline fog.

If you put a room of designers together and asked how their work actually gets from Figma into real code with AI, you would get a few seconds of silence and then a lot of different answers, none of them confident. That is not because you missed a memo. There simply is no memo yet. So let me walk through it in plain words.

The demos are real. Your confusion is also real.

You have seen the videos. Someone connects Figma to an AI tool, types a sentence, and a working screen appears in seconds. It looks like the whole problem is solved. Then you try it on your own real project, and it falls apart, and you assume you did something wrong.

You did not. The demos show you one clean layer working under perfect conditions. Your actual work needs three or four layers stacked together, and nobody shows you the stack, because the stack is where it gets messy and half-solved. The confusion comes from not knowing they are separate things at different stages that need different skills.

The design-to-code stack, one layer at a time

Layer 1: The MCP. The connection.

MCP is one of those new words flying around, so let’s first understand what it does and does not do.

Think of an AI assistant like Claude as very smart, but sitting in a sealed room. It can reason beautifully, but on its own, it cannot see your files, your Figma, your tools. It only knows what you paste into the chat.

An MCP is how you open a door in that room. The name stands for Model Context Protocol, but forget the words. The useful picture is USB-C. USB-C is one standard plug that connects your laptop to almost anything: a screen, a drive, a camera, so you do not need a different cable for each anymore. An MCP is the same idea for AI: one standard way to plug a tool into Claude. Plug in the Figma one, and Claude can look inside Figma. Plug in others, and it can reach your other tools the same way.

MCP is only the wiring, and Claude is the mind. And it is not plugged into one tool at a time; it sits in the middle like a hub, connected to all of them at once, so for example, it can pull your notes from Notion and read your design in Figma in the same moment, work out what matters, and turn it into code. The wiring just carries the signal. Claude is the part that understands what it is looking at.

So in real terms: without it, you take a screenshot of your design, paste it into the chat, and hope Claude understands the picture. With the Figma MCP connected, Claude reaches into the actual file and reads the real thing: your exact colours, your spacing, your fonts, how the layout is stacked. Not a photo of your design, but the design and structure itself.

What MCP fixes: Claude can finally see your work, details and structure properly in its context, instead of squinting at an image you paste.

Where does it come from? How to access it?
You don’t make an MCP; you connect to one that the tool’s makers have already built. For Figma, there are three options: you can switch them on via connectors (in Claude, the little plus button in the chat window) or install them, depending on which you pick.

The gap it leaves:
it reads your values and your structure, not your code. It sees a button and copies the colour and the corners, but it has no idea your team already has a real, reusable button sitting in the codebase (or not… or if you have a codebase at all). So it quietly builds a fresh lookalike every time. Your colours, yes. Your real component, no. This is fine for a quick website, but not enough for a scaling product.

Layer 2: The markdown (md) files. The memory.

The pipe lets Claude see your design, but it cannot see what is in your head. Your rules, the choices you would never write on the canvas but always make anyway.

A markdown file is where you put all the rules and reasons of your design. It is just a plain text note you write once and let Claude read, and it is the one place your reasoning actually survives, because your Figma file holds what things look like and your code holds what they do, but only this note holds why. Think of it like the handover note you would leave a freelancer joining your project.

It says things like here are my breakpoints, my colours and all you need to know. Here are the rules and reasons I always follow. “Use the quiet button for secondary actions.” “Never two loud buttons on one screen.” Claude, reading your Figma file, has no way to know any of that. This note hands it over.

What is a markdown file? Where is it and who makes it?
Markdown (md) is just a plain text file, saved with a .md ending, the same way a document is saved as .pdf, with a little light formatting like a README uses, and its charm is that humans and machines both read it with no translation in between. You rarely write it from scratch: you let the MCP read your Figma file, have Claude draft a first version from your colours, spacing and components, then refine it in the chat or by hand.

Where it lives depends on how you work. At its simplest, it is just a file sitting in your project folder. If you work with Claude, that file is called CLAUDE.md, and Claude reads it automatically every time, so you never have to re-explain yourself.

What it fixes:
Claude now works with your judgment, not just your pixels.

The gap it leaves:
A markdown file is guidance Claude reads, not a rule that runs, so it can only ever shift the odds in your favour, never hold the line. This is because Claude reads your rule and usually follows it, but can still interpret it loosely or quietly drop it when a screen gets complicated. Also, the note gets weaker the bigger it grows, since ten sharp rules get followed, but a sprawling rulebook starts competing with itself. On top of that, it goes out of date the moment you change something and forget to update it, so a note nobody keeps current keeps handing Claude yesterday’s directions with full confidence.

Layer 3: The skill. The recipe.

A skill is one step up from a note. A note tells Claude what to know.

A skill tells the AI how to do a specific job, the same way every time, like a fixed rule to enforce.

Picture the difference between explaining a dish out loud every night and writing the recipe card once, with steps and measurements. A skill is a small folder holding those steps, and sometimes a bit of actual code, that Claude picks up only when that job comes around. Instead of re-explaining “this is how we build our components” each time, the skill holds the method and runs it consistently, but only when it is needed.

What is a skill? Where does it live, and who makes it?
A skill is just a folder with a plain instruction file inside (a SKILL.md, markdown again) plus any extra bits it needs, like a script or a template. It stays out of the way until a task matches its description, then it wakes up.

What it fixes:
The tasks you do over and over no longer come out slightly different every time.

The gap it leaves:
It is still something you write and keep up to date, and it still does not tie itself to your design. It is a very good recipe. It is not a wire connecting two rooms.

Layer 4: The mapping. The real wire.

Everything so far still leaves your design and your code as two separate worlds. This is the layer that finally joins them, and it is also the one that needs developers.

The mapping is the wire between those two libraries. Its Figma name is Code Connect, but the picture is what matters: you formally tell the system “this button in Figma is that exact button in the code, and here is where it lives.” Once the wire is in place, Claude stops guessing and stops building lookalikes. It points at your real component, the same one already running in your product.

What it fixes:
The drift. Design and code stop being two separate snapshots and start pointing at one shared source of truth.

The gap it leaves:
It is real work, and it is not designer work. It is written in code, maintained by developers, checked whenever something changes, and wired into the way the product gets built. It has an owner and a running cost. Build it without a team to keep it alive, and it rots.

The honest shape of it

Look at the whole stack. Each layer covers the hole under it. The pipe lets Claude see your design. The note carries your rules. The recipe keeps your repeated jobs consistent. And the mapping, the top layer, is the only one that truly welds design and code together so they never drift.

But that top layer needs a codebase, developers, and constant upkeep. Most people do not have that, and should not pretend to. So for almost everyone, the design and the code will drift the moment either side changes, and that is not a sign you set it up wrong. It is simply what these tools are without the expensive wire: they take a snapshot and build from it.

I am told to just skip Figma and vibe code

You will also meet tools that seem to skip this whole stack: you describe a screen or drop in a design, and a working app appears. They are not cheating. But the same limits apply underneath: they bundle the pipe, and maybe a little of the note, but they skip the mapping entirely and keep no memory of your system, so they build a fresh lookalike, not your real component, and they drift the same way.

So, what should you actually use and do as a Designer in Figma?

Two things, in order. First, get your file clean. Then, match the stack to your size. That is the whole plan.

First: clean files before any layer.

All four layers can only work with what is actually in your Figma file. If your colours are loose hex codes instead of styles or variables, if your spacing is eyeballed instead of auto-layout, if your components are not really components set up in auto layout, then the pipe has nothing clean to read, and the result comes back messy.

Then, match the stack to your size.

With your file clean, do not overengineer the rest. Add layers only as you grow.

If it is just you

You are solo, maybe a quick site or a product still finding its shape. You need two layers: the pipe and a note. Connect the Figma MCP so Claude builds in your real colours and spacing, and keep one short markdown note listing your reusable pieces and when to use each, pasted in every time. Skip skills until you catch yourself doing the exact same job every day. Forget the mapping completely, you have no codebase to wire it to and no developer to keep it alive.

If you are a small team

A designer or two, a developer or two, and now things go out of sync between people, not just tools. You need three layers: the pipe, a shared note everyone reads, and maybe one skill for the thing you build over and over. But the layer that actually holds you together is not a tool at all, it is a human habit: whoever changes a component in the code updates the shared note in the same breath.

If you own a (serious) design system

You own the components that a whole company builds on, and you cannot eyeball your way out of drift. This is the one tier where all four layers earn their cost, including the mapping. Wire Code Connect or similar to your real components in Storybook or whatever you use, and add an automatic check that shouts the moment design and code fall out of line.

The one thing that stays yours

Whichever size you are, one part never moves down the stack. Knowing which of the ten versions Claude just made is the good one. Which spacing feels right, which button earns the attention, and when to stop. The layers move your work from Figma toward code. That is still you, and that was always the actual job.

Rethinking Figma in an AI world

As AI pulls product development closer to code, Config 2026 reveals Figma’s high-stakes gamble to survive an era of agentic workflows in a code-native environment

For more than a decade, Figma defined modern product design. It made collaboration effortless, turned design into a shared language, and became the default workspace for teams building software. But as AI reshapes how products are planned, prototyped, and shipped, the central question is the extent of Figma’s usefulness. Or rather, will the canvas remain at the centre of gravity, if at all?

Unlike the previous conference, Config 2026 makes that question more urgent. Figma no longer just defends a collaborative design canvas; it is actively expanding that canvas into code, motion, shaders, and agent-driven workflows. That shift reveals how much pressure the old model is under.

The Config Moment

Config 2026 arrives at a meaningful turning point. Figma is still the place where many product teams begin, refine, and communicate ideas, yet the broader workflow around it is changing fast. This is because AI is pulling more creative and production work into code editors, agentic environments, and structured workflows that remove the need for a designer to drag pixels around a canvas.

What changed at Config 2026 is that Figma is now responding to that pressure directly. Code layers bring code onto the canvas, Figma Motion brings animation and timing into the file itself, shader tools add more expressive visual effects, and the agent becomes more useful across connected workflows. This brings us back to the canvas being more than a surface. It was Figma’s business model for its collaboration engine and its defensive moat against command-line interfaces through seats.

The Seat Model Gets Pressured

Figma’s enterprise strength has always depended on breadth. Designers used it first, then product managers, engineers, marketers, writers, and executives followed. The more people needed to review, comment, inspect, or reference design work, the more seats Figma could sell. The same applies for the product bench, with the rapid expansion of Slides, Buzz, Sites, and now Motion and Weave.

AI complicates that logic. If an engineer can generate or inspect UI directly inside a coding environment, or if a product team can translate structured design intent into working software without opening a shared canvas or applications, the need for passive, underutilised seats weakens. Config 2026 actually confirms this tension by bringing code layers and agent workflows closer to the main product, Figma Design. In other words, Figma is acknowledging that an increasing number of people want to participate in product and code creation without relying on traditional file-based design behaviour.

That does not kill Figma’s seat model overnight, but it does chip away at the assumption that every stakeholder must enter Figma to participate or experience design. Thus, the deeper issue goes beyond whether the team has seats at the table (canvas). The table of collaboration resides in a code-native environment and workflow, where the moment of truth shifts closer to implementation, and that reduces the number of times a team needs to return to a design file as the source of record.

Figma’s Fightback

Figma is not standing still. Even before Config 2026, the company pushed beyond static collaboration and into AI-connected workflows. Features like MCP, Code Connect, and Figma Make on local code point toward a future where design data can move more fluidly into development environments and AI tools.

That is a smarter response than pretending the canvas can win by becoming a slightly better canvas. Figma’s best path is to become the system that preserves design intent, component logic, and implementation alignment across tools. In that model, Figma is becoming an operating layer.

This is also why the company’s product strategy feels more important than its individual features. A single new AI button will not change the story. What matters is whether Figma can make design data more portable, more structured, and more useful outside the file itself. Config 2026 suggests that Figma understands this, but it also shows how hard the transition is: the company still has to make the canvas relevant in a world where many teams want to begin and finish elsewhere.

Anthropic’s Advantage

While Figma is defending the canvas, companies like Anthropic are helping normalise a different workflow entirely. Claude Code represents a real threat to Figma’s design dominance via a more agentic way of building software. A solution that operates across codebases, edits multiple files, and helps teams ship working products faster.

That matters because it shifts expectations. Once a team gets used to prompting real software into existence, a static mockup can feel like an intermediate step rather than a necessary one. Claude’s prototypes are now giving the vibes of an actual product. Some even claim that it is the product, or at least close enough to change how teams think.

Therefore, the strongest threat to Figma is a workflow that makes drawing less central. Even with Config 2026’s new materials and code-aware features, the larger trend remains the same: if teams can skip past the canvas by accelerating from intent to implementation with less translation, the value of designing to handoff vanishes.

The End of Proprietary Gravity

Design teams have seen this movie before. Photoshop gave way to Sketch, Sketch gave way to Figma, and each shift brought a new file format, a new workflow, and a new vendor promise. Every migration taught the same lesson: proprietary tools are powerful until the market starts demanding portability.

That lesson is now becoming more relevant than ever. The future of design is likely to be built around reusable tokens, readable structures, and tool-agnostic metadata rather than locked files that only live well inside one platform. The winning systems will be the ones that can travel between editors, browsers, codebases, and AI agents without losing meaning.

So while we see Config 2026 pushing in that direction, with motion exporting in CSS, JSON, React, MP4, WebM, animated SVG and GIF formats, we can also admit that the .figma canvas may stop being the canonical source of truth. In an AI-driven workflow, the real asset is not the visual file but the structured reusable intent behind it.

The Real Test

Even in a changing market, Figma has real strengths. Its collaboration culture is deeply embedded in how teams work, its ecosystem is mature, and its product remains an efficient place to align complex groups around design decisions. Those advantages should not be underestimated.

But strength can become inertia if the company mistakes habit for permanence. Config 2026 shows a company trying to turn its strengths into a new platform layer rather than a static product category. That is the right instinct over previous runs. It is also a sign of how much the market has shifted, where Config announcements avoid the theoretical, cringy concept like automation, generation, and iteration (AGI) back in 2024. Despite the design industry still cheering and celebrating with Figma, serious designers are now asking a more uncomfortable question: what if the most valuable part of design isn’t the place where we draw it but the way it survives across the product development process?

That is the challenge Figma now faces. Not how to make the canvas prettier, but how to make design survive after the canvas stops being central.

The verdict will not come from conference applause or feature demos. It will come after the conference hype from workflow behaviour. If teams continue to spend their most important time inside Figma, the canvas remains dominant. If they begin spending more time in code editors, AI agents, and structured design systems that travel cleanly between tools, Figma becomes one platform among many.

Config 2026 makes this test more interesting, not less. Figma is clearly betting that it can keep the canvas relevant by making it more powerful, more expressive, and more connected to code and AI. That is a serious fightback strategy, and it confirms the scale of the challenge. The company is done competing with other design tools. It is now competing with an entirely different way of building products.

That is why this moment feels bigger than one (or multiple) product launch. It is a test of whether product design remains a canvas-first discipline or becomes an architecture of intent that only sometimes needs a canvas at all.

Could we experience another Config blockbuster next year? Unlike other years, Figma will confront a definition change in design. And in the AI era, that may be the ultimate consequential hurdle to overcome.

Give AI-built websites a real design with DESIGN.md

Give your coding agent a reusable design reference: colors, type, spacing, components, and the reasoning behind them.

So every new page follows a specific visual language, not the same generic AI layout.

Use a DESIGN.md to:

• Style your site without being a designer
• Match a style you like from any reference site
• Keep new pages in the same visual language
• Restyle existing pages without starting over
• Give your AI coder a reusable design brief

Style without design skills

Pick a DESIGN.md from a real site and hand it to your AI coder. It already carries the colors, type, and spacing, so you don't need to know any of it.

Find Designs

Design Systems Analysis

• BMW M: Motorsport automotive. Pure black canvas, M tricolor stripe accents, full-bleed photography.
• Dell (1996): Catalog-era PC retail web. Black page frame, flat color-block ribbon cards, Helvetica-Black over Times Roman.
• HP: Consumer electronics catalog. White canvas with electric blue accent, angular chevron motifs.
• Nintendo (2001): Y2K "console chrome" web. Brushed-periwinkle metal panels, amber nav on a dotted carbon bar, outlined Arial-Black box-art type.
• Airbnb: Travel marketplace. Warm coral accent, photography-driven, rounded UI.
• Airtable: Spreadsheet-database hybrid. Colorful, friendly, structured data aesthetic.
• Apple: Consumer electronics. Premium white space, SF Pro, cinematic imagery.
• Binance: Crypto exchange. Bold yellow accent on monochrome, trading-floor urgency.
• BMW: Luxury automotive. Dark premium surfaces, precise German engineering aesthetic.
• Bugatti: Hypercar brand. Cinema-black canvas, monochrome austerity, monumental display type.
• Cal.com: Open-source scheduling. Clean neutral UI, developer-oriented simplicity.
• Claude: Anthropic's AI assistant. Warm terracotta accent, clean editorial layout.
• Clay: Creative agency. Organic shapes, soft gradients, art-directed layout.
• ClickHouse: Fast analytics database. Yellow-accented, technical documentation style.
• Cohere: Enterprise AI platform. Vibrant gradients, data-rich dashboard aesthetic.
• Coinbase: Crypto exchange. Clean blue identity, trust-focused, institutional feel.
• Composio: Tool integration platform. Modern dark with colorful integration icons.
• Cursor: AI-first code editor. Sleek dark interface, gradient accents.
• Discord: Team & community chat platform. Deep-indigo canvas with Blurple gradients, heavy all-caps display type.
• ElevenLabs: AI voice platform. Dark cinematic UI, audio-waveform aesthetics.
• Expo: React Native platform. Dark theme, tight letter-spacing, code-centric.
• Ferrari: Luxury automotive. Chiaroscuro editorial, Ferrari Red accents, cinematic black.
• Figma: Collaborative design tool. Vibrant multi-color, playful yet professional.
• Framer: Website builder. Bold black and blue, motion-first, design-forward.
• HashiCorp: Infrastructure automation. Enterprise-clean, black and white.
• IBM: Enterprise technology. Carbon design system, structured blue palette.
• Intercom: Customer messaging. Friendly blue palette, conversational UI patterns.
• Kraken: Crypto trading. Purple-accented dark UI, data-dense dashboards.
• Lamborghini: Supercar brand. True black surfaces, gold accents, dramatic uppercase typography.
• Linear: Project management. Ultra-minimal, precise, purple accent.
• Lovable: AI full-stack builder. Playful gradients, friendly dev aesthetic.
• Mastercard: Global payments network. Warm cream canvas, orbital pill shapes, editorial warmth.
• Meta: Tech retail store. Photography-first, binary light/dark surfaces, Meta Blue CTAs.
• MiniMax: AI model provider. Bold dark interface with neon accents.
• Mintlify: Documentation platform. Clean, green-accented, reading-optimized.
• Miro: Visual collaboration. Bright yellow accent, infinite canvas aesthetic.
• Mistral AI: Open-weight LLM provider. French-engineered minimalism, purple-toned.
• MongoDB: Document database. Green leaf branding, developer documentation focus.
• Nike: Athletic retail. Monochrome UI, massive uppercase type, full-bleed photography.
• Notion: All-in-one workspace. Warm minimalism, serif headings, soft surfaces.
• NVIDIA: GPU computing. Green-black energy, technical power aesthetic.
• Ollama: Run LLMs locally. Terminal-first, monochrome simplicity.
• OpenCode: AI coding platform. Developer-centric dark theme.
• Pinterest: Visual discovery. Red accent, masonry grid, image-first.
• PlayStation: Gaming console retail. Three-surface channel layout, quiet-authority display type, cyan hover-scale.
• PostHog: Product analytics. Playful hedgehog branding, developer-friendly dark UI.
• Raycast: Productivity launcher. Sleek dark chrome, vibrant gradient accents.
• Renault: French automotive. Vibrant aurora gradients, NouvelR typography, bold energy.
• Replicate: Run ML models via API. Clean white canvas, code-forward.
• Resend: Email API. Minimal dark theme, monospace accents.
• Revolut: Digital banking. Sleek dark interface, gradient cards, fintech precision.
• Runway: AI video generation. Cinematic dark UI, media-rich layout.
• Sanity: Headless CMS. Red accent, content-first editorial layout.
• Sentry: Error monitoring. Dark dashboard, data-dense, pink-purple accent.
• Shopify: E-commerce platform. Dark-first cinematic, neon green accent, ultra-light type.
• Slack: Workplace messaging brand. Deep-aubergine primary with cream-lavender hero gradients and pill CTAs.
• SpaceX: Space technology. Stark black and white, full-bleed imagery, futuristic.
• Spotify: Music streaming. Vibrant green on dark, bold type, album-art-driven.
• Starbucks: Global coffee retail brand. Four-tier green system, warm cream canvas, full-pill buttons.
• Stripe: Payment infrastructure. Signature purple gradients, weight-300 elegance.
• Supabase: Open-source Firebase alternative. Dark emerald theme, code-first.
• Superhuman: Fast email client. Premium dark UI, keyboard-first, purple glow.
• Tesla: Electric automotive. Radical subtraction, full-viewport photography, near-zero UI.
• The Verge: Tech editorial media. Acid-mint and ultraviolet accents, Manuka display, rave-flyer story tiles.
• Together AI: Open-source AI infrastructure. Technical, blueprint-style design.
• Uber: Mobility platform. Bold black and white, tight type, urban energy.
• Vercel: Frontend deployment. Black and white precision, Geist font.
• Vodafone: Global telecom brand. Monumental uppercase display, Vodafone Red chapter bands.
• VoltAgent: AI agent framework. Void-black canvas, emerald accent, terminal-native.
• Warp: Modern terminal. Dark IDE-like interface, block-based command UI.
• Webflow: Visual web builder. Blue-accented, polished marketing site aesthetic.
• WIRED: Tech magazine. Paper-white broadsheet density, custom serif display, mono kickers, ink-blue links.
• Wise: Money transfer. Bright green accent, friendly and clear.
• xAI: Elon Musk's AI lab. Stark monochrome, futuristic minimalism.
• Zapier: Automation platform. Warm orange, friendly illustration-driven.

AI has made it incredibly easy to go from an idea to something tangible. What used to take hours of wireframing, iteration, and prototyping can now happen in minutes. Tools like Figma Make have made it possible to explore multiple directions, generate polished UI, and even create working prototypes with very little effort.

What’s interesting is that while the amount of design output has increased dramatically, the quality of product thinking hasn’t necessarily kept pace. I’ve seen AI-generated concepts that look great on the surface but begin to fall apart once you start thinking about edge cases, technical constraints, or how they fit into the broader product experience. AI only knows what it’s given. It doesn’t have the same understanding of product history, organizational context, user behavior, or business goals that teams build over time.

While AI has accelerated design execution, execution was never the hardest part of UX design. The real challenge has always been understanding the problem, navigating tradeoffs, and making decisions that hold up across an entire product experience. As generating screens becomes easier, evaluating whether those screens solve the right problem becomes the new bottleneck.

AI hasn’t eliminated design work. It has shifted where the work happens.

As AI becomes more integrated into design workflows, maintaining intentionality, coherence, and strong product judgment becomes increasingly important. In this article, we’ll explore where these gaps are beginning to appear, how AI is changing design workflows, and what designers can do to ensure speed doesn’t come at the expense of thoughtful design.

TL;DR

• AI has dramatically accelerated design execution, making it easier than ever to generate screens, flows, and prototypes.
• While output has increased, product thinking hasn’t necessarily kept pace. AI-generated concepts often overlook edge cases, technical constraints, and organizational context.
• The bottleneck in UX is shifting from creating solutions to evaluating whether those solutions actually solve the right problem.
• AI excels at generating interfaces but struggles with system-level thinking, cross-functional tradeoffs, and understanding how decisions affect the broader product ecosystem.
• As AI becomes more integrated into design workflows, reviews, critiques, and validation become even more important.
• Product judgment, systems thinking, and the ability to identify weak assumptions are becoming some of the most valuable skills for UX designers.
• The designers who thrive in AI-assisted environments won’t be the ones generating the most ideas. They’ll be the ones who know how to evaluate them, challenge them, and determine which ones are worth building.

The hidden value of friction in the design process

One of the biggest benefits of AI that designers keep hearing from leadership teams is the removal of friction from workflows. Before AI-assisted workflows became common, many of the slower parts of the design process naturally created opportunities for reflection. Designers spent more time discussing tradeoffs, validating assumptions, and understanding the problem before jumping into solutions. Those conversations helped teams align around technical considerations, user feedback, product goals, and business priorities long before a screen ever existed.

As AI becomes more integrated into our workflows, iteration cycles are compressed. The danger isn’t that AI prevents teams from understanding the problem. It’s that the speed of generation can create pressure to move into solutions before the problem is fully understood.

When concepts can be generated instantly, it’s tempting to skip over the critical thinking stages that once happened naturally. What appears to be inefficiency on the surface often served an important purpose. The discussions, critiques, and moments of uncertainty that occurred before a design took shape helped teams build confidence that they were solving the right problem in the first place.

While faster output can be valuable, the friction embedded within many design processes was often a blessing in disguise. It created space for reflection, alignment, and better decision-making. As AI removes some of that friction, designers need to be intentional about preserving the thinking that those slower processes enabled.

More output doesn’t automatically create better UX

One of the more interesting side effects of AI-assisted design is that generating ideas is no longer the bottleneck. The new friction is evaluating them.

In the past, a designer might bring a handful of concepts into a review session. Today, it’s possible to generate dozens of variations in the same amount of time. While that can be incredibly useful for exploration, it also creates a new challenge to separate strong ideas from weak ones.

Polished prototypes can create false confidence

I’ve noticed this when experimenting with AI-generated concepts myself.

The first few outputs often feel impressive because they look complete. The layouts are polished, the flows are structured, and the screens appear production-ready. But once you start reviewing them through the lens of a real product, the cracks begin to show. Important edge cases are missing. Interactions don’t align with existing patterns. Business requirements are overlooked. What initially looked like a solution turns out to be a starting point.

This is where more output can become misleading. Generating ten concepts instead of two doesn’t automatically lead to better UX. In some cases, it simply creates more noise for teams to sort through. The challenge shifts from creating options to understanding which options are actually worth pursuing. A polished prototype can create the impression that a problem has been solved when, in reality, the hard work of product thinking still needs to happen.

Understanding user needs, evaluating tradeoffs, and ensuring a solution fits within the broader product experience are still responsibilities that belong to the team. AI has made it easier to generate solutions, but it hasn’t made it easier to determine whether those solutions are the right ones.

AI struggles with system-level thinking

AI tools have come a long way from the first introduction of ChatGPT, but they are still far from perfect. No matter how human-like their response sounds, it can only respond based on what they are given in the prompt and the information the model is trained on. So while a response can sound intelligent, taking a closer look can see that they are lacking in details, specifics, and correlations to other things that they don’t know about.

AI understands screens better than systems

This limitation translates directly into AI design tools. You can generate a visually convincing dashboard in seconds, but visual polish doesn’t guarantee that the information architecture, workflows, or data relationships actually make sense. AI may not be connected to your design system, understand your component library, or know how your product has evolved over time. It can generate screens that look plausible without understanding whether they fit into the broader experience.

It may not know about the product’s history or why certain design decisions were made in the past. It doesn’t understand operational realities, technical constraints, or the cross-functional tradeoffs that shaped the product over time. Most importantly, it doesn’t understand how one decision affects the broader ecosystem.

Products don’t exist in isolation

A good example is how products evolve over time. Features rarely exist in isolation. A small change to onboarding can affect activation metrics. A change to permissions can impact support workflows. Introducing a new pattern in one area of the product may create inconsistencies elsewhere. These relationships are often invisible unless you’ve spent time working within the product and understanding how teams, systems, and users interact.

This is where UX moves beyond screen design. The most important design decisions are often about managing complexity across an entire ecosystem, something AI still struggles to reason about without significant context.

Without this context, you are left with surface-level design filled with broken edge-case handling, accessibility gaps, and fragmented patterns that don’t feel cohesive with the rest of your users’ workflows.

The rise of AI workflows, not just AI features

As a UX designer, I’m optimistic about how AI can enhance our workflows, improve collaboration, and help designers spend less time on repetitive tasks and more time solving meaningful problems. However, I’ve noticed that a lot of the conversation focuses on outputs. Teams are experimenting with prompts, comparing results, and finding ways to generate better screens, flows, and prototypes, which is understandable, but what’s received less attention is how AI changes the workflow around the work.

Recently, my team experimented with using AI to generate a new feature I was designing. The first result looked surprisingly complete. But after sharing it with engineering and product, we quickly uncovered missing permission states and technical constraints that the AI had no way of knowing about. The result wasn’t a failure, but it was a reminder that generating a solution is often the easy part.

Understanding whether that solution works within the realities of the product is where the real work begins.

AI changes how teams make decisions

Every meaningful technology shift changes more than just the tools we use. It changes how decisions get made. It changes who reviews what, when feedback happens, and how teams build confidence in their work.

AI-generated designs introduce new questions, like:

• What information was given to the AI?
• What context was missing?
• Why did it arrive at this particular solution?
• Is it accounting for technical constraints, accessibility requirements, edge cases, or existing product patterns?

That’s why teams can’t treat AI as a shortcut around critical thinking. If anything, it makes reviews even more important. The conversation changes from wondering what to create to if the team is headed in the right direction, not just because AI outputted a solution. Teams need opportunities to challenge assumptions, validate decisions, and identify what’s missing before work moves forward.

Product judgment is becoming the most valuable design skill

The organizations that get the most value from AI won’t be the ones generating the most screens. They’ll be the ones that build strong review habits around AI-generated work and know where human judgment needs to step in.

For designers, this changes where we create value. As execution becomes easier, our role becomes less about producing artifacts and more about evaluating them. Understanding the problem, identifying tradeoffs, spotting gaps, and maintaining coherence across the experience become far more important than generating another variation of a screen.

I’ve found that some of the most valuable design conversations today have very little to do with creating new ideas and much more to do with evaluating them. When reviewing AI-generated concepts, I’ve started asking myself a different set of questions than I would have a few years ago:

• Does this solution actually solve the problem?
• What assumptions is it making?
• What happens when a user falls outside the happy path?
• How does it fit within the broader product experience?

These are questions AI struggles to answer because they require context, tradeoff analysis, and an understanding of how products evolve over time.

Systems thinking becomes a competitive advantage

As a result, skills like systems thinking, prioritization, and product reasoning are becoming more valuable than ever. Designers need to be able to identify weak patterns, spot missing context, and recognize when a solution looks polished but introduces new problems elsewhere in the experience.

In many cases, knowing what not to build becomes just as important as knowing what to build.

Consistency becomes harder at scale

Another challenge is maintaining consistency. AI is very good at generating individual screens or flows, but users move between features, workflows, and states. Decisions made in one area of a product often have downstream effects elsewhere. Without someone actively looking across the entire experience, it’s easy for inconsistencies and design debt to accumulate over time.

In many ways, our value is shifting from creation to curation and judgment. The designers who thrive in AI-assisted environments won’t necessarily be the ones generating the most ideas. They’ll be the ones asking the right questions, identifying what’s missing, and ensuring that every decision contributes to a cohesive product experience.

How designers can stay intentional in AI-assisted workflows

It’s seems clear that AI is here to stay and we shouldn’t shy away from it. The goal isn’t to slow down progress or avoid using new tools. The real challenge, however is ensuring that speed doesn’t come at the expense of thoughtful design. Here are a few practices that I’ve found helpful as I continue to navigate the learning curve of AI-assisted workflows.

Slow down the decisions that matter

Not every part of the design process should be sped up. Yes, AI can help generate concepts quickly, but decisions around user needs, workflows, tradeoffs, and product direction still deserve careful thought.

Before moving forward with an AI-generated solution, create intentional moments for reflection and review. Consider whether the solution actually solves the problem and whether you’ve considered alternative approaches that may be more effective. Often, AI-generated ideas serve as a great foundation to continue building on top of, rather than a one-and-done solution.

Slowing down allows teams to hone in their focus on what matters and filter out the noise.

Design the edge cases, not just the happy path

As mentioned earlier, AI is great at generating the ideal state flow when given a brief, but real products are rarely that simple. Designers need to plan for error states, empty states, different user permissions, and ambiguous scenarios. These moments often determine whether a product feels reliable or frustrating.

When reviewing AI-generated concepts, I keep a mental checklist to review all the possible states and exceptions that could occur to ensure that it’s battle-tested and ready for production.

Review work at the system level

Strong product experiences are a result of consistency and coherence across workflows, features, and interactions. It’s common for complex products to have interdependencies where taking action in one part of the product has consequences elsewhere. An AI-generated solution may solve a problem in one area, but cause downstream effects in another. This is why system-level reviews will become crucial.

Designers should be aware of the interdependencies across features and understand how introducing a new feature may impact the overall experience. Watch out for inconsistencies across patterns, terminology, and interactions before moving forward with an AI-generated design.

Use AI as a collaborator, not an authority

As I’ve gotten used to incorporating AI into my workflow, I’ve realized that its strength is not in simply creating my designs for me. In fact, I still have an easier time creating designs manually, as I’m never too sure what AI will generate, leaving me to tweak the details, which can sometimes take as long as just creating the entire design from scratch. Instead, I’ve found that AI works best as a collaborator, like a second brain to riff ideas off of. They may not always be perfectly accurate, but they can definitely help me reason through tough problems, help me sort out my workflow when I’m uncertain how to proceed, and organize my thoughts into a plan to help me get to my solutions quicker.

By treating AI as a collaborator rather than an authority, I’m enhancing my ability to think critically by feeding it all of my thoughts. This gives me a greater level of certainty rather than just claiming its outputs as my work.

Protect the thinking behind the work

One habit I’ve become more intentional about is documenting the conversations that lead to good decisions. AI can generate solutions quickly, but critiques, collaborative sessions, and stakeholder discussions are still where much of the real learning happens. As teams adopt AI, it becomes important not to remove these conversations simply because a prototype can be generated in minutes.

The goal isn’t to preserve old processes for the sake of tradition, but to preserve the thinking that those processes enabled.

Conclusion

AI is making it easier than ever to generate screens, flows, and prototypes, but execution was never the hardest part of product design. The real challenge has always been understanding the problem, navigating tradeoffs, and making thoughtful decisions that hold up across an entire product experience.

As AI becomes more integrated into our workflows, it’s worth being intentional about where we apply our time and attention. Designers should challenge AI-generated outputs instead of accepting them at face value. Spend time exploring edge cases and system-wide impacts, not just the happy path. Review experiences as complete workflows rather than isolated screens. Most importantly, continue creating opportunities for critiques, discussions, and product reasoning.

The designers who get the most value from AI won’t necessarily be the ones producing the most output. They’ll be the ones who know when to slow down, ask better questions, and apply product judgment where it matters most. AI can help us explore more possibilities than ever before, but it’s still up to us to determine which ones are actually worth building.

ByteDance appears close to releasing Dreamina Seedance 2.5, with current rumors pointing to a possible July 9 launch. Public Dreamina and CapCut pages already reference the model, while third-party reports suggest an early-July rollout window rather than a confirmed date. Availability is expected across Dreamina, CapCut, and partner platforms that already work with ByteDance’s video stack.

Coming soon: Dreamina Seedance 2.5 is arriving on CapCut.

Seamless generation and editing. Up to 50 multimodal references. 30-second scenes in one shot. Finer creative control. More reliable results. It's built to make creating faster, smoother, and more intuitive.

The main new feature will be a move from short clips to 30-second standard video generation. Dreamina pages describe a long video workflow that can generate 30-second scenes, 90-second drafts, and 180-second outputs, making three-minute AI video the biggest claimed shift for this release. The open question is not only about duration but also about whether the model can keep character identity, motion, camera logic, and prompt intent stable across extensions.

You can create cinematic videos up to 30 seconds in standard mode or extend them to 180 seconds with the beta long-video mode.

For creators, advertisers, anime editors, social video accounts, and AI filmmakers, this would turn Seedance from a short-shot generator into a tool for longer sequences. The likely location is within Dreamina’s Seedance workflow, CapCut’s creator tools, and partner apps that use ByteDance’s model access. The evidence appears to come from public-facing product pages and platform copy, while the July 9 timing remains rumor-level until ByteDance posts a dated release notice.

The company behind the model is ByteDance, which has been tying AI video to its creator ecosystem through Dreamina, CapCut, TikTok-adjacent production flows, and API distribution. Seedance 2.0 is still positioned around motion stability, multimodal references, and audio-video generation, with listed durations of 4 to 15 seconds on BytePlus. Seedance 2.5 would push that strategy toward longer commercial storytelling and creator workflows, while Google’s Veo and Gemini Omni remain the clearest competitive pressure after OpenAI discontinued Sora’s web and app product on April 26, 2026.

Understanding the Dynamics of the AI Ecosystem with Pace Layers

When a sector goes too fast, it loses support

Without a doubt, the pace of the AI ecosystem is dizzying. Just processing it all is difficult enough. Scaffolding it, finding themes, and understanding the shape of it is nearly impossible.

Recently, Mike Migurski introduced me to Stewart Brand’s Pace Layers, a framework for organizing fields and categories by how fast they change. Brand writes:

Consider the differently paced components to be layers. Each layer is functionally different from the others and operates somewhat independently, but each layer influences and responds to the layers closest to it in a way that makes the whole system resilient.

From the fastest layers to the slowest layers in the system, the relationship can be described as follows:

Fast learns, slow remembers. Fast proposes, slow disposes. Fast is discontinuous, slow is continuous. Fast and small instructs slow and big by accrued innovation and by occasional revolution. Slow and big controls small and fast by constraint and constancy. Fast gets all our attention, slow has all the power.

In The Clock of the Long Now, Brand proposes six macro layers that represent a “healthy civilization”, as seen below:

Imagine: as these layers move at different rates, friction builds between them, slowing the upper layer and quickening the lower. This negotiation, translation, between the layers is constructive when their speeds are different, but in balance. When they’re not, things get weird.

Brand writes:

In a durable society, each level is allowed to operate at its own pace, safely sustained by the slower levels below and kept invigorated by the livelier levels above… Each layer must respect the different pace of the others. If commerce, for example, is allowed by governance and culture to push nature at a commercial pace, then all-supporting natural forests, fisheries, and aquifers will be lost. If governance is changed suddenly instead of gradually, you get the catastrophic French and Russian revolutions. In the Soviet Union, governance tried to ignore the constraints of culture and nature while forcing a five-year-plan infrastructure pace on commerce and art. Thus cutting itself off from both support and innovation, it was doomed.

The last 10 days have been a whirlwind of conferences: Foo Camp, Open Frontier, AI Engineering World’s Fair… Every night I’d come home and scribble down notes, hoping a structure or two would emerge to bring it all together.

I think Pace Layers is the best I’ve got:

Dwelling on this, a few stray notes emerge:

• So much of the current AI backlash can be linked to massive investment forcing lower layers to move faster than normal. Data centers are moving faster than the culture above them (the organizations, governance, and universities), charging the debate around their buildout with incredible emotions.
• Further, the speed of the data center layer should move faster than energy production, but if it moves too quickly we go from light friction to earthquake level seismic effects. Usually, data centers could probably be slotted in the “decades” bucket (from proposal to completion), but at the moment we’re pulling them into mere “years.”
• The advancements of models over the last 18 months or so have been driven by hired and synthetic data; the organic human data is essentially tapped. We’re not going to get another internet.
• The incredible speed of the first 2-5 layers are functioning as feedback to the models, hired expert data, and training methods layers. Because the layers below the training methods are just so much slower than the upper layers. This lack of feedback from the layers that usually support the upper ones is an issue. The upper layers (incredible as they are!) are screaming ahead while organizations and universities continue apace, unable to support their speed.
• This is why you can go to the AI Engineering World’s Fair and come away thinking everyone is building dark factories and automating entire enterprises, while non-developers from outside the greater-San Francisco AI complex wonder why data centers are necessary.

Meta and SpaceX selling compute capacity could mean that there's excess compute capacity that two of the largest buyers no longer need. This may result in downward capex revisions for the hyperscalers. It's possible demand simply isn't there for what's being built with AI. However, this is unlikely, as anyone willing to sell capacity finds buyers immediately.

We are building a public option for AI that is open, auditable, and in service of people over profit. However, the open source AI stack has gaps. Now, together, we can close them.

Building on work from leading open source AI experts at the Columbia Convening, MOF, Hugging Face and others, we undertook an ambitious effort to map the open source AI stack to understand what’s missing. We evaluated over 24,626 projects from foundation models through inference backends, assessing projects across openness, capability, and adoption.

We're looking for collaborators

If you want to review products, improve our methodology, or add your tools, reach out.

Own the Loop: A Field Guide to Agent Harnesses

The better a harness fits its model, the less of it is yours. The most capable coding agents today are model-native pairs: a frontier lab matches its best model to its own harness. Claude Code is...

A brief history of distillation in AI

This is a warm-up for Class 2 of our Training an Agent series, where we distill open models with TRL. First, a short history. Distillation is a hot topic in post-training. The latest GLM-5.2 release...

Full article content is not available for inline reading.

Read the original article →

Google appears to be building a dedicated inbox section inside the Gemini app for Business and Workspace customers. This feature has been spotted in recent builds but is not yet available to anyone. The layout centers on three filters that allow a person to sort items to follow up on, review what has been marked done, and check work that is ready for review. The framing points at an Inbox Zero workflow that pulls messages out of Gmail and into the Gemini surface itself, rather than layering help on top of the mailbox.

The more telling signal is the "Needs review" filter, which suits a proactive agent that triages email and data from connected sources, then files what it finds into a structured to-do list for a person to walk through. This pattern already runs through some other Google apps:

1. Daily Brief compiles urgent mail and calendar items into a morning summary.
2. Gemini Spark operates as a background agent that can archive newsletters and surface follow-ups.
3. The Gmail-based AI Inbox, shown to testers earlier this year, lifts deadlines and tasks to the top.

Placing this view inside Gemini aligns with the company's wider direction. Google has spent the past year transforming Gemini from a chatbot into a worker, adding a macOS app, browser-driven agent runs, and the no-code automation builder, now called Workspace Studio, which is accessible from Gmail. Folding email triage, task tracking, and those automations into one panel would push Gemini toward a consolidated desktop workspace, a single place where Computer Use and Browser control sit beside the mailbox. Whether that becomes a true super app for Workspace remains open, though rival agent platforms are moving in the same direction.

Key Points

• Amazon says it has deployed enough satellites to launch its Leo commercial service later this year.
• Leo will compete with SpaceX's Starlink, which was created four years before Amazon's announced its entry into the market, and has more than 10,000 satellites in its constellation.
• Network coverage will likely be limited to certain geographies until Amazon can bring more satellites online.

Amazon said it now has enough satellites in orbit to begin "initial service" of its Leo internet-from-space network later this year.

The company shipped 29 satellites into orbit around 12:30 a.m. ET on Thursday atop a United Launch Alliance Atlas V rocket. The mission brings Amazon's total constellation to more than 390 satellites, which is "enough to support continuous service across initial latitudes," Chris Weber, vice president of business and product for Amazon Leo, wrote in a post on X.

It's a key milestone for Amazon as the company tries to make Leo a competitor to SpaceX's Starlink in the low Earth orbit satellite market. In November, Amazon began offering an "enterprise preview" of Leo for select businesses, but it has yet to launch its service for consumers and government customers.

Amazon's initial commercial service will likely be limited to users in certain geographies. Future missions will "add coverage and capacity," Weber said.

SpaceX had a four-year head start on Amazon, launching Starlink in 2015. It has since amassed a constellation of around 10,000 satellites and more than 10 million subscribers. Amazon announced the creation of Kuiper in 2019, and later changed the name to Leo.

Amazon aims to build a constellation of roughly 7,700 satellites, but the effort has been slowed by a shortage of rocket capacity. In its January request for an extension on regulatory deployment deadlines, the company cited delays beyond its control, including a "shortage in the near-term availability" of rockets. Amazon in 2022 signed a historic deal to reserve rocket launches with ULA, Arianespace and Jeff Bezos' Blue Origin, before purchasing rides with SpaceX. Many of those providers have experienced delays with their launch vehicles.

Another setback came in May, when one of Blue Origin's New Glenn rockets exploded on the launchpad during a hot-fire test, just days before it was slated to carry a batch of Amazon satellites. The company is currently rebuilding the pad, and working to determine what caused the anomaly.

Bezos and Blue Origin CEO Dave Limp have said the company is determined to return New Glenn to flight later this year. New Glenn is a giant, partially reusable rocket that seeks to compete with SpaceX's Starship rocket and can carry heavier payloads of up to 45 metric tons to low Earth orbit.

Amazon said Thursday its next Leo mission will use ULA's Vulcan heavy-lift rocket, "which will carry even larger Leo payloads and help increase our deployment rate."

"With hundreds of flight-ready satellites standing by at the Cape and a new, dedicated vertical integration facility ready to support Leo Vulcan 1 and subsequent missions, we have a clear path to increase launch and deployment cadence, helping us quickly expand network coverage following an initial service rollout later this year," Melissa Wuerl, Leo's director of launch systems, said in a statement.

Key Points

• Under the new initiative, Chipmaker Nvidia aims to expand infrastructure access for emerging enterprises.
• Two initial partners are offering potential access to more than 200,000 GPUs, including via an upcoming data center in Batam, Indonesia.

Chipmaker Nvidia is expanding its support for fast-growing AI startups by introducing a partnership program that leverages its network of AI cloud service providers to offer these firms access to critical computing infrastructure, the company said Thursday.

Cloud-based AI firms, model builders and other enterprises will share both product and cloud revenue with Nvidia, which is positioning itself as an intermediary helping startups gain direct access to full-stack computing powered by Nvidia chips.

In its announcement, Nvidia named two initial Australian firms that will provide the compute power for the scheme.

Sharon AI will deploy up to 40,000 Nvidia GPUs, while AI infrastructure company Firmus Technologies says it is building a data center in Batam, Indonesia, which is expected to scale to 360 megawatts and house up to 170,000 Nvidia GPUs.

Nvidia's move illustrates the critical importance of access to scarce compute power for AI-oriented startups, with GPUs likened to oil and even reportedly tied to futures contracts as users grapple with fluctuations in cost and issues around availability.

Meanwhile, AI firms have increasingly entered into revenue and equity-sharing agreements with chipmakers in order to circumvent liquidity issues afflicting the sector.

OpenAI has inked a number of deals that have seen it buy shares or entertain investments from partners including Amazon and AMD, CNBC reported in January.

Nvidia earlier this month said it was aiming to raise debt, which sources said could amount to at least $20 billion. The firm intends to use the proceeds from the offering for general corporate purposes, including repayment and refinancing of existing debt.

A Field Guide to Fable: Finding Your Unknowns

Working with Claude Fable 5 keeps re-teaching me an old lesson: the map is not the territory. The map, a representation of the work to be done, is my prompts and skills and context, it’s what I give...

Threads now has 500 million monthly users. The app has become especially popular in Asia. Threads is now targeting one billion users. The app started showing ads in January - reaching its user goal could lead to revenue of at least $30 billion a year.

Key Points

• Apple has reportedly planned at least five new iPhone models through early 2027, according to Nikkei Asia.
• The company has reportedly raised its foldable iPhone production plans to 10 million units.
• It has also explored Chinese chip suppliers as AI demand strained memory supplies, Bloomberg reported.

Apple plans to launch at least five new iPhone models between the second half of this year and the first half of 2027, while increasing production plans for foldable devices, as it looks to vie for a greater slice of the market amid an industrywide component supply shortage, Nikkei Asia reported Thursday.

The U.S. tech giant has instructed suppliers to prepare to produce about 10 million foldable iPhones this year, up from an earlier forecast of 7 million to 8 million units, the report said, citing people familiar with the matter.

Ahead of launching its first-ever foldable device, Apple has already secured components for about 80 million smartphones spread across new models for the second half of 2026, according to Nikkei Asia.

Apple's total smartphone production for 2026 is expected to exceed 220 million units, the report said. Its scale and purchasing power in sourcing memory and components remain significantly stronger than most of its peers, even as shortages driven by AI-related demand ripple through the industry.

This has allowed Apple to navigate supply shortages better than Chinese rivals such as Xiaomi, Oppo, and Vivo, which have each slashed their annual production targets to below 100 million units, Nikkei Asia reported.

"Compared with Apple's bargaining power, the Chinese smartphone makers are in a weak spot in terms of getting more supplies of memory chips or increasing the prices," an executive at a supplier for both Apple and Xiaomi told Nikkei Asia. "It gives Apple a good motivation to launch the iPhones in spring and take more of their share."

Apple's efforts to secure components come as a global memory shortage driven by demand from artificial intelligence data centers pushes up costs across the industry.

Bloomberg reported Thursday that Apple is in talks to source memory chips for devices sold in China from Chinese manufacturers ChangXin Memory Technologies and Yangtze Memory Technologies, both of which are included on a Pentagon list of companies alleged to support Beijing's military. Apple has not confirmed the discussions, and Bloomberg reported that negotiations remain ongoing.

Apple is reportedly seeking to broaden its supplier base as memory shortages strain production across the consumer electronics sector.

Apple plans to introduce at least two new iPhones in the first half of 2027, including the standard iPhone 18 and a new iPhone Air, according to Nikkei Asia.

The aggressive product roadmap comes after Apple implemented price hikes for its MacBook and iPad lineups last week as memory and storage costs surged.

Apple did not immediately respond to CNBC's request for comment.

The core issue isn’t where your server sits. It’s who can be compelled to hand over what’s on it.

For years, cloud providers treated sovereignty as a geography problem. Pick a region. Choose a country. Keep your data local.

But laws such as the U.S. CLOUD Act changed the equation. Data access follows corporate control, not physical location. A hyperscaler operating infrastructure in Frankfurt remains subject to the laws governing its parent company. Region selection is a geographic control. Sovereignty is a jurisdictional one.

That distinction is increasingly shaping infrastructure decisions.

The EU’s Cloud and AI Development Act (CADA), proposed in June 2026, introduces a four-tier sovereignty framework for public sector cloud procurement. Canada’s federal government now scores cloud vendors on Canadian data residency and jurisdictional control. Many countries now maintain data localization, sovereignty, or residency requirements, and regulatory frameworks increasingly extend beyond data residency into questions of operational control, supply chain transparency, portability, and resilience.

The EU Data Act promotes interoperability and reduces barriers to switching providers. The AI Act introduces requirements around traceability, governance, and accountability for AI systems. NIS2 and DORA place greater emphasis on supply chain dependencies, operational resilience, and concentration risk. Similar conversations are emerging across jurisdictions worldwide.

Taken together, these developments point toward a future where infrastructure control is no longer merely a technical preference. It is increasingly becoming a regulatory expectation.

This shift is often framed as a privacy discussion. In reality, it is increasingly a resilience discussion. The same architecture that protects against foreign legal interference can also reduce exposure to sanctions, trade disputes, service suspensions, licensing changes, vendor exits, and other forms of external dependency.

For platform engineers, the challenge is straightforward: how do you meet sovereignty requirements without sacrificing the automation, portability, and operational efficiency that made cloud infrastructure attractive in the first place?

The pattern emerging in production

Across Europe, regulated enterprises are increasingly assembling sovereign platforms from open-source components rather than buying sovereignty as a premium feature from hyperscalers.

The pattern is becoming familiar. Kubernetes provides the orchestration and policy layer. GitOps provides operational consistency across jurisdictions. OpenStack supplies the underlying infrastructure. Together, they allow organizations to enforce sovereignty requirements through architecture rather than contracts.

These are not proof of concept deployments. National rail operators, major banks, and European telecoms are already using Kubernetes, GitOps, and policy driven automation to operate regulated workloads at scale across sovereign environments.

The pattern is consistent: Kubernetes for governance and orchestration, OpenStack for sovereign infrastructure, policy as code for enforcement, and declarative operations for repeatability.

Kubernetes as the sovereignty control plane

Building sovereign infrastructure is one challenge. Operating it consistently is another.

Most compliance programs still rely heavily on documentation, reviews, and manual processes. Those approaches work, but they don’t scale particularly well across hundreds or thousands of workloads. Kubernetes changes that by allowing sovereignty requirements to be enforced directly by the platform.

Admission controllers can enforce workload placement before a pod is ever scheduled. Node affinity rules ensure workloads only land on approved infrastructure within the correct jurisdiction. Namespace isolation creates clear boundaries between tenants, environments, or regions. Policy engines evaluate every API request against sovereignty requirements and reject non-compliant resources before they reach production.

Policy as code extends the same approach operationally. Sovereignty policies live in Git, are peer reviewed, tested through CI pipelines, and enforced automatically at deployment time. Tools such as OPA/Gatekeeper and Kyverno allow organizations to encode jurisdictional requirements directly into the cluster. The result is continuous enforcement rather than periodic verification. Every policy change is traceable. Every deployment decision is auditable.

At that point, sovereignty stops being a process and becomes a platform capability.

But Kubernetes still depends on an infrastructure layer underneath it. Compute, networking, storage, and identity all have to come from somewhere. If that foundation is tied to a platform operated by an organization outside your jurisdiction, some of the guarantees enforced at the Kubernetes layer become harder to maintain.

This is where OpenStack fits into the picture.

OpenStack provides the infrastructure services Kubernetes relies on while allowing organizations to operate those services within their own jurisdiction. Bare metal provisioning through Ironic removes the need for a proprietary hypervisor between workloads and hardware. Keystone keeps identity management self-hosted. Neutron provides network isolation under the operator’s control. Ceph delivers distributed storage on infrastructure you own and operate.

OpenStack can be deployed entirely within a controlled environment. No license servers. No mandatory telemetry services. No external dependencies required for day-to-day operations.

The combination is what makes the architecture work. Kubernetes provides the policy and enforcement layer. OpenStack provides the infrastructure foundation beneath it. Together they allow sovereignty requirements to be implemented in code, enforced automatically, and audited across the entire stack.

The hard part: Operating it

Sovereignty often means separate environments for each jurisdiction. Every upgrade, certificate rotation, RBAC change, security patch, and capacity planning exercise now happens across multiple clusters instead of one.

GitOps is what makes this operationally manageable.

A Git repository contains shared configurations and jurisdiction specific overlays. GitOps controllers running inside each cluster continuously reconcile against that desired state. No centralized control plane is required. Each cluster pulls and applies its own configuration locally.

The operational benefits are obvious, but the compliance benefits are just as important. Every change is reviewed, version controlled, and auditable. When someone asks what was running in a cluster at a specific point in time, the answer is already in the commit history.

The same principle applies to the software supply chain. SBOMs, image signing, and admission policies help ensure that only verified workloads reach production.

For organizations pursuing higher levels of sovereignty, visibility cannot stop at the operating system. Firmware, management controllers, and hardware components sit below the software stack and often have broad access to the host itself. That is why Hardware Bills of Materials and firmware verification are becoming part of the sovereignty conversation as well.

What comes next

The infrastructure buildout is already underway. CADA aims to significantly expand European data centre capacity over the coming years, but hardware alone does not create sovereignty. The platform layer matters just as much as the infrastructure underneath it.

AI is making that reality even more visible. As regulators place greater emphasis on how models are trained, governed, and audited, training infrastructure is increasingly being evaluated through the same sovereignty lens as the data itself.

Federated learning is one example of how this changes architecture. Instead of moving data to a central location, training happens where the data already resides. Sovereign Kubernetes clusters perform local training while only aggregated model updates move between jurisdictions. The same policies, namespace boundaries, and governance controls used for compliance become the foundation for distributed AI systems.

For platform teams, the question is no longer whether sovereignty requirements are arriving. They are already influencing procurement decisions, infrastructure design, and operational models.

The good news is that the building blocks already exist. Kubernetes provides the orchestration and policy framework. OpenStack provides the infrastructure foundation. GitOps, policy engines, software supply chain security, and identity complete the picture.

For years, cloud infrastructure optimized for centralization. Sovereignty is pushing in the opposite direction: more regional control, greater transparency, and stronger operational ownership.

The question is no longer whether sovereignty will shape infrastructure design.

The question is whether sovereignty remains something organizations document, or something their platforms can enforce.

Many of the technologies discussed in this article are developed and operated in the open. Explore VEXXHOST’s open-source work and contributions on GitHub: https://github.com/vexxhost.

I recently helped a client migrate from Azure DevOps to GitHub Enterprise Managed Users with Data Residency (ghe.com for short). And as part of the migration we rewired the exiting Azure Pipelines from the old repository located on Azure Repos to the new one on GitHub.

With GitHub Enterprise this is quite easy, create a new service connection in Azure DevOps and click the magic authorize button to install the app into GitHub automatically. Unfortunately no such luck in ghe.com, even though there is a special service connection type.

The screen mentions: "GitHub Organization should have azure pipelines app already installed", but there is no documentation anywhere on how to actually do that.

And unfortunately, it turned out we weren't the first to hit this snag. And the error message is not very helpful either:

I reached out to my contacts in GitHub and Azure DevOps to see if they knew what to do, but unfortunately, even they did not know.

I decided to play around some more and stumbled upon this well hidden url based on the url of the Azure Boards app by replacing boards with pipeline

https://{enterprise-slug}.ghe.com/apps/external-app/azure-boards
https://{enterprise-slug}.ghe.com/apps/external-app/azure-pipelines

And lo and behold, from this location you can install the Azure Pipelines app:

Click Install to and select your GitHub organization, after which the configuration of the service connection will work without a hitch:

Select the repos you want to enable for Azure Pipelines:

After which you are redirected to Azure DevOps to complete the installation:

If you now go back into the settings of your Azure DevOps Project, you should be able to create the new service connection:

Click Authorize after which you'll be able to Save the new service connection.

To rewire a pipeline, navigate to the pipeline you want to rewire and click Edit:

Click the ⁞ button and then select Triggers:

This brings you to the classic pipelines editor on which you can navigate to the YAML tab:

Select the Get Souces element in the pipeline

And rewire the pipeline to GitHub by selecting the GitHub tile and picking the Service Connection we previously created:

Rust is a reliable language that prioritizes correctness. However, during development, it can be a burden to account for every single error path immediately.

This article presents a few techniques that defer handling correctness to that you can stay longer in the happy path, as well as a library I developed to make this process more convenient.

🌱 100% of this article has been handcrafted without using generative AI tooling, if somewhat hastily. Take my typoes and broken sentences as proof of that.

What's worse for a Rust developer working on some elaborate logic than having rustc suddenly getting in the way? Be it because you didn't return a Result yet, or handle this error case, or implement that edge case, causes for compilation errors are numerous and can distract you from what you're attempting to achieve.

When this happens you're usually presented with two options:

1. Go with the distraction and immediately implement perfect error handling. Typically you'll have to refactor it a number of times before the PR is ready, making the whole process much more heavy than what is necessary.
2. Slap some manner of TODO later label on the error, downgrading it to a warning, and move on with your work for now.

This article covers some Rust techniques to achieve the latter option, downgrading an immediate error to a warning to be handled later.

Why do you want warnings?

Warnings occupy the perfect spot in the possible gamut of compilation interactions:

• Errors block other compilation steps (lifetimes, etc), and testing, impeding the common workflow:

  1. Write WIP code
  2. Test it
  3. Turn WIP code to production code

  Any error prevents going from (1) to (2)

• Completely suppressing any compilation message, on the other hand, runs the risk of forgetting about the issues and letting them sweep into production 😬

• Warnings don't block development, but need to be cleaned before merging the PR.

  At Meilisearch, we have a CI that runs the compiler with the -D warning flag, that will turn all warnings into errors at CI time, ensuring we don't ship WIP code accidentally.

Why don't you just read the f**king code???

Of course, I do. Reading code (yours, and the one from your colleagues) is an excellent barrier against bugs, and code reviews are a requirement to effectively share knowledge among your team.

All the techniques discussed in the article come as an additional manner of creating incentives for correctness and stronger guardrails when producing code at a professional scale. They increase quality but don't replace the basic good practices.

In my experience as a software engineer, anything that relies on humans never a mistake is deemed to fail eventually.

Without further ado, here are some techniques you can use in vanilla Rust to implement error deferral strategies.

Techniques with standard Rust

unwrap everywhere

While it is important that Rust gives us the structure to correctly address errors, it doesn't mean we need to address them immediately.

A common pattern for delaying the error handling is to simply unwrap your Result or Option, to get the value inside at the price of a panic on the error path.

Then when the rest of the code is ready, you can go back to your changes and selectively remove the unwrap that were added as a temporary measure, replacing them with proper error handling.

fn does_not_return_result_yet(some_params: SomeType) -> SomeReturnType {
  let converted = some_params.try_into().unwrap() // <- fallible conversion, ignore for now
  converted.try_foo().unwrap() // <- fallible operation, ignore for now
}

clone everywhere

Ownership is a similar issue, albeit slightly different and much less common than error handling. Properly maintaining ownership sometimes mandates design modifications, lest lifetime issues appear. In this case, a frequent mitigation option is to clone a value instead of moving it around.

Delaying the fixes for this category of issues is a bit more delicate, as the fix can entail changing the architecture of the code in non-trivial ways. Still, with experience, it is possible to identify early cases where you'll be able to remove the clones later by passing the appropriate variables higher in the stack.

the todo! macro

The Rust standard library provides a todo!() macro, whose purpose is described as:

Indicates unfinished code.

This can be useful if you are prototyping and just want a placeholder to let your code pass type analysis.

This is indeed extremely useful when shaping out an interface and needing to make some function definitions available, without actually having to provide a working implementation.

Because todo! diverges (it panics), the compiler is able to use it as a placeholder expression for most types, hence the reference to type analysis in the docs. A liberal use of todo allows deferring some code paths (and not just errors) to be implemented at a later time.

Digression: not to be confused with unimplemented!()

I saw at least once someone being wrong on the Internet about todo!(): they would state that it was meant to communicate to users of a program that some features were not yet implemented.

Rust actually has a dedicated macro for this use case, and its name is unimplemented!().

todo!() has a different audience: the author of the code and their reviewers. No todos are meant to hit production, they are a WIP Rust amenity only.

mut, unused variables, etc.

The default warnings provided by the compiler are very useful while developing. They will catch:

• unused bindings
• needlessly mutable bindings
• unused results and options

All of these tend to appear when code isn't finished developing. For instance, a function whose implementation consists solely of todo!() will typically have such warnings about their parameters.

⚠️It is important to resist the urge to temporarily fix these warnings during development, by prefixing the bindings with _ or other tricks. You risk handing over the unfinished code if you artifically fix the warnings. Ignoring warnings has its place, but it ain't that!

Leaving a memo: the doc comment trick

It often happens while I'm coding that I think of something that needs fixing or handling, like an edge case or some incomplete/broken logic. In contrast with explicit error handling, this kind of issues is not tracked in the type system, so I usually leave a comment with a reminder to fix the issue, if I don't want a distraction from the logic I'm currently implementing.

This is fine and dandy, but I need to remember to actually go back there after the fact 😅

As a means to do this, I often use a trick where such comments are actually prefixed with a triple slash, as if they were documentation comments.

Rust warns about documentation comments that don't document anything, so putting a documentation comment above a statement generally achieve the goal of generating a warning associated with the issue described in the comment.

Where are my warnings? Limitations of vanilla Rust

These techniques served me well for years (I've been writing Rust for more than a decade now...), however I have my grievances with them.

Generally, the most aggravating issue is that not all of these techniques reliably generate a warning associated with the issue that needs solving before sending the code into production.

Forgetting to remove an unwrap and watching it burst into flames after the release is one of the worst feelings you can get as an engineer 😓

todo! is a frequent and weird offender in this category. As stated earlier, it often results in warnings, but these are indirect and caused by e.g. function parameters going unused or mutation not taking place when it should. Because of this, it can become a bit treacherous, as one is used to often have a warning when using todo! and rely on their absence to hit the merge button.

unwrap and clone are even worse, they never emit warnings, so you have to be extra careful in re-reading to remove them before the final PR.

An additional issue with them is that some of them are meant to reach production. With the "unwrap liberally" strategy outlined in this article, we are conflating the normal use case of unwrap with its "WIP" use case.

Lastly, "TODO doc strings" are kind of a hack, perhaps we could say it is a happy little accident? In this capacity, it has unintuitive failure modes, because doc comments are currently forbidden on expressions, so will cause a compilation error. Doesn't frequently happen, but we also run the risk of using the /// in locations where they actually document something, which isn't exactly nice to ship and won't have the expected warning.

Introducing the wip crate

In response to all these small pains for writing WIP Rust, I decided to develop a crate called wip to make it easier to implement the "warnings for later" strategy.

Mostly it consists of a collection of tools you can use in every context while you're developing Rust.

Using wip

Add wip to your dependencies like you would for any other crate:

cargo add wip

Optionally, you can import the prelude provided by wip in the files where you want to use wip:

use wip::prelude::*;

As wip relies on the deprecation warnings of Rust, I suggest you disable the feature of your editor to strike calls to deprecated functions, otherwise it becomes visually invasive.

wip::wip!, a warning-emitting todo!

The first tool provided by wip is a wip! macro, that works exactly like todo!, but always emits a warning on use. Use it everywhere you'd use todo!, to mark code as unfinished.

Why not todo!?

Earlier versions of wip used the todo naming, however it would prevent us from using the prelude, as Rust doesn't like it when a dependency attempts to shadow an std macro...

unwrap_wip and clone_wip

These methods are implemented as extension traits on Result and Option. When in scope, they work exactly like the regular unwrap and clone, except that they emit a warning on usage.

They also clearly signal that you don't mean for the unwrap/clone to stay in final code! Although, keep in mind it doesn't always end up happening that way...

wip::fixme!, a non-panicking macro for memos

One of the tools I use the most from wip is the fixme! macro. It is a non-panicking variant of wip!, that doesn't stand for an expression. It works the same as TODO comments, sans the hackish part and with a reliable warning.

Embracing our WIP future

I have been using the wip crate for a few of the my latests PRs, including some of my more unreasonably large ones. Usually I add the wip dependency to crates in the workspace during development, and then remove it when rewriting history before review.

It has been a liberating experience, squarely falling in the category of "I didn't expect to need it that much". It makes focusing on the happy path much more straightforward, while not forgetting things for later.

What's not to love?

Only drawbacks I can think of:

1. One needs to write enough context when adding a fixme, otherwise they can become hard to fix when coming back much later. This is usually not a problem, as wip are usually contained to (usually) short PRs by a single author (or pair programmed).
2. Sometimes the number of wip-induced warnings can become a bit discouraging, again especially on larger PRs. Perhaps avoid those 😅

Still, I'm quite sure I'll keep using wip. How do you use warnings during development? What do you think would be some useful additions to the current API?

t0-alpha is a 102M-parameter, open-weights probabilistic time-series forecaster that patches 32-step windows, uses a causal transformer, and predicts quantiles. It beats every classical baseline and lost to Seasonal Naive on only 1 of 97 task configurations. Strong zero-shot forecasting is here, but production value likely comes from calibrated probabilistic outputs, stricter leakage controls, stronger tuned baselines, and routing/ensembling rather than more transformer tweaks.

Claude Science, an AI workbench for scientists, is now available

AI has the potential to dramatically accelerate the pace of scientific discovery and the development of healthcare interventions. Since launching our efforts in the life sciences last fall, we’ve worked to improve our model capabilities, make connections to the scientific ecosystem via MCPs and skills, and launch partnerships in an effort to realize this potential.

Today, we’re introducing our most significant expansion of these efforts: Claude Science, an AI workbench for scientists. Claude Science is an app that integrates the tools and packages that researchers most commonly use, produces auditable artifacts, and provides flexible access to computing resources.

Introducing Claude Science

Scientific research is often tedious. Researchers must work across dozens of databases, each with their own schema, contend with file formats that require bespoke data pipelines and viewers, and transition between a roster of tools: PubMed, Jupyter, R, a cluster terminal, and more.

Claude Science brings these fragmented tools into a single research environment where scientists can conduct all stages of their work. It helps you analyze literature and execute multi-step research, produces detailed artifacts, and lets you iteratively refine figures and manuscripts until they’re ready for publication. Every output carries an auditable history of how it was made, so you can validate and reproduce the results. Like a Jupyter Notebook, you can access Claude Science wherever you already work—locally on macOS or Linux, or on a remote machine over SSH or with an HPC login node.

Users interact with a generalist coordinating agent with access to over 60 curated skills and connectors pre-configured for genomics, single-cell, proteomics, structural biology, cheminformatics, and more. These agents can spin up others and engage with specialist agents created by users. And a reviewer agent checks citations and calculations, flagging and correcting errors.

We are releasing Claude Science today in beta for Claude Pro, Max, Team, and Enterprise users, and will continue to refine the platform as we collect feedback from users.

How it works

Rich scientific artifacts, fully reproducible. Scientific research is inherently visual, so Claude Science generates figures and manuscripts alongside the code that created them. It natively renders rich scientific artifacts, including 3D protein structures, genome browser tracks, chemical structures, and more. You can chat with the agent about any detail, annotating figures and manuscripts in-line so the agent knows what to address to make them publication-ready.

When it generates a figure, Claude Science includes the exact code and environment that produced it, a plain-language description of how it was created, and the full message history. This allows you to understand the inputs, making the work easier to validate and reproduce even months later. You can ask Claude Science to make edits to figures in plain language—removing gridlines, for example, or changing an axis to log scale—and the agent will edit its own code.

Manages your compute and scales on demand. Large analyses—folding a protein, for example, or running a genomics pipeline over a massive dataset—often require researchers to shift their focus to setting up a computing job, waiting while it’s sent to a cluster, checking whether it succeeded or failed, and pulling the results back. Claude Science handles this process for you. It drafts a plan, asks before reaching new resources, and lets you review or revoke any decision before writing and submitting the job to the computing resources your lab already uses (your own HPC cluster over SSH, or your Modal account for compute on demand), scaling the analysis from a single GPU to hundreds as needed.

Because its agents work inside a running session that holds context in memory, even massive datasets only need to be loaded once. It runs on your lab’s own infrastructure—your laptop, Linux box, or HPC login node—so large or sensitive datasets never have to leave the systems they’re already on, and only the context needed for each step of the analysis is sent to Claude. As the pipeline runs, a reviewer agent inspects the outputs, flagging incorrect citations, untraceable numbers, and figures that don’t match their underlying code, and self-correcting as it goes. You can fork the session at any point to compare two approaches without losing the original thread.

Domain-ready on day one. Scientific knowledge is scattered across hundreds of specialized sources. In biology, for example, relevant data might sit across resources such as UniProt, PDB, Ensembl, Reactome, ClinVar, ChEMBL, GEO—each with its own schema and query language—as well as in journals and preprint servers, and domain-specific open models. When you ask Claude Science a question in plain language, specialist agents query and synthesize across all of these sources so you don’t have to navigate them individually. Claude Science uses the skills in NVIDIA’s BioNeMo Agent Toolkit to connect natively to the life sciences models and libraries in BioNeMo, including Evo 2, Boltz-2, and OpenFold3.

Scientists already have models, datasets, and pipelines they trust. Claude Science can connect to these as well, saving any pipeline as a reusable skill or accessing your lab’s preferred tool using a connector, with future sessions inheriting them automatically. This customizability allows you to access Claude, your proprietary data, and the validated tools you already rely on in one conversation. Claude Science benefits from our partners’ specialized expertise and platforms, while more scientists reach their tools through Claude.

What scientists are doing with Claude Science

Over the past few months, researchers have worked with Claude Science in beta for tasks like single-cell RNA sequencing analysis, CRISPR screen design, protein structure prediction, cheminformatics, and more.

Manifold Bio designs tissue-targeting medicines—which home to a specific organ or cell type, so the drug acts where it’s needed and spares the rest of the body—and tests how millions of candidate binders corresponding to hundreds of targets distribute through a living body at once. Manifold used Claude Science to nominate the targets for its latest experiments. For each tissue and target, Claude Science assessed surface expression, trafficking, and safety, ranking candidates against the criteria Manifold has learned from its own internal proprietary data. What set Claude Science apart from a general coding assistant, Manifold said, was that it could do this end-to-end, gathering the right data and applying the right judgment with the context of past programs built in.

Jérôme Lecoq, a neuroscientist at the Allen Institute, used Claude Science to build a multi-agent “computational review template” comprising about 20 custom skills geared towards writing long-form reviews. The sub-agents read through thousands of papers, pulling the central claim and the key quantitative finding, and storing them in an evidence state database. Then the pipeline constructs a narrative arc, writing the review section by section and delegating each to its own specialized sub-agent. Within each section, dedicated agents generate quantitative cross-study figures directly from the evidence database. A key component of the workflow, enabled by Claude Science, is the use of actor-critic pairs: one agent creates content while a separate reviewer agent evaluates it for accuracy and citation fidelity.

Before Claude Science, it could take Lecoq’s team as many as two years to write such a review. He now has about 10 reviews, many more than 100 pages, with citations that were checked over by reviewer agents. The team is now working with domain experts to further refine the AI-based critic agents.

And Stephen Francis, an associate professor and epidemiologist at the UCSF Brain Tumor Center, has used Claude Science to support studies on the molecular epidemiology of glioma, a type of primary tumor that begins in the glial cells of the brain. His lab investigates the genetic basis for how thousands of small-effect germline variants combine to shape individual susceptibility. Although this work predated Claude Science, Francis said the app has dramatically accelerated the analysis, enabling comprehensive germline workups across multiple approaches in roughly one-tenth the time it previously took. His group independently validated Claude Science’s results, confirming that it can produce both rapid and robust analyses.

Getting started with Claude Science

The Claude Science app is available in beta on macOS and Linux for Pro, Max, Team, and Enterprise plans. We’re sharing it early so scientists can start to use it on real problems and tell us how to refine it.

Team and Enterprise users will need their admin to enable Claude Science. We now have a Team plan offering discounted seats for active scientific labs at academic institutions and nonprofit research organizations.

We’ll also be supporting up to 50 Claude Science AI for Science projects, providing up to $30,000 in credits. Modal will also be providing up to $2,000 in compute for select projects. We are looking for projects that span domains and explore the boundaries of science, with an early focus on biology and biomedical research. Applications are open through July 15, 2026, with award notifications sent out by July 31. Projects will run from September 1 to December 1, 2026.

To stay up-to-date on product announcements, provide feedback, and learn from others in the Claude Science community, join the AI for Science Discourse community.

Get started with Claude Science at claude.com/science.

After giving it an Apple Design award for innovation, Apple has now bought Rabbit 3 Ties, Inc, which made a visual Swift development tool called Play.

Following Apple's acquisition of the open-source Swift Package Index, it's now been revealed that it entered a deal with the Play company. Rather than an outright acquisition, the deal reportedly sees Apple acquiring assets and having the option to hire certain staff.

So where Swift Package Index, for one example, is expected to continue in its present form, Apple appears to have bought Rabbit 3 Times to asset-strip it. It may also be an acquihire, where Apple bought the firm in order to get its staff for other projects.

Based in New York, Rabbit 3 Times was founded in 2021, officially in Delaware where state laws are particularly favorable to businesses. Since then, it has been producing and selling a visual Swift tool for iOS and macOS.

In a cross between Shortcuts and Xcode, Play: Create Better Apps was a free tool that let developers quickly mock up Swift projects and immediately see what they would look like in use. Developers could then export Play projects to Xcode through a paid service.

The development tool won an Apple Design Award for Innovation in June 2025. According to the European Union, Apple reported the deal in February 2026.

Shortly afterwards, Rabbit 3 Times announced that it would cease supporting its Play apps for iPhone and Mac starting April 20, 2026. Conceivably, Apple could now incorporate Play into its Apple Creator Studio, but so far it appears that the app is gone.

The paid Play to Xcode service was made free after Apple's acquisition, specifically "to help with the transition."

This change was announced on the firm's website, which has now been taken down. "We're working on something new," was the only detail the announcement gave of why things were changing. "It has been an incredible journey," the company said.

Acquisitions that are of a particular scale or are considered to be significant to EU users, are required to be reported under the Digital Markets Act (DMA). The EU then publishes these reports, although no sooner than four months after filing.

The EU's listing describes Play/Rabbit 3 Times as offering "iOS and macOS tools for designing, prototyping, and generating SwiftUI code in real-time."

As part of an ongoing legal dispute with three Hollywood studios, AI startup Midjourney is seeking to compel those studios to reveal how they use AI themselves.

Disney and Universal sued Midjourney for alleged copyright infringement last year, noting that the startup’s models could generate images of characters, such as Bart Simpson and Darth Vader, which are owned by the studios. A few months later, Warner Bros. sued Midjourney as well.

The startup argues that training its AI models on images of copyrighted characters is permitted under fair use.

The current dispute revolves around the documentation the studios will need to produce during the discovery process. A judge previously ruled that the studios would indeed have to provide information about their generative AI usage — but only when it led to “consumer-facing” videos and images.

In its latest filing, Midjourney seeks to overturn that limitation, arguing that it “unfairly” allows the studios “to cherry-pick only those documents they believe support their market harm claims while depriving Midjourney of documents that would support its defenses.”

Midjourney goes on to claim that the “documents [the studios] are withholding are precisely those that would reveal whether, behind closed doors, they are doing exactly what they are suing Midjourney for doing.”

For example, the startup says that if the studios are developing image-generating AI models “for internal use in storyboarding or ideating content for film or TV, that evidence would equally demonstrate that it is an industry custom, even among the studios themselves, to download and train AI on unlicensed copyrighted content.”

In the filing, the startup also argues that the studios should reveal all the prompts they used in Midjourney, as well as the resulting outputs, not just the prompts that produced the allegedly infringing images.

The studios’ lead attorney David Singer previously claimed Midjourney was seeking this documentation as part of a “fishing expedition.”

He also said the studios “do not seek to stop AI technology or even shut down Midjourney’s business,” but rather “simply want Midjourney to stop copying their movies and TV shows and to stop distributing, publicly displaying, publicly performing, and creating derivative works that include copies of [their] famous characters without authorization.”

The difference between the map and the territory is the unknown. When Claude runs into an unknown, it needs to make a decision based on its best guess. The more work that is done, the more unknowns Claude might run into. This post discusses how to use Claude to help discover unknowns. Discovering your unknowns before starting a project is a cheap way to find out what you didn't know before it gets expensive to fix.

One of the biggest challenges for robot makers is ensuring that they don't hurt humans. The machines are getting bigger and heavier, which increases their potential for damage. The International Organization for Standardization is expected to publish a standard for humanoid robot safety by mid-2028, but robot makers are already coming up with their own solutions. This article looks at some of those solutions.

WordPress now powers 41.5% of the web, down from a peak of 43.6% in early 2025.

GitBiased gives engineering teams a customizable dashboard for GitHub activity, bringing PRs, CI checks, issues, DORA metrics, deployments, and weekly summaries into one shared view.

Apple is reportedly testing four new 11-inch and 13-inch iPad Pro models for a spring 2027 launch, focusing mainly on internal upgrades such as faster chips and potentially vapor chamber cooling, while also planning a redesigned 14-inch entry-level MacBook Pro that could arrive as early as the first half of 2027. Bloomberg also says Apple is accelerating the launch of its M7 chip to better handle AI workloads, following the upcoming M6 generation. Although memory and chip shortages could still affect the roadmap, these devices are expected to contribute to what could be Apple's biggest product year yet in 2027.

Many digital products are designed around an unrealistic "ideal user" who patiently adapts to confusing interfaces, causing people to blame themselves instead of poor design when something goes wrong. Inclusive UX writing—using clear, supportive, non-judgmental language—reduces cognitive burden for everyone, especially people with disabilities, anxiety, ADHD, or language barriers, reflecting the curb-cut effect, where designing for those at the margins improves experiences for all. Every piece of interface copy is a design choice, and thoughtful language can make users feel guided rather than judged.

Genspark is your all-in-one AI workspace. Slides, docs, images, video, code, and design — all in one place.