Google will invest at least $10 billion in Anthropic, and that amount could rise to $40 billion if Anthropic meets certain performance targets, Bloomberg reports.

The investment follows Amazon's $5 billion initial investment in Anthropic a few days ago; the Amazon deal also leaves the door open to further investment based on performance. Both investments value Anthropic at $350 billion.

Anthropic has seen rapid growth in the use of its Claude models and related products, such as Claude Code, which promises to significantly increase the speed and efficiency with which companies or individuals can develop software. (The reality varies from big improvements to setbacks, depending on the nature of the project and company, how Claude Code is used, and many other factors.)

Several factors contributed to Anthropic's success in recent months, including controversies around OpenAI and its ChatGPT product and models, more robust agentic workflows, and new products like Claude Cowork, which does some of the same things for general knowledge work tasks as Claude Code does for software development.

The result has been a dramatic increase in demand for Anthropic's services, leading to outages and other problems. Anthropic has been testing solutions to reduce demand, like imposing limits during peak hours, or exploring removing some of the most compute-intensive tools from cheaper service plans.

These investments are meant to help close the gap between demand and supply of compute for Claude Code and its ilk. Amazon and Google are providing chips suitable for AI training and inference and cloud compute capacity to help Anthropic scale up quickly.

This has become a common scheme for investment in AI companies like Anthropic; established companies like Microsoft have products and services that can help new AI companies like Anthropic scale, so the former invests in the latter so the latter can, in turn, pay for the former's products and services.

This is not the first time Anthropic has received investment from Google, even though Google is ostensibly competing with Anthropic over AI models.

Andon Labs is running an experiment to see whether AI agents can run real-world endeavors. It opened a retail boutique on April 10 run by an agent named Luna. Luna has so far struggled with employee schedules and seems to be unable to stop ordering candles. The experiment's mission was to make a profit, but it has lost $13,000 since the shop's opening.

Anthropic has released the Memory feature for Claude Managed Agents, now accessible in public beta. This allows developers and enterprise teams to have agents remember and use information from prior sessions, making it possible for agents to accumulate knowledge over time without requiring manual prompt updates. Memory is designed as a filesystem-based layer, meaning data is stored as files that can be exported, managed through APIs, and scoped with permissions for various organizational needs.

The release is aimed at developers, enterprise customers, and technical teams using Claude Managed Agents. The feature is available immediately in public beta to all users of Managed Agents, with access through the Claude Console and programmatic interfaces. It supports a range of platforms by integrating with the existing Claude agent infrastructure.

Anthropic's approach with this release centers on transparency and control. All memory changes are logged, with audit trails for each session and agent, giving organizations granular control to roll back, redact, or manage data. This sets it apart from earlier versions and competitive offerings that may not provide the same level of programmatic control and auditability. Early adopters such as Netflix, Rakuten, Wisedocs, and Ando are already leveraging memory to streamline workflows, reduce errors, and accelerate processes. Industry observers note that the ability for agents to build memory over time could shift how companies automate complex workflows and manage organizational knowledge.

Anthropic, the developer of Claude, is recognized for focusing on enterprise-grade AI tools that prioritize safety, transparency, and developer control. This release aligns with their strategy of offering advanced agent capabilities for businesses seeking robust, auditable AI solutions.

Google appears to be preparing a major shift in how consumers interact with the Gemini app, with new strings referencing usage limits surfacing in the latest build. The signals point toward a credit-based system coming to the core chat surface, where users would receive a monthly allowance to spend across models and features, with the option to top up when they run out. Currently, Gemini relies on fixed prompt quotas and time-bound caps tied to each subscription tier, while Google's credit mechanics have been confined to Flow, Whisk, and Antigravity, plus top-ups available to AI Pro and AI Ultra members.

Extending credits into the main Gemini app would bring Google closer to the flexible consumption model already in place at OpenAI, Anthropic, and Notion, and xAI is expected to follow suit with the Grok Build rollout. For power users, the change would mean more predictable budgeting for heavy workloads, particularly those involving agentic tasks, Deep Research, Deep Think, or long multimodal sessions. It would also give Google a cleaner lever to introduce premium features without forcing users to make a steep jump from AI Pro at $19.99 to AI Ultra at $249.99.

Alongside the credits signal, a dedicated images section has appeared in the web UI, labeled NEW. At this stage, it is unclear whether it simply provides a distinct home for image generation, teases an updated model, or points to a more comprehensive image editor built directly into Gemini. Google had a burst of activity on this front in late 2024 with Whisk and ImageFX, but that track went quiet before the recent consolidation into Flow. A proper in-app editor within Gemini, pairing Nano Banana 2 and Nano Banana Pro with canvas-style tools, would mark the return of that project to the core product rather than a standalone Labs surface.

Strategically, this fits a broader consolidation underway at Google. Developer program perks have already been folded into AI Pro and Ultra, consumer subscriptions are linked to AI Studio credits, and the company is unifying its billing spine. A shared credit pool covering the Gemini app, AI Studio, Antigravity, Flow, and a revived image editor would be the logical next step, especially with Jules, Gemini CLI, and the rumoured Gemini desktop app moving toward coding-heavy workloads that demand heavier compute budgets. Timing favours Google I/O on May 19 and 20 as the likely unveil moment, alongside the Stitch redesign, Jitro, AI Studio Build expansion, and the broader Skills rollout.

This post is my personal opinion based on my testing and observations. I'm pretty confident in my test methodology, but William O'Connell is human and can make mistakes, check important info, etc.

How much of your code is AI? That question would've been gibberish to me five years ago, but of course the last few years have seen an explosion of "AI-enhanced" IDEs and other software development tools. Software companies are spending huge sums of money to provide these tools to their staff, and rapidly cycling through them as the space continues to evolve.

I don't make heavy use any of these in my personal life, but I have gotten to try a handful of them through various employers. One such tool is Windsurf, a VSCode fork that most people know as the one they assume shut down after Google bought out their key leadership last year. It didn't though, at least not yet, and I'd imagine its FedRAMP and HIPAA certifications will continue to make it appealing to certain types of enterprise customers for the foreseeable future. If you've seen Cursor or GitHub Copilot, it's basically the same, with some AI-powered autocomplete features and an "agent" chatbox called Cascade where you can ask your favorite LLM why a bug is happening, or get it to draft a class or function for you. In theory these types of agents can develop features and even whole applications on their own, but in my experience the results are pretty inconsistent, so I tend to stick to simpler requests.

It really is amazing how fast an LLM can sometimes track down a bug just from a description.

One thing that's very important to any enterprise rolling out a tool like this is metrics:

• Are employees using it?
• How much time is it saving?
• Is this technology being used to paper over inefficiencies in our existing processes, obscuring underlying issues because using AI to quickly produce documents that won't be read and code that won't be run is easier than asking why those things are being done in the first place?

Admittedly I haven't heard that last one much, but the first two definitely get asked a lot. To help with this, Windsurf offers a dashboard of analytics at both the individual and team level. It includes things like the number of autocomplete suggestions accepted, the number of messages sent to Cascade, and which models are being used the most. It also includes a metric called "% new code written by Windsurf" (or sometimes "PCW"), which they seem quite proud of, since it gets top billing on the dashboard and they wrote a whole blog post explaining it.

The pitch is pretty simple: how much of the code did a developer write by hand, and how much did they generate with AI? When I first learned about this feature my guess would have been 10, maybe 20% AI, depending on the project and whether you include unit tests (LLMs are pretty good at those). So you can imagine my surprise when I opened the dashboard and saw this:

Don't worry employers, I didn't screenshot my work computer. This is a recreation.

Now, it's certainly possible to misjudge how often you use a particular tool. If the number had been 40%, or even 50%, I wouldn't have been that shocked. But 98%? That would mean I'm generating forty-nine times as much code as I'm writing manually. If that were true wouldn't I have run through my token budget by now? Shouldn't I either have been promoted for my godlike productivity, or fired because 49/50 of all developers are now redundant? You'd think, but Windsurf says this result is pretty normal:

"...customers should expect PCW values of 85%+, often 95%+. This is not a hallucination and is accurate given how we compute this metric, though there are a number of caveats that we will cover later in this section."

"Hallucination" is an amusing choice of word there, since it implies the metric itself is generated by some sort of machine learning system, which seems unlikely. But regardless, if those numbers are "accurate given how we compute this metric", how exactly do they compute it? To their credit, they go into a fair bit of detail:

"To compute PCW, we take the number of new, persisted bytes of code that can be attributed to an accepted AI result from Windsurf (i.e. Tab suggestion, Command generation, or Cascade edit) and the number of new, persisted bytes of code that can be attributed to the developer manually typing. ... We take these measurements whenever a commit is being made. This way if the AI added a lot of code but the developer deleted a lot of it before committing the code to the codebase, then we are not incorrectly inflating the W number. Similarly, any bytes of code that come from the developer manually editing an AI result will get attributed to the developer (D) as opposed to Windsurf."

That all sounds pretty reasonable, but I was still skeptical of the number I was seeing. I wanted to know for sure where that 98% was coming from, and what it actually meant. So I signed up for a personal Windsurf subscription, installed the editor, and ran some tests.

The Math Behind the Curtain

My original plan was to use mitmproxy to watch the outgoing network traffic from the IDE, and see what numbers it was reporting as I took different actions. That turned out to be easier said than done though, because Windsurf is quite chatty on the network, sending many requests to various domains while in use, and even pretty often when I'm not touching it at all.

Additionally, Windsurf makes heavy use of protobuf, a data encoding scheme that I'm pretty sure Google invented to annoy me personally, because it makes it much harder to interpret and debug the traffic between clients and servers. If you don't have the associated definition file, a protobuf message is basically just a list of simple values (int32, bytes, etc.) with no human-readable labels. Because of this it was hard for me to tell which messages were related to the PCW metric, or what exactly they were communicating to Windsurf's cloud backend.

Luckily, I found an easier way. It turns out that even though the dashboard says "Analytics update every three hours", it actually shows new data almost instantly. And while the UI only shows the overall percentage, the response from the web server actually includes some additional data. It's protobuf as well, but since it's a webpage the source code is all immediately accessible, and of course the frontend code includes a copy of the message definitions so it can make use of the data.

So I was able to decode the GetAnalytics response and pull out these fields (among others):

• user_bytes
• codeium_bytes
• total_bytes
• percent_code_written

Windsurf used to be called Codeium, so clearly that one represents the AI-generated bytes. And as you'd expect, the percent_code_written is equal to codeium_bytes / total_bytes. So far so good, but what causes those values to change?

Windsurf says they take measurements "whenever a commit is being made", but that doesn't match my testing. Whether the folder I'm in has a git repo set up or not, as soon as I make additions to a file the user_bytes value increases, and if I delete some of those lines it decreases. Whether I do a commit (using Windsurf's git UI) between those two actions makes no difference as far as I can tell. What does make a difference is restarting the editor; it seems to forget the history of how each line was generated, so deleting code I wrote before the restart doesn't deduct from user_bytes, and deleting code Cascade wrote before the restart doesn't deduct from codeium_bytes. There is a line in the PCW article that alludes to this ("We currently do not have instrumentation to measure PCW across sessions"), but obviously that's a pretty major gap in functionality, and it doesn't actually address why the described git integration appears to be nonexistent.

To test how exactly the byte counts are being computed, I performed a few tests where I took specific actions and checked how much each value had increased. To keep things simple I disabled the AI autocomplete features (which I find more distracting than helpful anyway) and just focused on the Cascade chat experience. I created a file, human_file.js, and I typed out a single line:

console.log('This line was written by a human.');

49 characters exactly. Then I told Cascade to create a second file (ai_file.js) and to write a similar line of the same length.

The result:

user_bytes: 855 -> 901 (+46)
codeium_bytes: 7387 -> 7437 (+50)

So the system did seem to be working, but we have a discrepancy right off the bat. The line is definitely 49 characters (50 with a newline at the end), so why is user_bytes only reporting 46? Well this is where some technicalities start to emerge. Windsurf says that they measure "code that can be attributed to the developer manually typing". The Windsurf editor is a lightly modified version of VSCode, and like most code editors, VSCode has a feature that automatically adds closing symbols (end quotes, closing parentheses, etc.) without the user manually typing them. I suspect that because those characters are being added by that feature, they're technically not "the developer manually typing", and therefore are not counted.

If that's what's going on, then in my opinion that's already a pretty serious knock against the reliability of Windsurf's metrics. Counting closing symbols when the LLM outputs them, but not when VSCode auto-adds them, obviously biases the stats to increase the percentage of code attributed to AI (even if the effect is fairly slight). As it turns out, there may be some not-so-slight biases as well.

Continuing my test, I wrote out a simple function, and asked Cascade to write a similar function in its own file. Finally I copy/pasted Cascade's function into the human file, and asked Cascade to copy my function into its file.

Here's the final tally:

user_bytes: 1054 (+199)
codeium_bytes: 7807 (+420)

So for this session, Windsurf is reporting that Cascade generated more than twice as much code as I wrote, even though we each produced an almost identical file. I never touched ai_file.js, Cascade never touched human_file.js, and the two files are the same length (actually human_file.js is 21 bytes longer because Cascade used Unix-style line endings). Yet somehow my PCW for this session would be around 68%. The trick here is that much like with the auto-added closing symbols, it seems like any text the user pastes doesn't count towards user_bytes. I guess from a certain perspective that could sound reasonable (if you pasted code from StackOverflow you didn't really "write" it), but the way it plays out in practice quickly becomes absurd.

In another test I hand-wrote two functions in a single file, then moved them both to a second file (as one might do when refactoring). For the first I cut and pasted, for the second I asked Cascade to move it for me. The result? Cutting the first function deducts it from user_bytes, and pasting it doesn't count for anything. Cascade deleting the second function also deducts it from user_bytes, but the lines added to the new file count towards codeium_bytes. So even though I did 100% of the writing and 50% of the refactoring, Windsurf reports that 100% of the code I produced in that session was generated by AI.

In my opinion these biases make Windsurf's PCW metric basically useless. By being so picky about what counts as a human contribution, and being as generous as possible to the LLM, Windsurf (intentionally or accidentally) tips the scales towards reporting absurdly high percentages, regardless of where most of the code is actually coming from or whether it eventually gets committed.

Who Else?

So that seems... bad, but of course Windsurf is just one of many AI-enhanced IDEs out there (and it's owned by Cognition, makers of Devin, who don't have a stellar track record). What about the other products on the market? As far as I can tell Google's Antigravity editor doesn't have any comparable metrics. GitHub Copilot does provide stats on how many lines of code it generated, but not as a percentage of the total. Amazon Kiro is the same. I did find one popular editor with a metric similar to Windsurf's PCW though: Cursor, with its "AI Share of Committed Code". So how does it stack up?

Sadly Cursor only offers analytics on their business-focused "Team" plan, making this one of my costlier blog posts, but I'll do almost anything for science. Right off the bat things are looking better, with a more nuanced and considered description of their measurement approach:

"Cursor keeps a log of the signature of every AI line (Tab or Agent) that is suggested to the user during their chat session. These lines are stored and later compared to the signatures of each line in subsequent git commits that were written by the same author. ... We use the following definitions: Cursor AI: Any line that can be attributed to Cursor Agent or Tab based on diff signatures. Other: Any line of code that can't be detected as being written by Cursor"

So rather than splitting hairs about the various ways a programmer can add text to a file, they simply divide the total lines in a commit into "AI" and "Other". Sounds great, but does it work?

Well, the git integration certainly does. While Cursor does also use protobuf, it's easy to tell that it's sending an event called "ReportCommitAiAnalyticsRequest" whenever I do a commit, and that message clearly includes information about the different files and what seem to be the line ranges produced by different methods. We can also see the results on the Cursor website, though it takes a while for them to appear. Running my same test from before, we get a much more reasonable result:

I'm not sure why the bar graph doesn't go to 100%.

Certainly a lot closer than the 67.9% that Windsurf reported. I'm actually not sure what caused it to report 20 AI lines vs 18 "other" lines; I did the test as several separate commits and the IDE commit history shows the first commit adding 1 line to each file and the second commit adding 20, so that should be a total of 21 for both. I did manage to capture the protobuf message the IDE sent for the second commit, and it seems to be showing (correctly) that lines 3 through 21 of ai_file.js were written by the Composer 2 model, and 3–21 of human_file.js were added manually.

Thanks to pawitp for this handy protobuf decoder tool.

So I'm not sure why a few lines seem to have gone missing, but regardless the behavior does more or less match what I'd expect from Cursor's description.

Unfortunately, the line-based approach has other flaws that don't show up in this test. For instance, I pasted in a (bogus) 100-line JavaScript file, and then told Cursor to change all the double quotes to single quotes (updating escape characters where necessary). Some might argue that that's an overly simple task to delegate to an LLM (as opposed to an IDE or linter feature), but with some companies giving employees basically unlimited token budgets, and the very low cost of some of the cheaper flash/nano models, I don't think it's that unrealistic. As you'd expect, Composer 2 handled it flawlessly, touching 49 of the 93 non-blank lines in the file.

The main difference between Windsurf and Cursor seems to be color saturation.

The gotcha here is probably pretty obvious. I was expecting to say "see, I added this code manually, but now that Cursor has changed the quote marks it counts all the lines containing quotes as AI-generated". That wasn't what actually happened, though.

Somehow, Cursor counted the entire file as AI, even though we can see from the diff that it left plenty of the lines unchanged. And remember that the entire file is exactly 100 lines long, including some blank ones, so it's not just a case of excluding lines that are considered too simple to be counted. My best guess is that the system that tracks which lines were added by the AI is designed to work with contiguous blocks of code (like drafting an entire function), and if there are too many gaps in the generation it just gives up and calls the whole thing one AI block.

Regardless, this is another case where the AI tool seems to be claiming credit for 100% of the code produced, even though arguably zero lines of code were actually "AI generated", and many of them weren't touched by the tool whatsoever. It looks like both IDEs sometimes wildly overestimate how much they're being used in a coding session.

Weights and Biases

One takeaway here is that it's just very hard to measure the contribution LLMs make to a codebase. Sometimes the best use cases are inquisitive prompts like "Is there already a different solution to this elsewhere in the codebase?" or "Are there any edge cases this logic doesn't cover?", which don't necessarily produce any code at all. On the flipside, I'm a big believer in a philosophy expressed concisely by Jack Diederich:

"I hate code, and I want as little of it as possible in our product."

Measuring the value of an LLM by the number of bytes or lines it produces has all the same problems as measuring developers that way; adding a lot of code doesn't necessarily mean you're adding a lot of value, and sometimes the hardest and most productive work is cleaning up and simplifying what's already there. Besides, when a developer is making heavy use of tab complete, etc. there's not always a clear-cut answer to "was this line of code written by AI", even if you were looking over their shoulder as they wrote the file. So perhaps it's foolish to expect algorithmic answer to that question.

Still, it's notable that the bias always seems to be towards reporting a higher AI percentage. Whether that number is truly meaningful or not, "what percent of my team's code did Windsurf write" is a very appealing statistic for a manager or executive. Execs love announcing that 30%, 75%, even 100% of their code is AI-generated. And of course high numbers are great for AI companies, because they underscore the value they bring to software teams and help justify their high subscription costs. But as a developer, skewed metrics can be harmful. If 50% of my team's code is AI-generated, will management expect features to be implemented twice as fast? If 90% is AI, do we even need a team?

Again to their credit, Windsurf does push back on that type of thinking in their blog post:

"Writing code is not the same as software development. This is only capturing some level of acceleration while writing code, and does not capture time taken in architecture, debugging, review, deployment, and a number of other steps."

To be sure, all metrics are only as good as your understanding of their limitations. If everyone internalizes that these percentages should only be used to compare trends over time, with the absolute values being essentially meaningless (and not comparable across tools), then maybe the details of how they're computed don't matter. But a sentence like "98% of our new code was written by Windsurf" creates a gut feeling that's hard to talk yourself out of, even when you know there are caveats. And I wonder if the impact of these stats could go beyond press releases and 🚀-laiden Slack posts. Since code is protected by literary copyright, and AI-generated works aren't copyrightable, the legal team might get nervous when they hear that the vast majority of their company's code "can be attributed to AI".

Ultimately, I don't really know what percentage of the code I commit is from an AI model. I don't know what the "correct" way to calculate that would be, or if it's worth calculating at all. I'm confident that these tools save me some amount of time, but I also know it's easy to overestimate how much. What I am certain about is that these vendors have a lot of money riding on whether or not AI is fulfilling its grandiose promises; massively accelerating strong developers and completely replacing weak ones.

Perhaps it is, but I'm not going to trust them to measure it.

Monitoring LLM behavior necessitates adopting the AI Evaluation Stack, separating tests into deterministic assertions (syntax and routing integrity) and model-based evaluations (semantic quality). Engineers use offline pipelines for pre-deployment regression testing with human-reviewed "Golden Datasets" while online pipelines monitor real-world performance for drift and failures. A continuous feedback loop from production telemetry ensures AI systems adapt, maintaining high performance as user behavior evolves.

The World Can't Keep Up With AI Labs

Late last year a new AI psychosis kicked off. This time it was coding agents.

People started saying this is a new era in programming, blah blah blah.

Karpathy tweet, late winter

A few months later, we've got more than just claims. We've got numbers. And they say something unusual is happening in the market.

Coding agents are the first AI product people are paying for at volume and regularly. Because it directly speeds up their work. It's too early to claim businesses are replacing whole processes with agents across the board. But compute demand has started growing faster than anyone can build it out.

Here's why this moment is different, why nobody's ready, and what I took from it personally.

The Numbers

OpenAI and Anthropic might go for an IPO soon. That's why they're eagerly posting how fast their revenue is growing.

And it's a ton of money.

Anthropic is up 3x since the start of the year. And they're already a big company. This is impressive, because the bigger you are, the harder it is to keep growing at the same pace.

OpenAI on the left, Anthropic on the right.

Even during past boom moments, nobody hit numbers like these (with a caveat, see below). Zoom during the pandemic, Google at IPO, Coinbase cashing in on commissions during the crypto hype. These are companies 5-10x smaller than Anthropic, in special situations, and they still grew slower!

The best growth years for big companies. Only ones that were already large. Revenue measured at start vs end of year.

The caveat. First, vaccine makers during the pandemic were also up there. Second, Anthropic's numbers are a projection for the rest of the year based on early data. And they count things a bit differently than OpenAI. None of that changes my conclusion, which is..

Cash is a solid tell for real demand for agentic systems.

Last year when a bunch of people suddenly figured out ChatGPT could generate cool images, that didn't translate into serious money.

Meanwhile, in January alone, Claude Code commits on GitHub (in publicly accessible repos) went from 2% to 4%. If that sounds small, keep in mind it's one month, and that's without Codex, Copilot, or Devin. By end of year Dylan Patel forecasts Claude hitting 20%+.

Claude commits on GitHub.

Even if a $100 subscription only automates a small slice of the work, that's nothing compared to a developer's salary. For a median developer at $350-500 a day, the subscription has 10-30x ROI if it handles just the simplest, most routine 10% of their work.

There's plenty to argue with here.

Let me even lay out the weak spots in my own logic.

So their revenue is growing, fine—the labs are still unprofitable as businesses. They have every incentive to pump the hype to pull in the most risk-tolerant companies. The ones paying are early enthusiasts, not big companies. And enthusiasts come and go. Plenty of bubbles have popped exactly this way.

Agents are unstable and still randomly screw up. Who's to blame when things go wrong? You can't replace humans yet, because serious businesses care about reliability. And where do senior engineers come from without juniors if you stop hiring?

Agents only handle a narrow set of tasks well. Even if writing code is faster, shipping a product still gets bottlenecked by gathering requirements, architecture, review, testing, and our beloved stakeholder zoomcalls and compliance.

I decided at some point you have to commit and pick a side, even without conclusive evidence.

The finish line can be moved forever. There was a time when reasoning was completely out of reach for ML models. Same for decent image generation, or speech that didn't sound like a robot. There was a time nobody believed machines would learn to play Go. You get the idea.

Metaphor from Tegmark's Life 3.0. Computers gradually learn harder and harder tasks. Over time there's less and less they can't do. Like water filling a map from the bottom up.

Ilya Sutskever, back when he was still at OpenAI, often mentioned an internal meme—Feel the AGI.

He was one of the first to believe deep learning would gradually change our lives. Yes, there's a lot we don't know, but everything keeps moving in that direction, and that matters. Everyone gets it at their own moment. When a neural net does something you usually do yourself, manually, that's a special feeling.

I've lost count of how many of those moments I've had in 10 years of following neural nets. So I'm not interested in the bubble-or-not debate anymore. I'm interested in watching the water level rise.

Personally, I have enough evidence that agents can now do valuable work that companies are willing to pay for.

And the thing is, demand has plenty of room to grow. Agents often don't work out of the box. You have to adapt to them, and the fastest and most curious people do that best. Everyone else will catch up bit by bit.

The Industry Isn't Ready For This

To avoid talking about "the industry" in the abstract, let me split it into 3 layers.

• AI labs make models. OpenAI, Anthropic, DeepMind.
• Hyperscalers build datacenters. Google, Amazon, Microsoft, Meta.
• Chipmakers make chips. Nvidia, TSMC, ASML.

And at every layer, companies are scared.

People online love talking about bubbles. Turns out, all these companies are well aware bubbles happen. And to avoid going bankrupt, each one is cooking up its own workaround.

Dario Amodei says he builds the company's plans off a pessimistic revenue scenario. Funny thing is, this year they're already beating that by 1.5x. And only 3 months of the year have gone by. They're beating the optimistic scenario too.

Dwarkesh asked him straight up in an interview: why? Dario genuinely believes in massive future upside from AI. He writes long essays about it, pitches a country of geniuses in a datacenter. And yet he doesn't want to bet everything on that future.

Dario says it's risky because of a cash flow gap in the business model.

Here's how it works. They provide neural nets to users. They pay hardware owners for inference and make money from subscriptions and APIs. In parallel, they pour money into research on the next generation model. Which won't start making money for another year or two.

They regularly spend more than half of revenue on research.

You're not just balancing income and expenses—you're also balancing investment in future growth. If you invest big and the growth doesn't show up, you're in serious trouble.

Anthropic has been running in this mode for three years straight. Growing 10x every year. Dario figured 2026 would be when it ends. Because the bigger you are, the harder it gets. You are gonna slow down at some point.

What he didn't mention in the interview, is that their margins are growing slower than forecast. Costs are growing multiple times faster than they'd planned.

Dario says he wants to push the company into profitability in a few years. To do that they need to improve margins. That means slowing growth and investing conservatively, only on the most efficient things.

The logic adds up. But slowing down isn't really working. They look ready to 10x again this year. But the resources to support that aren't there.

Anthropic doesn't have enough compute for this many power users.

They rent GPUs from hyperscalers. And they can't just walk into a datacenter and ask for more. Because the datacenter owner is also exposed to bubble risk. So capacity is booked out in advance.

For Anthropic to make $30B a year, someone had to spend $80B on infrastructure. Betting it would pay off in a few years.

Amazon will spend around $200B this year, Google $180B, Meta $125B, Microsoft $105B. That's a setup for trillions in economic value in the coming years.

And a cash flow gap risk if the value doesn't materialize.

The industry is one long value chain. Everyone in it tries to lower their own risk by locking expectations into contracts. Which reduces the whole chain's ability to react to surprises. Like the sudden arrival of coding agents.

So every year labs hit some new bottleneck. And constraints keep sliding further upstream, toward players further from the end user. Because their risks are higher and their contracts are even less flexible.

A New Bottleneck Every Year

In 2023 everyone was chasing GPUs. More specifically, TSMC factories didn't have enough capacity for the final chip-to-module assembly (CoWoS). In 2024 came the HBM memory shortage for those same modules. In 2025 GPUs got better, but datacenter buildout became limited by power supply. In 2026 it turned out even when you have the power, the US grid can't deliver it to datacenters at the volume needed.

1 - Memory

Modern models need more memory than before. I mentioned earlier that companies spend hundreds of billions a year on infrastructure. Roughly 30% of that goes to memory.

And they have to buy expensive HBM instead of cheap DDR. Because high bandwidth reduces GPU idle time while memory processes its part.

Turns out memory is the most expensive thing in a GPU.

Memory prices are probably going to keep rising unless someone figures out how to work around it. They could easily go up another 2-3x, because SK Hynix and Samsung control 90% of the market. And memory demand is only growing.

2 - Energy and Datacenters

xAI proved datacenters can be built pretty fast.

But they eat power like a small city. And when such a thing suddenly shows up in some region within six months, the electricity grid just can't handle that.

Surprisingly, Dylan Patel isn't that worried about energy. New power plants, transformer stations, and plain old transmission towers take a long time to build. But while the grid catches up to the new load, you can power datacenters off industrial gas turbines. Literally roll up to the datacenter with a dozen trailers full of generators and you're good (tho people start to worry about that being far from clean energy).

There are also piston engines, solar with batteries, hydrogen reactors, marine ship engines… Basically, every trick the fuel industry has invented in its entire history. Together with more efficient grid usage, that can add up to hundreds of gigawatts.

Right now GPUs alone consume 13GW. Add the rest of the datacenter and you can multiply by 2.

The blocker for building datacenters and reactors fast is a shortage of skilled labor, especially electricians.

So, expensive and labor-intensive. But turns out it's still easier than the semiconductor supply chain.

3 - Semiconductors

There are factories (mostly TSMC) that assemble GPUs of a specific era (based on designs from Nvidia or Google). For example, on the 3-nanometer process.

And there just aren't enough factories built.

This can't be fixed quickly because these are some of the most complex industrial facilities on the planet. Building one takes 2-3 years and a pile of specialized equipment and chemistry.

The hardest piece is the lithography machines (EUV scanners). They're needed to etch chips onto wafers. The wafers then get paired with memory into modules, and that's how you get a GPU.

These machines cost ~$350M each. Only one company from the Netherlands makes them—ASML. Around 50 machines a year.

The machine.

By a rough estimate, by 2030 there will be around 700 of them worldwide. That's on the order of 200 gigawatts of compute. And at the end of 2025 we were using ~27 gigawatts. Note that that's before the agent hype of early 2026.

So there's room to grow, but the shortage will be permanent—bottlenecked by factory construction, wafers, and lithography machines.

These are the kinds of constraints you can't just throw money at, unlike memory and datacenter energy.

You can see it clearly in Google's behavior.

They have their own chip designs. And they still buy a quarter of their capacity from Nvidia. They'd love to make their own, they just can't.

The share is dropping, but it's still a lot, considering their own chips are better!

All chips are assembled at TSMC factories to someone else's designs. And Google and Amazon (who also have their own designs) slept through the moment when Jensen Huang locked in contracts for 70% of 3-nm capacity. That's great for TSMC—they're at the end of the production chain and need stability.

Nvidia is also living the dream, selling cards at 6x production cost.

And Google even sold its own capacity to Anthropic through GCP. What a company.

So What?

So, the industry isn't ready for the agent boom.

Because it came on too suddenly. To a market where what ultimately matters is long-term contracts on complex chip-making infrastructure.

Anthropic right now has 2.5 gigawatts of compute, and by the end of the year they need 5-6. The only way to get that much is the "Other" category. CoreWeave, Bedrock, Vertex, Foundry. Scraps from anyone whose capacity is still available, at premium prices.

And they want to become a profitable company, so they can't afford to burn cash.

Hence the bad news.

The ones who'll probably suffer are us.

The most obvious move is for them to just cut limits and raise prices.

The other week they moved OpenClaw onto the API. And they said so in a nice and honest way. Sorry guys. We're tightening belts, here's $20 as an apology for the inconvenience.

They also rolled out different tiers depending on time of day. I've already run into it a couple of times, when Claude just ran out of capacity. During "off-peak" hours, under pressure from people optimizing for discounted tokens.

Denied.

I pulled two takeaways from this for myself.

1 - Don't put all your eggs in one basket.

For example, when building a skill, make it work on any model. I'm obsessed with Claude, but OpenAI and Google are in way better shape on compute access.

So I've learned to swap models depending on the task. I pay the minimum subscription to every lab. And when the limit runs out, I just switch models.

I'm not using Chinese open-source. Don't use Deepseek, for the love of god.

2 - Get anxious about not making money off AI.

Neural nets aren't a way for me to make more money. They're on my expense sheet, and they pay for themselves by giving me more options and more time.

But if they roll out some $1000 tier, I won't be able to pull that off. Right now that sounds absurd. But remember the example with a real person's salary. As long as $1000 of spend brings in $5000 of profit, you're winning.

And whoever can't pull that off will be stuck on the free tier watching ads.

Image Generators are Generalist Vision Learners

Recent works show that image and video generators exhibit zero-shot visual understanding behaviors, in a way reminiscent of how LLMs develop emergent capabilities of language understanding and reasoning from generative pretraining. While it has long been conjectured that the ability to create visual content implies an ability to understand it, there has been limited evidence that generative vision models have developed strong understanding capabilities. In this work, we demonstrate that image generation training serves a role similar to LLM pretraining, and lets models learn powerful and general visual representations that enable SOTA performance on various vision tasks. We introduce Vision Banana, a generalist model built by instruction-tuning Nano Banana Pro (NBP) on a mixture of its original training data alongside a small amount of vision task data. By parameterizing the output space of vision tasks as RGB images, we seamlessly reframe perception as image generation. Our generalist model, Vision Banana, achieves SOTA results on a variety of vision tasks involving both 2D and 3D understanding, beating or rivaling zero-shot domain-specialists, including Segment Anything Model 3 on segmentation tasks, and the Depth Anything series on metric depth estimation. We show that these results can be achieved with lightweight instruction-tuning without sacrificing the base model's image generation capabilities. The superior results suggest that image generation pretraining is a generalist vision learner. It also shows that image generation serves as a unified and universal interface for vision tasks, similar to text generation's role in language understanding and reasoning. We could be witnessing a major paradigm shift for computer vision, where generative vision pretraining takes a central role in building Foundational Vision Models for both generation and understanding.

Stash

Your AI has amnesia. We fixed it.

Every LLM starts every conversation from zero. Stash gives your agent persistent memory — it remembers, recalls, consolidates, and learns across sessions. No more explaining yourself from scratch.

Open source. Self-hosted. Works with any MCP-compatible agent.

Quick Start

git clone https://github.com/alash3al/stash.git
cd stash
cp .env.example .env   # edit with your API key + model
docker compose up

That's it. Postgres + pgvector, migrations, MCP server with background consolidation — all in one command.

What It Does

Stash is a cognitive layer between your AI agent and the world. Episodes become facts. Facts become relationships. Relationships become patterns. Patterns become wisdom.

An 8-stage consolidation pipeline turns raw observations into structured knowledge — facts, relationships, causal links, goal tracking, failure patterns, hypothesis verification, and confidence decay. Each stage only processes new data since the last run.

Works with Claude Desktop, Cursor, Windsurf, Cline, Continue, OpenAI Agents, Ollama, OpenRouter — anything MCP.

Learn More

alash3al.github.io/stash →

License

Apache 2.0

Efficient Video Intelligence in 2026

Five years ago, video understanding mostly meant action recognition on Kinetics-400 or short-clip captioning on MSR-VTT. Today, vision-language models reason about hour-long footage, on-device tracking segments any object at 16 FPS on a phone, and a single 100M-parameter encoder can match domain experts across image understanding, dense prediction, and VLM tasks. The shift came from rethinking what a video model needs to do, and from taking deployment constraints seriously.

This post walks through where efficient video intelligence stands in April 2026, following how a video system processes its input from raw frames through spatial perception, long-form temporal understanding, multimodal fusion and reasoning, and the deployment stack that makes any of it shippable.

A note up front: the post leans heavily on research from my own group, including EUPE, the EfficientSAM / Efficient Track Anything / EdgeTAM compression line, LongVU, Tempo, EgoAVU, VideoAuto-R1, DepthLM, and ParetoQ. I have tried to place each piece against the parallel and competing work in its section, but this is a perspective from inside one research program rather than a neutral survey.

Why Video Is Harder Than Text or Images

Token volume. A single minute of 30 FPS video at 224x224 resolution and ViT-B/16 patches produces 1,800 frames times 196 patches per frame, or 352K visual tokens before any text or audio, and an hour is 21M tokens before compression. No frontier LLM context window absorbs this naively, so every video model has to compress somewhere.

Information sparsity. Adjacent frames are usually nearly identical, and the interesting events are rare and unevenly distributed. A surveillance camera at 1 FPS over 24 hours produces 86,400 frames, and the question of interest may depend on three of them. Sampling every frame is wasteful, but uniform sampling drops the frames that matter, so adaptive selection is required.

Multi-modality is intrinsic. Video without audio is half a signal in egocentric, conversational, and many healthcare contexts, even though much surveillance footage is silent and sports broadcast audio is mostly commentary. Video with audio doubles the embedding cost and adds synchronization requirements, and training a native multimodal model is a different problem than bolting an audio adapter onto a vision encoder.

Vision Encoders: From Specialists to Universals

The first thing a video model does is encode each frame. Until recently, that meant picking an encoder family and accepting its weaknesses. Image-text contrastive models (CLIP, SigLIP, SigLIP 2) are the default VLM front-end for semantic retrieval but weak on dense prediction. Self-supervised ViTs (DINOv2, DINOv3) excel on dense prediction (segmentation, depth, correspondence) because their training objective preserves fine-grained spatial structure, but their features are not aligned to language. Segmentation foundation models (SAM, SAM 2 and the compressed variants below) are specialists for object proposals and tracking. Dense-prediction specialists (DepthAnything, MiDaS, DepthPro, DepthLM) handle depth.

A production video system on a wearable, robot, or smart camera cannot ship a separate backbone for each of these capabilities, and neither compromising on capability nor paying the memory-and-latency penalty is acceptable.

Agglomerative encoders and EUPE

The agglomerative-encoder thread addresses this directly. AM-RADIO (Ranzinger et al., Nvidia, CVPR 2024) introduced multi-teacher distillation for compact universal vision encoders, distilling CLIP, DINOv2, and SAM into a unified student. Theia (Shang et al., The AI Institute, CoRL 2024) targeted embodied-agent perception by distilling from CLIP, DINOv2, ViT, SAM, and Depth-Anything for robot learning. DUNE (Sariyildiz et al., Naver Labs Europe, CVPR 2025) extended this further with heterogeneous 2D and 3D teachers (DINOv2, MASt3R, Multi-HMR). The shared insight: vision foundation models trained for different objectives produce complementary feature spaces, and a small student can inherit the union if the distillation is set up well.

Our recent work on the Efficient Universal Perception Encoder (EUPE) advances this thread by adding an intermediate proxy-teacher step. The recipe:

1. Train a large proxy teacher by distilling from a diverse teacher pool: DINOv2 and DINOv3 (self-supervised dense features), the SAM family (SAM, SAM 2, SAM 3) for segmentation, and CLIP / SigLIP / SigLIP-SO400M for vision-language alignment.
2. Distill the proxy teacher down into a compact student under 100M parameters.

The intermediate step matters because direct multi-teacher distillation into a small student loses signal: the teachers disagree at the feature level and the student capacity cannot represent the union. A single proxy resolves the disagreements first, then transfers a coherent feature space.

The released family includes ViT-T/S/B and ConvNeXt T/S/B variants, all under 100M parameters, with weights on Hugging Face. Evaluation spans image classification (ImageNet, ObjectNet, SUN397, iNaturalist), dense prediction (ADE20K and COCO segmentation, NYU and KITTI depth, SPair matching), and vision-language tasks (VQA, image-text retrieval). EUPE matches or exceeds same-size domain experts across these domains. For video systems, which are particularly sensitive to per-frame inference cost, a single backbone covering classification, dense prediction, and VLM front-end means fewer encoders to load and amortize, and the latency win compounds with every frame in the stream.

Efficient Attention for Long Sequences

Once frames are encoded, attention becomes the bottleneck. Standard self-attention is O(n²) in sequence length, which is unaffordable for long video. Three families of remedies have stabilized.

Sliding-window and sparse attention. LongLLaMA, Mistral's sliding-window, and DeepSeek's Native Sparse Attention. Each restricts attention to a local or learned subset of tokens.

Linear attention. Performer, Linformer, and Nyströmformer (Xiong et al., AAAI 2021), which uses Nyström-based low-rank approximation of the softmax kernel to achieve linear complexity. Recent production systems extend this thread: Qwen3-Next pairs Gated DeltaNet (a linear-attention variant) with full attention in a 3:1 ratio. These approaches help when sequence length dominates compute.

Hybrid architectures. Mamba-Transformer hybrids (Jamba, Nvidia Nemotron Nano 2) keep self-attention for short-range relationships and use SSM blocks for long-range dependencies. For video this maps naturally: most spatial reasoning is local, while temporal reasoning extends across many frames.

The structural pattern that holds for video is factorized spatial-temporal attention. Spatial attention within a frame is O(P²) where P is patches per frame and small; temporal attention across frames is O(T²) where T is frame count and can be large. Full attention on the spatial axis combined with linear or sparse attention on the temporal axis works well for most workloads, and recent open-weight video VLMs (Qwen3-VL, LLaVA-Video) converge here.

Segmentation and Tracking on Device

Once you can encode and attend efficiently, the next question is what to extract from each frame, and segmentation and tracking are the workhorse primitives.

SAM (Kirillov et al., Meta, ICCV 2023) defined the prompt-driven segmentation foundation model, and SAM 2 (Ravi et al., Meta, 2024) extended it to video with a memory module that maintains separate FIFO queues for recent and prompted frames, plus object pointers, with temporal positional embeddings on the recent queue only. Several parallel lines take different architectural paths: XMem (Cheng et al., ECCV 2022) introduced the multi-store memory architecture (sensory, working, long-term) that informed many later designs; DEVA (Cheng et al., ICCV 2023) decouples task-specific image-level segmentation from a universal temporal propagation module trained once and reused across tasks; and Cutie (Cheng et al., CVPR 2024 Highlight) reads object-level memory through a query-based object transformer rather than propagating pixel-level features. SAM 2 and its compressed descendants dominate the foundation-model production stack today, while Cutie, DEVA, and XMem hold advantages in long-persistence, decoupled-task, and tight-memory regimes respectively.

Most of our work here has been on compression. EfficientSAM (CVPR 2024 Highlight) introduced SAMI, a masked image pretraining recipe that distills SAM's image encoder into much smaller backbones; the released ViT-T and ViT-S variants reach within a few mIoU points of the full SAM ViT-H at a fraction of the cost, and the open-source release made on-device segmentation practical for the first time. Efficient Track Anything (ICCV 2025) extended this to video with two changes: a plain non-hierarchical ViT replaces SAM 2's hierarchical encoder, and an efficient memory module reduces the cost of frame feature extraction and memory computation within SAM 2's bounded memory bank, yielding roughly 2x speedup on A100 with 2.4x parameter reduction at performance comparable to SAM 2, and ~10 FPS on iPhone 15 Pro Max. EdgeTAM (CVPR 2025) pushed further onto consumer silicon with a 2D Spatial Perceiver that compresses per-frame memory aggressively while preserving the spatial structure needed for accurate tracking, hitting J&F scores of 87.7 / 70.0 / 72.3 / 71.7 on DAVIS 2017, MOSE, SA-V validation, and SA-V test while running at 16 FPS on iPhone 15 Pro Max. That is the first time foundation-model-grade video tracking has been deployable on a consumer mobile device.

Most per-frame computation is redundant across adjacent frames, so memory-efficient propagation drives the production gains, not raw model size.

3D and Depth from Video

Segmentation and tracking handle 2D structure, but video also carries strong cues for 3D through parallax, motion, and temporal consistency. The methods that have stabilized are still predominantly image-based, applied per-frame or fed into multi-view reconstructors that treat sampled frames as views; truly temporal-video-native depth is an active but immature area. Extracting metric depth used to require specialized architectures.

DepthLM (ICLR 2026 Oral) shows that a vision-language model with a 3B-parameter backbone, trained with standard text-based supervised fine-tuning and no architecture change, can match or beat dedicated specialists like DepthPro and Metric3Dv2 on metric depth benchmarks. The recipe has three pieces: visual prompting that renders markers on images rather than using text coordinate prompts; intrinsic-conditioned augmentation that unifies focal length to resolve camera ambiguity during training; and supervised fine-tuning on sparsely labeled images, with just one labeled pixel per training image.

DepthLM is the VLM-based entry in a four-way race for metric depth. The dedicated specialists, DepthAnything (Yang et al., CVPR 2024) trained on 1.5M labeled and 62M+ unlabeled images and DepthAnything V2 (NeurIPS 2024) trained on ~595K synthetic-labeled and ~62M pseudo-labeled real images, plus DepthPro (Bochkovskii et al., Apple) and Metric3D v2, still set per-task SOTA on most depth benchmarks. The diffusion-prior approach is best represented by Marigold (Ke et al., CVPR 2024 Oral), which fine-tunes a pretrained image diffusion model and gets strong zero-shot generalization at the cost of latency. The reconstruction family, including DUSt3R and MASt3R (Naver Labs Europe) and the more recent VGGT (Visual Geometry Grounded Transformer, Wang et al., Oxford VGG and Meta AI, CVPR 2025 Best Paper), predicts 3D scene structure, camera parameters, and depth jointly from sparse views, which is useful when geometry matters more than per-pixel depth. Specialists win on raw accuracy, reconstruction wins when camera pose is needed, diffusion priors win on out-of-distribution generalization, and VLM-based approaches like DepthLM win when the same model handles depth and higher-level reasoning.

The implication is structural: if 3D understanding rides on the same VLM that handles reasoning, the stack collapses two perception models into one, and for an AR headset or a robot that simplifies deployment substantially.

Long-Form Video Understanding

Spatial primitives describe what is in a single frame. The harder problem is understanding what an entire video means as length grows from seconds to hours.

LongVU (ICML 2025) addresses this with spatiotemporal adaptive compression. The four-stage pipeline:

1. Temporal redundancy removal via DINOv2. Sample at 1 FPS, compute DINOv2 features within non-overlapping 8-frame windows, drop frames whose features are highly similar to neighbors. Roughly 45.9% of frames are retained after this stage. DINOv2 is used here because its vision-centric self-supervised features are well-suited to inter-frame similarity pruning, while SigLIP is retained downstream for language-aligned semantics.
2. Feature fusion. Extract SigLIP features from the surviving frames and combine them with DINOv2 features through a Spatial Vision Aggregator.
3. Cross-modal query selection. Compute attention between frame features and the LLM's text-query embeddings; retain the top-Nh frames at full 144 tokens and reduce the rest to 64 tokens, balancing detail against budget.
4. Spatial Token Compression. In sliding windows of 8 frames, the first frame keeps full token resolution while tokens in subsequent frames whose cosine similarity to the corresponding anchor token exceeds 0.8 are pruned, yielding about 40.4% additional token reduction.

LongVU is built on Qwen2-7B (with a Llama 3.2-3B lightweight variant) and reaches 60.6% on VideoMME and 65.4% on MLVU with 1 FPS adaptive sampling, outperforming uniform-frame baselines like LLaVA-OneVision while using a fraction of the tokens.

Our follow-up Tempo pushes adaptive token allocation further. A small VLM up front acts as a query-aware compressor: it reads the question first, then routes token budget per-segment, swinging from 0.5 to 16 tokens per frame depending on relevance. The compressed representation is handed to a larger LLM for downstream reasoning. At an 8K visual token budget, the 6B Tempo model reaches 52.3 on LVBench (where videos average over an hour), beating both GPT-4o and Gemini 1.5 Pro at that budget.

LongVU and Tempo sit in a broader thread of compression approaches. LLaMA-VID (Li et al., ECCV 2024) takes aggressive context-token compression to an extreme: each frame is reduced to two learned tokens, a context token encoding instruction-guided information and a content token capturing visual cues, which enables very long videos at the cost of some spatial detail. VideoChat-Flash (ICLR 2026) introduces hierarchical clip-to-video token compression (clip-level during encoding, then video-level in the LLM context) inside a multi-stage short-to-long training scheme, achieving roughly 50x compression with minimal performance loss and 99.1% needle-in-a-haystack accuracy on 10K-frame inputs. PLLaVA and successors apply parameter-free pooling at the projection layer. Frontier multimodal models with very long native context windows (Gemini 2/3 with 1M+ tokens, recent Qwen3-VL variants) go the other way: rather than compress aggressively, they push the budget upward and let attention sort out relevance. The tradeoff is concrete: aggressive compression preserves on-device feasibility but can drop information, while large native contexts preserve information but require frontier-tier compute. LongVU sits at the on-device end of the spectrum, Gemini at the frontier end, and different deployment targets pick different points.

Long-form video understanding is dominated by token budget, and the field is converging on some combination of adaptive token allocation, memory mechanisms, and language-guided pruning. The open question is whether these techniques can work in streaming mode, where the model cannot see the whole video upfront, rather than batch; nobody has solved that cleanly.

Audio-Visual Fusion

Beyond length and spatial structure, audio is what disambiguates many videos, especially egocentric and conversational footage, and how a model fuses audio with the visual stream is a separate architectural choice from anything covered above.

Encoder stitching is the historical default: separate audio and visual encoders feed pooled embeddings into a language model. Cheap and modular, but cross-modal alignment is shallow because the encoders never see each other's data during training. Native multimodal training treats text, image, video, and audio tokens uniformly through a shared backbone. Qwen3-Omni is the strongest open-weight example as of April 2026, with state-of-the-art results on 22 of 36 audio and audio-visual benchmarks (32 of 36 among open-source models) while sharing weights with the visual stack, and Gemini's native multimodal architecture follows a similar internal pattern.

EgoAVU (CVPR 2026 Highlight) takes a third path. Rather than propose a new fusion architecture, EgoAVU builds the first large-scale egocentric audio-visual benchmark and dataset and evaluates how existing VLMs (Qwen2-VL, Gemini, LLaMA 3) perform when audio embeddings are stitched alongside the visual tokens. Audio in egocentric video carries distinct information from third-person video: ambient sound, hand-object contact noise, the wearer's own voice, and conversational partners are all anchored on the wearer's body in ways they are not in YouTube-style footage. The evaluation shows that audio adds substantial signal on egocentric understanding tasks and that stitched audio encoders into existing VLMs are already a strong baseline; the headroom is in better data and training, not in radical architectural changes.

Native multimodal wins at scale, but egocentric data is underrepresented in pretraining corpora and wearables are the deployment target where this distribution dominates. Benchmark-driven progress on the egocentric slice matters more for wearable products than for cloud video generally.

Reasoning Over Video

Encoding, compression, and fusion produce a representation; reasoning is what turns that representation into an answer. A VLM that watches a video and answers in one forward pass often fails on temporally-extended questions, because compressing hours of footage into a fixed-length representation and reading the answer back out drops too much nuance.

VideoAuto-R1 (CVPR 2026) starts from a counterintuitive observation: for RL-trained video VLMs, direct answering often matches or beats chain-of-thought reasoning while costing a lot more tokens. The proposed recipe is "reason-when-necessary." During training, the model first generates an initial answer, then performs reasoning, then outputs a reviewed final answer; both the initial and reviewed answers are supervised through verifiable rewards. At inference, the confidence of the initial answer determines whether to spend tokens on reasoning at all. The result: state-of-the-art accuracy on video QA and grounding benchmarks while reducing average response length roughly 3.3x (from ~144 to ~44 tokens). Thinking-mode activates rarely on perception-oriented questions and often on reasoning-intensive ones, which suggests that explicit reasoning helps but is not always necessary, and gating it on confidence is a meaningful efficiency win.

Several lines have converged on related patterns. Video-of-Thought (Fei et al., ICML 2024) introduced step-by-step video reasoning that decomposes a complex question from low-level pixel perception to high-level cognitive interpretation, paired with the MotionEpic VLM that grounds reasoning in spatial-temporal scene graphs. VideoTree (Wang et al., CVPR 2025) builds a query-adaptive hierarchical tree by iteratively selecting the keyframes most relevant to the question, achieving strong long-form QA without any training. Plan-and-execute approaches in the broader VLM-agent literature share the same structural pattern with different implementations. Single-pass video VLMs fail predictably on long-horizon questions, and the field has settled on two-stage inference. The remaining question is whether the reasoning step should be explicit (interpretable, easier to debug, slower) or implicit through learned routing (faster, harder to introspect).

Deployment: Where Video Intelligence Actually Runs

Video deployment splits into three tiers, and the choice between them is driven as much by economics, latency, and data residency as by raw model capability.

Cloud. Frontier APIs like Gemini's video understanding endpoints and the multimodal flagships from OpenAI and Anthropic that accept image and audio (with video typically handled via frame sampling); specialized providers like Twelve Labs (Marengo embeddings and Pegasus video LLM with hour-scale temporal segmentation); hyperscaler services like AWS Rekognition Video, Azure Video Indexer, and Google Video Intelligence. The cloud tier gets you the largest models and the longest context with no client-side complexity, but it pays in round-trip latency (hundreds of milliseconds minimum), cost (10-100x edge inference per task), and bandwidth that breaks for continuous video at scale.

Edge servers. On-prem GPU appliances or smart camera bridges, like Verkada's bridges, Hayden AI's on-device units, or industrial-inspection servers running Cosmos NIM. This tier trades the cloud's latency and data-residency problems for a hardware investment and a fragmented stack across customers, and supports mid-size models in the 3-30B range.

On-device. Mobile SoCs, AR glasses silicon, embedded NPUs. Apple Intelligence on iPhone, Qualcomm Robotics RB5/RB6 in robotics, Qualcomm Snapdragon AR1 in Ray-Ban Meta and Snapdragon XR2 Gen 2 in Quest 3. Zero-latency, fully private, no bandwidth, and it scales with device shipments. The cost is a tight power budget (1-30W), limited memory bandwidth, and a fragmented runtime landscape.

For continuous video the math forces the choice. A body-cam recording 12 hours per shift cannot ship 100GB per day to the cloud per officer, so the fast-thinking layer has to live on the device, with cloud or edge servers used for the deeper queries. Hybrid architectures, not pure cloud or pure on-device, are the production default.

Quantization Recipes for Video Models

Video models inherit the quantization recipes that have stabilized for LLMs and VLMs.

• W4A16 (4-bit weights, 16-bit activations) is the default for VLMs and VLAs at the edge. Recent open releases including the Embedl-quantized Cosmos-Reason2 (2B) variants show the recipe holds across multimodal architectures with minimal accuracy loss.
• NVFP4 (4-bit weights and 4-bit activations in NVIDIA's FP4 format with per-block-of-16 FP scales) unlocks Blackwell-tier hardware (Jetson AGX Thor) and is the production-grade upgrade where supported.
• W8A8 remains the safer fallback for mature vision and segmentation models.
• Sub-4-bit quantization (W2A16, ternary, mixed precision) continues to improve. Our ParetoQ (NeurIPS 2025) work mapped the full quantization Pareto frontier and showed that at 2 bits and below, models learn fundamentally different representations than at 3-4 bits; for a fixed memory budget, a larger 2-bit model can beat a smaller 4-bit model. That shifts the design space for very-low-power video deployment, though it still requires QAT and is not yet standard for production VLMs.
• KV cache quantization matters more for video than for text. The KV cache for a long video can dominate memory, and rotation-based methods like SpinQuant (which jointly quantize weights, activations, and KV cache) have been particularly effective at compressing it to 3-4 bits per element.

Runtime Stack

For PyTorch-based deployment, ExecuTorch (Meta) is the natural path. ExecuTorch reached 1.0 GA in October 2025 and now powers Meta's on-device AI across Instagram, WhatsApp, Messenger, Facebook, Quest 3, and Ray-Ban Meta, with backends spanning Apple Core ML, Qualcomm QNN, Arm, MediaTek NeuroPilot, and Vulkan. For video pipelines, ExecuTorch's support for streaming inference and selective recomputation matters because re-encoding every frame from scratch is wasteful. Other paths cover other ecosystems: Apple Core ML for Apple platforms, LiteRT-LM plus Qualcomm QNN for Android, Nvidia Isaac plus NIM on Jetson, Intel OpenVINO for x86 industrial. No single runtime wins, and production video systems usually ship the same model compiled for several backends.

What's Still Hard

Several problems remain open across the stack.

Continuous-stream understanding at hour-plus durations. LongVU and similar techniques assume batch mode where the whole video is available. Streaming mode, where the model has to maintain understanding while video keeps arriving, is much harder. Memory mechanisms, retrieval-augmented architectures, and incremental token compression are all in progress; none are solved cleanly.

Sparse-event detection. Most production video is uninteresting. Finding the three frames out of 86,400 that matter, without paying for full inference on all 86,400, requires hierarchical attention or learned selection. Schema-driven extraction over known classes now ships commercially (Twelve Labs' Pegasus pulls structured metadata against a customer-defined schema); open-set "show me anything anomalous" remains unsolved.

Cross-camera and cross-clip reasoning. A surveillance ops team often wants to ask questions across many cameras and many time windows. Library-scale retrieval over indexed videos ships (Twelve Labs' Marengo ranks moments across a video library), but that is ANN retrieval over independent embeddings, not joint reasoning. Multi-stream attention, cross-camera identity persistence, and global temporal reasoning are all open.

Real-time sub-watt inference for AR glasses. Today's mobile NPUs do tens of TOPS in tens of milliwatts, but an AR glass AI assistant needs to do continuous video understanding inside a 1-3W envelope that includes everything else the system runs. EUPE-style universal compact encoders, EdgeTAM-style efficient tracking, and aggressive quantization all help, but the gap to always-on Gemini-grade understanding on glasses is still 5-10x in compute efficiency.

Closed-loop evaluation. Public benchmarks measure accuracy on curated multiple-choice question sets. Production systems care about latency under load, drift under deployment shifts, robustness to camera placement and lighting, and intervention rates. Closed-loop methodology lags benchmark accuracy by a wide margin.

Audio-visual generative consistency. When video models generate or edit content rather than understand it (out of scope for most of this post), keeping audio synchronized with visual events is unsolved, which is why most current text-to-video models ship without working audio.

Cross-modal grounding stability. When a VLM is asked "what is the man in the blue shirt doing?", the model often fails not on language understanding but on grounding the referent across frames. Timestamp-level grounding ships commercially (Pegasus localizes answers to start/end times); spatial grounding (bounding boxes, referent IDs across cuts) still requires bolting on SAM 2 or Grounding DINO.

Closing

A handful of patterns recur across encoding, perception, compression, fusion, reasoning, and deployment. Compress where redundancy is highest, which for video is almost always the temporal axis. Distill universal encoders from multiple teachers rather than ship a fleet of specialists. Factorize attention along the physical structure of the data: spatial within frames, temporal across frames, cross-modal across modalities. Treat quantization as the default rather than as a late optimization. Gate reasoning on confidence rather than running it on every input.

The encoder, compression, and fusion patterns are now stable; the streaming, sub-watt deployment, and closed-loop evaluation patterns are not. The open problems left in efficient video intelligence are mostly about scaling the stable recipes to streaming inputs, sub-watt power envelopes, and production deployments where evaluation has to track a system rather than a benchmark. The work ahead lives in the deployment stack at least as much as in the model layer.

Scaling Test-Time Compute for Agentic Coding

Test-time scaling has become a powerful way to improve large language models. However, existing methods are best suited to short, bounded outputs that can be directly compared, ranked or refined. Long-horizon coding agents violate this premise: each attempt produces an extended trajectory of actions, observations, errors, and partial progress taken by the agent. In this setting, the main challenge is no longer generating more attempts, but representing prior experience in a form that can be effectively selected from and reused. We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories. Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes while discarding low-signal trace details. This representation enables two complementary forms of inference-time scaling. For parallel scaling, we introduce Recursive Tournament Voting (RTV), which recursively narrows a population of rollout summaries through small-group comparisons. For sequential scaling, we adapt Parallel-Distill-Refine (PDR) to the agentic setting by conditioning new rollouts on summaries distilled from prior attempts. Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0. For example, by using our method Claude-4.5-Opus improves from 70.9% to 77.6% on SWE-Bench Verified (mini-SWE-agent) and 46.9% to 59.1% on Terminal-Bench v2.0 (Terminus 1). Our results suggest that test-time scaling for long-horizon agents is fundamentally a problem of representation, selection, and reuse.

Weiyao Wang spent eight years at Meta — his first job out of college — helping build multimodal perception systems and contributing to open-world segmentation projects, including SAM3D. His final day at Meta was last week, and he has since joined Thinking Machines Lab (TML).

His move to TML comes as the AI startup expands on multiple fronts. It just signed a multibillion-dollar cloud deal with Google, giving it access to Nvidia's latest GB300 chips and making it one of the first startups to run on the hardware.

The agreement, announced this past Tuesday at Google Cloud Next, follows an earlier partnership with Nvidia, and puts TML in the same infrastructure tier as Anthropic and Meta. (Meta reportedly held talks to acquire Thinking Machines around this time last year and has more recently been picking off TML's founders one by one.)

The talent picture remains fluid. Wang and Kenneth Li — a Harvard PhD who spent 10 months at Meta before joining TML this month — are the latest examples of a talent grab that runs in both directions. Business Insider reported last week that Meta has now poached seven of TML's founding members. A review of recent hires shows Thinking Machines is raiding Meta right back. At least, it appears based on a review of LinkedIn profiles, that TML has been hiring more researchers from Meta than from any other single employer.

The most prominent is Soumith Chintala, TML's CTO, who spent 11 years at Meta and co-founded PyTorch, the open source deep learning framework that now underpins most of the world's AI research. He left Meta in late 2025 and was appointed CTO earlier this year. Piotr Dollár, another 11-year Meta veteran who served as research director and co-authored the influential Segment Anything model, is now on TML's technical staff. Andrea Madotto, a research scientist in Meta's FAIR division focused on multimodal language models, joined TML in December. James Sun, a software engineer with nearly nine years at Meta working on LLM pre- and post-training, also made the jump.

TML has drawn talent from beyond Meta, too. Neal Wu — a three-time gold medalist at the International Olympiad in Informatics and a founding member of the buzzy coding startup Cognition — joined early this year. Jeffrey Tao came via Waymo, Windsurf, and OpenAI. Muhammad Maaz previously held a research fellowship at Anthropic. Erik Wijmans arrived from Apple. Liliang Ren spent two and a half years on Microsoft's AI Superintelligence team pre-training OpenAI models for code before joining in March.

The startup's headcount now stands at around 140.

Meta's pay packages — seven figures, no strings attached — are well known by now. For researchers weighing their other options, the calculus may be as simple as this: Thinking Machines Lab is right now valued at $12 billion. Though that figure would've been unimaginable for a company at this stage in any previous tech cycle (it has released just one product so far), compared with the record-breaking valuations of OpenAI and Anthropic, there's still a lot of financial upside.

Reached Friday morning, a spokesperson for TML declined to comment for this story.

OpenAI has published a five-principle framework for the development of artificial general intelligence. It is the company's most prominent statement of intent since its 2018 Charter. The lab claims it will resist letting the technology consolidate power in the hands of the few. The framework arrives at a time when US and European regulators are tightening oversight of frontier AI labs.

Cursor's $60 Billion Escape Hatch

SpaceX Secures Option to Acquire Cursor

SpaceX announced this week that it has secured an option to acquire AI coding startup Cursor for $60 billion, or pay $10 billion for the work they're doing together if it doesn't end up acquiring the company. The agreement gives Cursor access to SpaceX's Colossus supercomputer and a path to reducing its dependence on Anthropic and OpenAI, whose models currently power much of Cursor's product and whose fees have weighed heavily on its margins.

The timing of the SpaceX deal is interesting. Just a few weeks earlier, Cursor had been quietly attempting to raise billions in private markets, but had encountered a lack of interest from late-stage investors like Iconiq, many of whom had just deployed capital into OpenAI and Anthropic and weren't ready to back a competitor at a $50 billion valuation. Cursor's gross margins were negative 23% as of January, an unusual position for a company generating $2.7 billion in annualized revenue. The company ultimately lined up $2 billion from Nvidia, a16z, and Thrive before canceling the round following after the deal with SpaceX was announced.

The Cursor deal complicates an already complicated financial picture for SpaceX ahead of its June IPO. Reuters reported this week that SpaceX took out a $20 billion bridge loan last month to refinance debt tied to X and xAI, reducing total debt from $22 billion to $20 billion, with repayment potentially contingent on IPO proceeds. xAI spent $12.7 billion in capital expenditures last year while generating only $3.2 billion in revenue. Cursor lost nearly $900 million in its last fiscal year. This may indicate that SpaceX is using its IPO momentum to paper over a collection of cash-hungry businesses before public market investors get a full look at the S-1.

Anthropic's Holdback Test Draws Criticism

A tweet claiming that Anthropic was no longer offering Claude Code access to Pro subscribers paying $20 per month made the rounds on social media this week. This was seen as an indication that the company would need to take drastic measures to maintain service for customers amidst an ongoing compute shortage. Anthropic has not said how many Claude Code users it has currently, or how many of those users are Pro or Max subscribers.

As it turns out, the situation was overstated. Anthropic's head of growth, Amol Avasare, responded directly to the post, saying that "we're running a small test on ~2% of new prosumer signups. Existing Pro and Max subscribers aren't affected." In AB testing, this is called a "holdback test," in which the value of a feature is measured by excluding a small subset of users from accessing it. Nevertheless, many questioned the wisdom of running a test like this on a tool as widely used as Claude, and competitors were quick to pile on. OpenAI directly implied it was a violation of customer trust, and Sam Altman mockingly replied "ok boomer" to Amol's post.

OpenAI's Changing Tune

On April 14, just one week after Anthropic said Mythos was too powerful to release publicly because of cybersecurity concerns, OpenAI published a blog post announcing that its newest model would not be released broadly to the public and would instead be accessible only to verified partners via a tiered cybersecurity access program introduced in February called Trusted Access for Cyber (TAC). This specifically pertained to a variant of GPT-5.4, with OpenAI saying "we are fine-tuning our models specifically to enable defensive cybersecurity use cases, starting today with a variant of GPT‑5.4 trained to be cyber-permissive: GPT‑5.4‑Cyber."

Then, on April 23rd, OpenAI announced that its newest model, GPT-5.5, would in fact be available to all users, though only via chatbot, not API. The press release said this was because "API deployments require different safeguards," but also that "We'll bring GPT‑5.5 and GPT‑5.5 Pro to the API very soon". On April 24th, the company seemingly changed their mind or implemented the safeguards very fast, and GPT-5.5 became available via API that afternoon. The company's evolving stance has fueled skepticism about the actual risks of the models and supported the view that current limits are motivated by the business model, not safety concerns.

Latest Research

• Earlier this year, Chai Discovery, which builds generative models for antibody design, announced a partnership with Eli Lilly. See our new report here.
• Sierra is an AI customer service startup that surpassed $100 million in ARR last year. See our new report here.
• Thatch is an ICHRA administrator that grew revenue 8x in 2024 and 4x in 2025. See our new report here.

What We're Reading

• Anthropic is investigating how unauthorized users gained access to Anthropic's restricted Mythos cybersecurity model by combining a third-party contractor's credentials with data from a breach at AI startup Mercor Inc. to locate the model's endpoint.
• xAI held talks in recent weeks with French lab Mistral and AI coding firm Cursor about a three-way partnership to close the gap on Anthropic and OpenAI, on top of SpaceX's newly disclosed $60 billion call option to acquire Cursor, with ex-Mistral cofounder Devendra Chaplot already running pretraining at xAI.
• Apple announced that Tim Cook will step up to executive chairman and SVP of Hardware Engineering John Ternus will take over as CEO effective September 1, 2026, concluding Cook's 15-year tenure during which Apple's market cap grew from ~$350 billion to $4 trillion.
• Blue Origin successfully recovered the New Glenn first stage rocket in its second flight last week, but the rocket's payload, an AST SpaceMobile communications satellite, ended up in an "off-nominal orbit," temporarily grounding New Glenn.
• Tesla expanded its Robotaxi service to Dallas and Houston, but Robotaxi Tracker data suggests "there may be just one vehicle operating in each city so far," with tiny geofences relative to each city's footprint.
• GitHub paused new sign-ups for paid Copilot plans and tightened usage limits after agentic workflows consumed more compute than monthly fees cover, with VP of product Joe Binder writing, "It's now common for a handful of requests to incur costs that exceed the plan price."
• Amazon committed up to $25 billion in additional funding to Anthropic ($5B immediately, $20B tied to "certain commercial milestones") at Anthropic's $380B valuation, with Anthropic pledging $100B+ on AWS over 10 years, CEO Andy Jassy said the commitment "reflects the progress we've made together on custom silicon."
• Meta Superintelligence Labs rolled out its "Model Capability Initiative" on Meta US employees' work laptops to capture mouse movements, keystrokes, and periodic screenshots as training data for computer-use agents. Despite internal backlash, the company has clarified that there is no option to opt out of the initiative.
• Commerce Secretary Howard Lutnick told a Senate hearing that despite Trump lifting the H200 export ban four months earlier, no H200 chips have been sold to Chinese firms, saying "The Chinese central government has not let them, as of yet, buy the chips."
• Trump issued an executive order directing the FDA to prioritize clinical trials and "Right to Try" access for psilocybin, MDMA, and ibogaine, and allocated $50 million in HHS matching funds for state research programs, triggering rallies in psychedelic drug-developer stocks.
• The Trump administration is nearing a deal to lend Spirit Airlines up to $500 million in exchange for warrants giving the US government a potential 90% equity stake post-bankruptcy, sparking pushback from Transportation Secretary Sean Duffy, who asked, "If you do Spirit, who comes next?"
• About 30K Samsung workers rallied at the Pyeongtaek chip complex in South Korea, demanding 15% of operating profit be paid out to chip-division employees. This would amount to more than 40 trillion won (~$27 billion), averaging $400K+ per worker, with the union threatening an 18-day strike starting May 21, following Samsung's record Q1 operating profit forecast of 57.2 trillion won.

We're joining forces with Aleph Alpha to provide the world with an independent, enterprise-grade sovereign alternative in an era of growing AI concentration.

This transatlantic alliance would combine Cohere's global AI scale with Aleph Alpha's strong research excellence and deep institutional relationships, forging a globally competitive AI champion backed by Canadian and German ecosystems.

By pooling top-tier engineering talent and computational resources across two G7 nations, the partnership aims to significantly accelerate the development of next-generation frontier models and systems while providing a secure alternative to dependence on any single vendor or infrastructure stack.

The market for AI services is projected to surpass $1 trillion annually, with sovereign AI needs representing nearly $600B of that total (McKinsey, March 2026). The partnership uniquely bridges the gap between these segments with its sovereign-first approach, capturing the critical intersection where sovereignty requirements meet broad enterprise AI adoption.

Our Co-Founder and CEO, Aidan Gomez comments: "Combining the strengths of Cohere and Aleph Alpha accelerates our global expansion and advances our mission to deliver sovereign AI to nations around the world. Organizations globally are demanding uncompromising control over their AI stack. This transatlantic partnership unlocks the massive scale, robust infrastructure, and world-class R&D talent required to meet that demand.

Built on the bedrock of shared Canadian and German values—where privacy, security, and responsible innovation are paramount—we are uniquely positioned to be the world's trusted AI partner. Together, we will give enterprises and governments across Canada, Europe, and the world the technology to move from exploration to rapid, secure implementation, with the absolute certainty that their data remains their own."

Through the planned deal, collectively we aim to deliver a secure alternative for customized AI in highly-regulated sectors - including the public sector, finance, defense, energy, manufacturing, telecommunications, and healthcare. Aleph Alpha's experience in deploying AI in long-standing customer relationships provides an important foundation of this sovereign offering. As part of this partnership, the combined entity will partner with the companies of Schwarz Group, an international leader in the retail industry, to deploy a sovereign offering on its cloud service STACKIT.

Ilhan Scheer, Co-CEO of Aleph Alpha adds: "Aleph Alpha is in a unique position in Europe. We develop specialized large language models for Europe without compromising on Sovereignty, Transparency and Regulatory Compliance. By living this responsibility, we serve as a trusted and strategic partner to public sector and enterprise customers in Europe. Together with Cohere, we are building a real counterweight for organizations that refuse to outsource control over their AI to a single provider or jurisdiction, giving European institutions and enterprises access to powerful, yet controllable AI they can truly own."

In addition, the companies of Schwarz Group intend to back our upcoming Series E funding as lead investor with a $600M (€500M) structured financing commitment. The round is already attracting strong interest from the world's leading investors who recognize the necessity of an independent global AI powerhouse.

In a joint statement, Rolf Schumann and Christian Müller, Co-CEOs of Schwarz Digits, said: "With this investment, the companies of Schwarz Group position themselves as lead investors for digital sovereignty and infrastructure. Building this infrastructure is a strategic necessity to help shape the AI revolution based on values such as trust, fairness, and responsibility. The establishment of STACKIT, Schwarz Digits' sovereign cloud infrastructure, as the technical backbone of this transatlantic AI initiative empowers organizations to strengthen their digital independence and maintain control over their data. This is true leadership in digital sovereignty."

An amateur just solved a 60-year-old math problem—by asking AI

A ChatGPT AI has proved a conjecture with a method no human had thought of. Experts believe it may have further uses

By Joseph Howlett edited by Lee Billings

Liam Price just cracked a 60-year-old problem that world-class mathematicians have tried and failed to solve. He's 23 years old and has no advanced mathematics training. What he does have is a ChatGPT Pro subscription, which gives him access to the latest large language models from OpenAI.

Artificial intelligence has recently made headlines for solving a number of "Erdős problems," conjectures left behind by the prolific mathematician Paul Erdős. But experts have warned that these problems are an imperfect benchmark of artificial intelligence's mathematical prowess. They range dramatically in both significance and difficulty, and many AI solutions have turned out to be less original than they appeared.

The new solution—which Price got in response to a single prompt to GPT-5.4 Pro and posted on www.erdosproblems.com, a website devoted to the Erdős problems, just over a week ago—is different. The problem it solves has eluded some prominent minds, bestowing it some esteem. And more importantly, the AI seems to have used a totally new method for problems of this kind. It's too soon to say with certainty, but this LLM-conceived connection may be useful for broader applications—something hard to find among recently touted AI triumphs in math.

"This one is a bit different because people did look at it, and the humans that looked at it just collectively made a slight wrong turn at move one," says Terence Tao, a mathematician at the University of California, Los Angeles, who has become a prominent scorekeeper for AI's push into his field. "What's beginning to emerge is that the problem was maybe easier than expected, and it was like there was some kind of mental block."

The question Price solved—or prompted ChatGPT to solve—concerns special sets of whole numbers, where no number in the set can be evenly divided by any other. Erdős called these "primitive sets" because of their connection to similarly indivisible prime numbers.

"A number is prime if it has no other divisors, and this is kind of generalizing that definition from an individual number to a collection of numbers," says Jared Duker Lichtman, a mathematician at Stanford University. Any set of prime numbers is automatically primitive, because primes have no factors (except themselves and the number one).

Erdős also came up with the Erdős sum, a "score" you can calculate for any primitive set. He showed that the sum had a maximum possible value—and conjectured that this value must hold only for the set of all prime numbers. Lichtman proved Erdős right as part of his doctoral thesis in 2022.

Erdős also noticed that the score drops if all of a set's numbers are large—the larger the numbers, the less large the score could become. He guessed that as the set's numbers approached infinity, the maximum score would drop to exactly one. Lichtman tried to prove this, too, but got stuck like everyone else before him.

Price wasn't aware of this history when he entered the problem into ChatGPT on an idle Monday afternoon. "I didn't know what the problem was—I was just doing Erdős problems as I do sometimes, giving them to the AI and seeing what it can come up with," he says. "And it came up with what looked like a right solution."

He sent it to his occasional collaborator Kevin Barreto, a second-year undergraduate in mathematics at the University of Cambridge. The duo had jump-started the AI-for-Erdős craze late last year by prompting a free version of ChatGPT with open problems chosen at random from the Erdős problems website. (An AI researcher subsequently gifted them each a ChatGPT Pro subscription to encourage their "vibe mathing.")

Reviewing Price's message, Barreto realized what they had was special, and experts whom he notified quickly took notice.

"There was kind of a standard sequence of moves that everyone who worked on the problem previously started by doing," Tao says. The LLM took an entirely different route, using a formula that was well known in related parts of math, but which no one had thought to apply to this type of question.

"The raw output of ChatGPT's proof was actually quite poor. So it required an expert to kind of sift through and actually understand what it was trying to say," Lichtman says. But now he and Tao have shortened the proof so that it better distills the LLM's key insight.

More importantly, they already see other potential applications of the AI's cognitive leap. "We have discovered a new way to think about large numbers and their anatomy," Tao says. "It's a nice achievement. I think the jury is still out on the long-term significance."

Lichtman is hopeful because ChatGPT's discovery validates a sense he's had since graduate school. "I had the intuition that these problems were kind of clustered together and they had some kind of unifying feel to them," he says. "And this new method is really confirming that intuition."

Editor's Note (4/28/26): This article was edited after posting to correct the description of the Erdős sum and to clarify Jared Duker Lichtman's full name.

Key takeaways

• The deployment starts with tens of millions of Graviton cores, with the potential to expand.
• Meta is now one of the largest Graviton customers in the world.
• The deal builds on Meta's long-standing AWS relationship and use of Amazon Bedrock at scale to support its next generation of AI.

Meta has signed an agreement to deploy AWS Graviton processors at scale. The deal marks a significant expansion of a long-standing partnership between the two companies as Meta builds its next generation of AI. The deployment starts with tens of millions of Graviton cores, with the flexibility to expand as Meta's AI capabilities grow. The deal reflects a shift in how AI infrastructure gets built: while GPUs remain essential for training large models, the rise of agentic AI is creating massive demand for CPU-intensive workloads—real-time reasoning, code generation, search, and orchestrating multi-step tasks. Graviton5 is purpose-built for these workloads, giving Meta the processing power to run them efficiently at scale. The chips will power various workloads at Meta, including supporting the company's AI efforts. That work requires infrastructure that can handle billions of interactions while coordinating complex, multi-step agent workflows—exactly the kind of CPU-intensive work Graviton is designed for.

AWS Graviton chips powering AI workloads

As organizations increasingly adopt agentic AI—autonomous systems that can reason, plan, and complete complex tasks—the demand for high-performance, energy-efficient compute infrastructure has never been greater. Meta is building at the forefront of agentic AI, and its broad Graviton deployment reflects a simple reality: agentic workloads like code generation, real-time reasoning, and frontier model training are CPU-intensive, and purpose-built chips are the most efficient way to power them. The Graviton5 chip features 192 cores and a cache that is five times larger than the previous generation, which reduces delays in how quickly those cores communicate with each other by up to 33%. That means faster data processing with greater bandwidth—key requirements for agentic AI systems that need to continuously reason through and execute multi-step tasks. Graviton is built on the AWS Nitro System, which uses dedicated hardware and software to deliver high performance, high availability, and high security. The Nitro System enables bare-metal instances for direct access to the hardware while providing the same familiar Elastic Network Adapter (ENA) and Amazon Elastic Block Store (Amazon EBS) devices that allow Meta to run its own virtual machines without performance compromises. The range of Graviton5 instances also supports the Elastic Fabric Adapter (EFA), enabling low-latency, high-bandwidth communication between instances. This is essential for Meta's agentic AI workloads, where large-scale tasks need to be distributed across many processors working in coordination. As a longtime AWS customer, Meta has relied on AWS's highly scalable and secure cloud infrastructure to power its global businesses. "This isn't just about chips; it's about giving customers the infrastructure foundation, as well as data and inference services, to build AI that understands, anticipates, and scales efficiently to billions of people worldwide," said Nafea Bshara, vice president and distinguished engineer, Amazon. "Meta's expanded partnership, deploying tens of millions of Graviton cores, shows what happens when you combine purpose-built silicon with the full AWS AI stack to power the next generation of agentic AI." "As we scale the infrastructure behind Meta's AI ambitions, diversifying our compute sources is a strategic imperative. AWS has been a trusted cloud partner for years, and expanding to Graviton allows us to run the CPU-intensive workloads behind agentic AI with the performance and efficiency we need at our scale," said Santosh Janardhan, head of infrastructure, Meta.

Energy efficiency benefits of Graviton

AWS Graviton5 is built on 3-nanometer chip technology—a manufacturing process that produces smaller, more efficient processors. Because AWS designs its chips from the ground up and controls the full process from chip design through server architecture, it can optimize performance and efficiency in ways that off-the-shelf processors can't match. The result is infrastructure that delivers stronger performance while maintaining leading energy efficiency, helping Meta pursue ambitious AI goals while staying on track with sustainability targets. Graviton5 delivers up to 25% better performance than the previous generation. As the demand for AI compute grows across the industry, the efficiency of the underlying infrastructure becomes increasingly important—both for managing costs and reducing environmental impact. The deal signals a new chapter in how large-scale AI infrastructure gets built—and how purpose-built chips like Graviton can help companies like Meta deliver smarter, more personalized experiences to billions of people worldwide. Get more details on AWS Graviton5 and how to get started with Graviton.

The national lab thesis is legitimate for nations, but for everyone else, it's a solution to a problem they don't have.

Anthropic appears to be building a tool within Claude Code called Bugcrawl, which surfaces as a dedicated entry in the side navigation. Once opened, the screen presents a repository selection UI alongside a warning that the feature consumes tokens at a high rate, so it's suggested to start with a small repository before pointing it at anything substantial. That caveat alone hints at the scale of work the agent would be carrying out in the background.

The most plausible read is that Bugcrawl will set Claude loose across an entire codebase to hunt for general bugs and propose fixes, while the Security tab already shipping in Claude Code for Enterprises targets vulnerabilities specifically. If Anthropic pushes the concept further, the same loop could extend into end-to-end product testing, where Claude spins up a local instance of the app, walks through user flows, and reports regressions. How feature specifications or test criteria would be passed into a run is still an open question, since the only screen visible so far is the repository picker.

For Anthropic, the move slots cleanly into the Claude Code expansion of recent months, which has already produced Claude Code Security in February and Claude Code Review in March, both built around multi-agent investigation of code. Bugcrawl would round out that lineup by tackling general correctness and quality, the broader, fuzzier category that sits between security scanning and PR-level review. It also fits the wider competitive picture, with OpenAI's Codex, xAI's Grok Build, and Google's Jules each pushing toward agents that reason across full repositories rather than single files.

The likely audience is engineering teams on Team and Enterprise tiers, where the token burn warning is easier to absorb. No release window has surfaced, and the feature does not appear in production builds. Given the cadence of Code Security and Code Review landing within weeks of each other, a research preview on the same web surface looks like the most likely path.

HashiCorp released Vault 2.0 under IBM's versioning model with two-year support, introducing identity-based security, workload identity federation without static credentials, performance improvements, and breaking changes while adding SCIM, SPIFFE support, and enhanced PKI automation.

DigitalOcean Dedicated Inference: A Technical Deep Dive

Getting a model to answer 10 inference requests concurrently is tricky but simple enough; getting it to handle 2,000 engineers hitting a coding assistant with long contexts, all day, without runaway costs, is where teams stall. A working endpoint is only the beginning. Teams need to identify the supporting hardware and wire up the right components—serving, scaling, observability, and cost guardrails—so the deployment can support expected SLAs and SLOs under real, sustained load.

DigitalOcean already offers Serverless Inference on the DigitalOcean AI Platform: a fast path to models from OpenAI, Anthropic, Meta, or other providers, with minimal setup and token-based pricing. This offering works well for many use cases. However, when you need your own weights, predictable performance on dedicated GPUs, and economics that favor sustained, high-volume token generation over pay-per-token bursts, a different approach makes sense

Dedicated Inference, our managed LLM hosting service on the DigitalOcean AI Platform, fills that gap.

Dedicated Inference deploys and operates an opinionated inference stack on dedicated GPUs, with Kubernetes-native orchestration under the hood. You interact through the control plane and APIs you already use in the DigitalOcean ecosystem; the data plane exposes public and private endpoints so applications inside, or outside, your VPC can call your models securely.

The service is designed to collapse a vast combinatorial space—GPU SKUs, runtimes, routers, autoscaling policies—into guided defaults so teams hit production milestones faster than DIY stacks, while retaining knobs that matter for model serving: replicas, scaling behavior, and advanced optimizations as you roll out your product roadmap.

What we manage vs. what you control

Every managed product draws a line between operator-owned and customer-owned concerns. Dedicated Inference aims to put day-two operations—cluster lifecycle integration, ingress, core serving and routing components, and the glue between them—on the platform side, while leaving model choice, capacity, and workload-specific tuning with you.

Typically platform-managed:

• Provisioning and lifecycle of the underlying orchestration footprint in line with product design (for example, managed Kubernetes integration and GPU pool coordination).
• Core inference engine and orchestration integration, including patterns that matter at scale: intelligent routing, autoscaling hooks, and production-oriented serving paths.
• Endpoint creation, health and scaling workflows, and the operational automation required to keep endpoints aligned with declared configuration.

In your hands:

• Selecting models (including bring-your-own-model paths where supported), GPU profiles, and replica counts appropriate to your SLOs and budget.
• Configuring scaling behavior and, over time, advanced serving options that map to your latency, throughput, and cost goals.
• Connecting applications via stable HTTP APIs consistent with common LLM client stacks.

Dedicated Inference overview

Dedicated Inference builds on industry-standard building blocks so customers benefit from community momentum and continuous improvement:

• vLLM as a capable, widely adopted inference engine for large language models on modern GPUs.
• LLM-d as a Kubernetes-oriented stack for distributed inference patterns—precise prefix-cache aware routing, scaling concerns that differ from traditional HTTP services, and room to grow into more advanced topologies as workloads demand.

This combination reflects a deliberate choice: meet customers where they are today (OpenAI-compatible APIs, familiar GPU offerings on DigitalOcean) while staying aligned with where the ecosystem is moving on routing, replication, and scale-out inference.

For readers who want more depth on why LLM routing differs from classic load balancing—and how prefix cache awareness changes the game—see our article on Load Balancing and Scaling LLM Serving.

High-level architecture

The system design separates a control plane (how endpoints are created, updated, listed, and deleted) from a data plane (how chat/completions request traffic reaches your models). That mirrors the management requests, which take a path built for regional placement and durable lifecycle work. Inference requests take a direct, low-latency path in front of your GPUs.

Control plane: central entry, regional execution

What does "control plane" mean here? In this split, the control plane is everything that handles management traffic: management rpc for Dedicated Inference endpoints, plus the durable bookkeeping that turns your declared intent into running DI infrastructure. It is separate from the data plane, which is the hot path for inference (chat/completions-style) requests once an endpoint is healthy.

Central API layer: Requests originating from the DigitalOcean Cloud UI, automation workflows, or the public API are routed through a centralized API layer first. This layer maintains the mapping of endpoint ownership by region and transparently forwards each request to the appropriate regional backend. This design follows a multi-region fan-out model, where regional endpoints are addressed using stable identifiers.

Regional Dedicated Inference service: Each region operates a control-plane service responsible for the full lifecycle management of instances within its scope. This includes persisting the desired state, reconciling it with the observed state, advancing lifecycle status (e.g., creating → active), and enqueuing the workflows that provision or mutate the underlying infrastructure. In this context, lifecycle refers to the state machine governing transitions from requested to running and reachable. An instance represents the managed inference deployment as a whole—encompassing both its control-plane record and the associated backing resources.

Separate worker-style components perform integrations that need retries, backoff, and idempotency—calling the Kubernetes API, watching object status, and publishing lifecycle updates back to the core service. This is similar to the reconciler pattern familiar from Kubernetes operators: observe desired state, act, repeat until reality matches intent. Use case: transient API errors or slow node startups do not wedge the user-facing API; the system keeps trying and surfaces a clear status instead of a half-applied state

DigitalOcean Kubernetes and capacity: The platform first helps ensure that sufficient DigitalOcean Kubernetes capacity is available within the target VPC, attaches the required GPU and supporting CPU node capacity, rolls out the managed inference stack (gateway and model workloads), and creates regional network load balancers for public and private endpoints.

Data plane: VPC-native traffic, gateway-routed requests

Client Contract & Endpoint connectivity: Once an endpoint is active, clients send OpenAI-compatible API requests (for example, HTTPS to /v1/chat/completions-style routes). Your public endpoint FQDN resolves to an external regional network load balancer (L4); your private endpoint resolves to an internal regional network load balancer, so the same inference stack can be reached from the public internet or stays on your private VPC network. In both cases, traffic is OpenAI-shaped JSON carrying model ID, messages, and generation parameters.

Cluster placement and VPC isolation: Inside the VPC, workloads run on DOKS. One cluster per VPC can host multiple Dedicated Inference deployments, each isolated by a Kubernetes namespace. Desired gateway and model wiring is expressed as Custom Resources (CRDs—Custom Resource Definitions): declarative objects you kubectl apply (or the platform applies for you) instead of imperative scripts.

Inference Gateway and Kubernetes Gateway API: After the NLB, traffic hits the Inference Gateway, implemented with the upstream Kubernetes Gateway API—the community standard for describing HTTP/TLS routing into a cluster.

Gateway API Inference Extension (inference-aware routing): Below the gateway, the Gateway API Inference Extension teaches routing about inference signals: queue depth, replica health, and KV cache affinity (preferring a replica that already holds key/value tensors for a reusable prompt prefix so work is not recomputed from scratch). KV cache is the saved attention state for prior tokens; inference-aware routing is deliberately not simple round robin, because the cheapest replica is often the one that already cached your prefix.

Endpoint Picker: On CPU nodes, the Endpoint Picker is the component that talks to the Inference Extension and selects which GPU replica should receive each request.

Model Service and inference pools: On GPU nodes, the Model Service—backed by inference pools in configuration—runs your model replicas (distinct pods/processes that can serve the same model ID). Each replica reports load, KV, and cache metadata upstream so the Endpoint Picker's choices stay accurate through rollouts, crashes, and autoscaling events. Replica is a horizontally scaled copy of the same model server; pool is the grouping of those replicas for routing and capacity.

Who is Dedicated Inference for?

Dedicated Inference is aimed at builders who already know they need GPUs, but who would rather not become a full-time inference platform team:

• Teams that self-host on raw GPU Droplets or Kubernetes and want to offload orchestration, baseline optimizations, and repetitive infra work while keeping API-level ownership of their applications.
• Teams that have graduated from Serverless Inference and need hardware-level control or BYOM without abandoning managed operations.
• Organizations with consistent inference demand where predictable GPU-hour economics and performance isolation matter more than pure burst elasticity.

Inference is no longer a novelty layer; it is part of the core application stack. That shift raises the bar for reliability, performance, and cost predictability. Dedicated Inference is for teams that need production-grade, dedicated GPU inference with a managed path from model selection to a stable endpoint—so you spend engineering cycles on products and prompts, not on reinventing the serving platform.

This post details how to implement PCI-DSS-compliant data protection and audit logging on Google Kubernetes Engine (GKE). It covers customer-managed encryption keys (CMEK), tokenization, DLP scanning, and 12-month immutable audit trails. The implementation framework addresses specific PCI requirements by securing cardholder data with controlled encryption keys that can be instantly revoked during breaches, while maintaining automated logging across GKE clusters, GCS buckets, and BigQuery to answer assessor questions like "show me every time someone accessed cardholder data in the last 90 days."

Software quality is driven by user perception—shaped more by repeated issues and UI/UX experience than by isolated bugs—making trust slow to build but easy to erode. To manage this, teams should focus on monitoring user “golden paths” with symptom-based metrics tied to underlying system signals, ensuring they capture both user experience and root causes effectively.

Elasticsearch's simdvec is a hand-tuned SIMD kernel library that accelerates vector distance computations across all query types, using techniques like bulk scoring, prefetching, and architecture-specific optimizations to significantly outperform alternatives—especially at large scale when data exceeds CPU cache. Its biggest advantage comes not from raw compute speed but from efficiently hiding memory latency, enabling faster, more scalable vector search across diverse data types and hardware.

AI agents are increasingly capable of autonomously discovering and exploiting CI/CD misconfigurations, as demonstrated by a campaign targeting GitHub Actions workflows through injection, permissions abuse, and unpinned dependencies. Datadog IaC Security addresses these risks by scanning workflows pre-merge, enforcing best practices, and expanding detection coverage for triggers, supply chain integrity, and runtime security gaps.

Streaming platforms are evolving beyond scale to address operational and financial challenges through flexible architectures, interoperability, predictable pricing, and real-time visibility. Modern CDNs prioritize efficient global delivery and proactive monitoring to meet rising expectations for high-quality live event streaming.

We often treat determinism and predictability as synonyms, but they are not the same.

A system is deterministic if the same starting conditions always lead to the same result. A system is predictable if we can actually foresee that result with the tools, time, and knowledge we have.

Determinism is a system characteristic.

Predictability, on the other hand, often depends on our capabilities, and it usually exists on a spectrum. Weather is a good example. The laws of physics governing the atmosphere have not changed, and they are deterministic. Yet our ability to predict the weather has improved over decades simply because our measurements, models, and computing power improved.

But it's not always on us. Stephen Wolfram has described the concept of computational irreducibility. A system is computationally irreducible if the only way to know its future state is to simulate every step. There are also chaotic systems where tiny measurement errors grow rapidly, making them practically unpredictable.

Some systems are both deterministic and predictable, like planetary motion over short time scales. Others are deterministic but not predictably so in practice, like weather or turbulent flows. Conversely, some systems are not deterministic at the level of individual events but are still predictable statistically, such as casino games or population averages.

In essence: determinism does not guarantee predictability, and predictability does not require determinism.

Back to coding

From the perspective of someone who needs a piece of software, development has never been deterministic. When you ask a software developer to build something, you cannot predict exactly what code they will write, how long it will take (so many jokes about this…), or which edge cases will fail first. The same is true when you ask an AI agent. Both are problem-solving processes operating under uncertainty.

One can argue in favor or against the competency of humans or LLMs when it comes to coding, but determinism has never been a human trait.

In most cases, developers build software to satisfy other people's needs, and what these people really care about is whether the resulting code is predictable enough to rely on: whether the system behaves correctly most of the time, whether failures are visible, and whether they can be fixed quickly.

This distinction is important. From a software developer's pov, asking an LLM to "build a program that sorts 1000 numbers" may not have a predictable result (code). But the end user only cares if the resulting program will always sort any 1000 numbers correctly.

And then there is the environment. Modern software runs on stacks that are far more complex than any single developer can fully reason about: hardware, kernels, drivers, libraries, network conditions, configuration files, container layers, dependency versions. So, even if, as a developer, you write the code to do excatly what you want it to do, in a totally predictable way, running the code may yield less predictable results. While deterministic, the system as a whole is so complex that its behavior cannot be predicted perfectly in advance.

We have quietly accepted that bugs are a natural part of software development (i.e. there will be cases where it behaves unpredictably), because we recognize the complexity of the endeavor. Instead of expecting perfect foresight, the industry built practices around uncertainty: tests, staging environments, observability, rollbacks, reproducible builds.

So, the meaningful question is not about determinism. The meaningful question is which workflow produces more predictable outcomes under real conditions, and which human or AI is a better fit at each case/stage.

Some final thoughts

It's worth looking at how we build some of the most safety-critical software. DO-178C is the "Software Considerations in Airborne Systems and Equipment Certification". It is a key document in the aeronautic industry, providing guidelines for the development of safety-critical airborne software.

The approach of DO-178C is based on the formulation of appropriate objectives and on the verification that these objectives are achieved. The DO-178C authors acknowledged that objectives are more essential and stable than specific procedures. The ways of achieving an objective may vary between companies, and they may vary over time with the evolution of methods, techniques, and tools. DO-178C never states that one should use design method X, coding rules Y, or tool Z. DO-178C does not even impose a specific life cycle.

DO-178C is objective-oriented: the focus is on formulating objectives and verification that the objectives are achieved, a framework that could work both for human coders and LLMs.

References

1. Computational Irreducibility (Wikipedia). The image, Not Random, Blue, is inspired by the concept.

2. Software has bugs. This is normal.

3. Also depending on mood, health, work relationships and other human factors.

4. Even leaving bugs aside, there is a big discussion of which aspect of a compiler is predictable. Modern compilers tend to generate binaries that behave predictably, but are implemented in ways unexpected by the vast majority of developers. For example this article describes a case where the compiler decides to replace an O(n) algorithm written by the developer, with an O(1) one!

5. Side thought: If someone proved (not observed) that humans or LLMs can generate predictable results in some non-trivial cases, that would be really interesting. Tbh, if I had to bet that such a proof exists, I'd put my money on finding one for LLMs.

6. An introduction to DO-178C

Databricks is excited to partner with OpenAI on GPT-5.5, their latest frontier model. GPT-5.5 is OpenAI's strongest frontier model for agentic work in enterprise, complex document reasoning, and long-horizon coding agents. GPT-5.5 also now powers Codex, OpenAI's coding agent.

GPT-5.5 Features and Benefits

GPT-5.5 is the smartest frontier model yet and the next step toward a new way of getting work done. It understands what you're trying to do more quickly and can take on more of the work itself. Codex, OpenAI's coding agent, is now powered by GPT-5.5, with stronger reasoning and execution capabilities for developer workflows.

The same strengths that make GPT-5.5 great at coding also make it powerful for everyday work on a computer. Because the model is better at understanding intent, it can move more naturally through the full loop of knowledge work: finding information, understanding what matters, using tools, checking the output, and turning raw material into something useful.

It can write and debug code, research online, analyze data, create documents and spreadsheets, operate software, and move across tools until a task is finished. Instead of carefully managing every step, you can give GPT-5.5 a messy, multi-part task and trust it to plan, use tools, check its work, recover from ambiguity, and keep going.

GPT-5.5 sets the state-of-the-art performance

To understand how these improvements translate into real enterprise workloads, we evaluated GPT-5.5 on OfficeQA, Databricks' benchmark for document-heavy, multi-step analytical tasks customers perform every day. OfficeQA, built from 89,000 pages of U.S. Treasury Bulletins, measures a model's ability to retrieve information across documents, interpret complex tables, and perform precise calculations grounded in real enterprise data.

When given the right documents (OfficeQA Pro LLM with Oracle PDF + Web Search), GPT-5.5 scored 64.66%, a decent jump from GPT-5.4's 57.14%, representing a ~13% improvement and a new state-of-the-art on this benchmark. This tests the ceiling of what the model can do when retrieval is already handled.

In a full-agent workflow eval (OfficeQA Pro Agent Harness), where the model must find the right documents, parse them, and compute answers on its own using the Codex agent harness, GPT-5.5 scored 52.63%, up from GPT-5.4's 36.10%. That's a 46% reduction in errors, showing that GPT-5.5's gains aren't just theoretical; they hold up in realistic, end-to-end enterprise workflows.

GPT-5.5 is coming soon to Databricks. Bring frontier reasoning to your enterprise data, securely, and at scale.

Amazon Elastic Container Registry now automatically syncs OCI referrers like signatures, SBOMs, and attestations via pull through cache, eliminating manual retrieval and enabling seamless verification workflows across supported AWS regions.

Airtable cut archive storage costs by about 100x by moving cold, mostly immutable MySQL data into S3 as partitioned Parquet files and querying it with embedded Apache DataFusion. The dataset became 10x smaller, while S3 was about 10x cheaper per byte. A Flink-based migration, bulk and shadow validation, tiered caching, custom secondary indexes, and Parquet bloom filters preserved interactive latency and enterprise guarantees.

Internal tables store and manage both data and metadata within the database system, while external tables only store metadata and reference data that lives outside the system, leaving the underlying data untouched. Internal tables enable tighter lifecycle management, whereas external tables decouple storage and compute, making it easier to scale, share, and access large datasets without moving or duplicating data.

Background Coding Agents: Supercharging Downstream Consumer Dataset Migrations (Honk, Part 4)

This is part 4 in our series about Spotify's journey with background coding agents (internal codename: "Honk") and the future of large-scale software maintenance. See also part 1, part 2, and part 3.

In Part 2, we explored how we enabled our Fleet Management tools to use agents to rewrite our software automatically. We also explored how to write good prompts that allow the agent to best work without needing human input. In this blog post, we give a case study of how one team at Spotify used Honk with our Backstage and Fleet Management platforms to ease the pain of migrating thousands of dataset consumers onto new dataset versions — saving an estimated 10 engineering weeks in the process. We also share what we learned about how to make our data landscape more autonomous-coding-agent–friendly in the process.

Dataset migrations can be painful

As any data team knows, getting users to migrate to new endpoints can be a slow and painful process, both for the data owners and the downstream teams that use the datasets day-to-day.

At the end of last year we needed to deprecate two of the most heavily-used user datasets in order to release new versions with additional dimensions that would unlock new features. These deprecated datasets had ~1,800 direct downstream data pipelines between them and indirectly impacted several thousand more across the entire company.

We faced the prospect of migrating ~1,800 direct downstream data pipelines in only six months, across three very different pipeline frameworks that we use at Spotify: the SQL-based BigQuery Runner and dbt frameworks, and the Scala-based Scio.

We estimated that it would have taken around 10 engineering weeks of effort to complete these migrations manually. Facing that much work, we explored how Backstage, Fleet Management, and Honk might be able to automate some of the complexity.

Simplifying fleet migrations with Backstage

Before we could begin making any code changes, we had to first understand the lineage of our deprecated datasets so we would know which repositories to make those changes in. This is where Backstage's endpoint lineage and Codesearch plugins came in.

Each endpoint's Backstage page gave a clear list of downstream consumers, giving us an immediate sense of the scale of our migration. With Codesearch, we wrote queries that would find target repositories across the Spotify GitHub Enterprise landscape, and mark them as in-scope for our migrations, which we orchestrated using our Fleetshift plugin.

With Honk, context is key

As we discussed in Part 2, context engineering is a key part of the process when working with background coding agents. With our target repositories now identified quickly via Backstage, this was the part of the build that took the most time and iteration to get right, and also where we learnt the most.

One of the major challenges for Honk in this migration was the fact that it had to deal with three different data pipeline frameworks, two of which are reasonably consistent in style and substance across the company (BigQuery Runner, dbt), and one of which isn't (Scio). This lack of standardisation across our data landscape made it hard to write all-in-one prompts for Honk that could truly capture all available permutations of what it would encounter.

Although we are adding these features now, at the time of these migrations, Honk did not have access to Claude skills or custom configurability when it runs. This was a design choice made to establish guardrails around the range of possible outcomes during the migration. This meant that the prompt given to it had to be comprehensive, because it could not do things like use MCPs to go and read dataset schemas that you had not given it, or read external documentation for more context.

Trying to write a good, fully-comprehensive prompt for Scio pipelines, which can vary hugely between teams due to the relative flexibility that the framework provides, got very unwieldy without having access to outside Claude skills. We therefore made the decision not to continue trying to make Scio migrations work at that time, and focused on the other two pipeline frameworks.

For the dbt and BigQuery Runner pipeline frameworks, which were much more standardised, we initially attempted to generate a good context file by asking Claude to re-purpose a migration guide that was written for human engineers. However, the resulting context was not comprehensive enough, and Honk was left to make assumptions about how to map from one dataset field to another that were often incorrect. Once we adjusted for this, and made all mappings clear using tables in the context file — keeping in mind that Honk could only access the context we had written for it and little else — we began to see solid performance across the majority of target repositories.

Having these fine-grained instructions also allowed us to specify where Honk shouldn't try to perform a field migration, for example, in cases where a use case–specific judgement call was required. In these cases, we asked Honk to leave the fields unchanged, but to add comments above them with links to human engineer migration guides to make the task as easy as possible for the team that would later review the pull request.

One final challenge we encountered was that, unlike with Scio pipelines, the BigQuery Runner and dbt repositories across the company rarely used any build-time unit testing. This meant that one of Honk's key features, its ability to verify its work and then adjust based on the results, was unavailable to us, and we had to rely on the downstream owning teams to perform their own manual testing before merging the automated PRs.

That said, we successfully rolled out 240 automated migration PRs using Fleetshift. Here, Backstage and Fleetshift greatly simplified the ongoing monitoring and management of our shifts by providing an overview UI that gave us a snapshot view of migration progress, and the ability to easily click through and view any of the automated PRs without manually searching for the repositories. This was invaluable for troubleshooting, progress monitoring, and facilitating communication with the owning teams.

What did we learn for the future

It became clear during this project that the success of using our Fleet Management tools with Honk for large-scale, complex migrations is going to depend on the strategic push to consolidate and standardise our data landscape. Similarly, we must enforce requirements for testing and validation across repositories so that agents like Honk can verify their work in an automated fashion. Both of these elements will be critical in enabling background coding agents across Spotify.

In addition to that, there are exciting features on the Honk roadmap that will also enhance its performance on complex tasks. The Honk team is working on a feature that will allow the agent to spend some time gathering its own context, for example by reading JIRA tickets or documentation, before it begins to perform code changes. This reduces the need for such comprehensive context files to be written up front, and should improve the quality of the resulting code changes by making full use of the Claude Code capabilities.

With both of these wider, strategic changes taking place, and alongside that the underlying Claude Code agents improving in capability all the time, we look forward to seeing Fleet Management using Honk excelling on tackling more and more complex migrations in the future and reducing manual toil for our engineering teams.

Learn more about Fleet Management and our background coding agent Honk:

• Honk, Part 1: 1,500+ PRs Later: Spotify's Journey with Our Background Coding Agent
• Honk, Part 2: Context Engineering
• Honk, Part 3: Predictable Results Through Strong Feedback Loops
• On-demand webinar: How Spotify Built Honk
• Now available: Fleetshift for Spotify Portal — perform complex code changes at scale, just like we do at Spotify

Discord improved experimentation by removing redundant metrics, grouping related ones, and focusing on a small set of clearly defined “north-star” and guardrail metrics. Adding too many metrics to experiments increases multiple-testing issues and metric correlation, which can require stricter statistical corrections and make real effects harder to detect.

Databases Were Not Designed For This

There is an implicit contract at the foundation of every database architecture decision you have ever made. You probably never wrote it down. Nobody does. It just… existed.

The contract goes something like this: the caller is a human-authored application, running deterministic code, issuing predictable queries, reviewed by a developer before deployment. Writes are intentional. Connections are brief. When something goes wrong, a human notices. The database can be dumb and fast because the application layer is smart and careful.

For forty years, this contract held. It shaped how we designed schemas, sized connection pools, granted permissions, and thought about failure modes. It worked because the assumption was correct.

It is no longer correct. Agentic AI systems violate this contract at every layer simultaneously.

In this article, I break down exactly which assumptions are failing, why they matter, and what to do about it - with concrete patterns and code. Let's dig right in…

Assumption - Deterministic Caller

In every application you have deployed before agents, the queries hitting your database were authored by a human.

• developer wrote the SQL
• developer code-reviewed it
• developer tested it and deployed it.

This assumption runs so deep that the tooling reflects it automatically: the Postgres query planner builds statistics around observed query patterns, caching layers warm up on repeated queries, and connection pools are tuned around the expected number of concurrent queries of a known complexity.

Agents work differently; they reason their way to queries. Different reasoning paths produce different queries against the same tables.

An agent working on a customer analytics task might issue a join across five tables that has never been issued before, hold the connection while it thinks about the result, then issue a completely different follow-up. Your indexes cover the happy path. Your connection pool is sized for your observed peak. Neither of those holds when the agent can build any query depending on the data it needs.

Statement Timeouts

Statement timeouts are your first line of defense. A human-authored query that takes 30 seconds is a bug that someone will notice. An agent query that takes 30 seconds might be a reasoning loop that no one is watching.

So, set timeouts at the role level, not just the application level.

CREATE ROLE agent_worker;
ALTER ROLE agent_worker SET statement_timeout = '5s';
ALTER ROLE agent_worker SET idle_in_transaction_session_timeout = '10s';

The idle_in_transaction_session_timeout is especially important. Agents that pause mid-reasoning while holding an open transaction could be a legitimate situation.

Assumption - Writes are Intentional

The most dangerous assumption in database architecture is that every write was reviewed by a human before it happened. This was basically true for your entire career, but not anymore.

Agents write autonomously. They write based on their current understanding of the task, which may be wrong. Agents write in loops when their tools return unexpected results. Agents write on retries when a transient network error makes them 'think' the first attempt failed. Agents can even write thousands of rows in the time it takes you to get a Slack notification that something looks off.

Here's a real documented failure pattern - an agent calling a legacy API receives HTTP 200 with an empty result set. The API failed silently because the database connection pool was exhausted downstream. The agent interprets "no data" as "no problem" and proceeds to process 500 transactions with incomplete data. No exception was raised. No alert fired. The log showed "decision: approved" on every record.

The core fix here is to design your write paths assuming the caller might be wrong, might retry, and might not be watching the results.

Soft Deletes Everywhere

Never let an agent hard-delete anything. Use soft deletes as a baseline for any table an agent can write to

ALTER TABLE orders ADD COLUMN deleted_at TIMESTAMPTZ;
ALTER TABLE orders ADD COLUMN deleted_by TEXT; -- 'agent:customer-support-v2', 'user:abc123'
ALTER TABLE orders ADD COLUMN delete_reason TEXT;

-- Agents query this view; they never see deleted rows and can't accidentally undelete
CREATE VIEW active_orders AS
  SELECT * FROM orders WHERE deleted_at IS NULL;

The deleted_by column is more important than it looks. When you are debugging what happened two hours ago, "show me everything agent X deleted" is a query you will want to run.

Append-only Event Logs

For operations where the stakes are higher - financial records, inventory changes, user state mutations - consider going further and making the table append-only. The agent never issues UPDATE or DELETE. It issues INSERT with a new state and a reason:

CREATE TABLE order_state_log (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  order_id UUID NOT NULL REFERENCES orders(id),
  previous_status TEXT,
  new_status TEXT NOT NULL,
  changed_by TEXT NOT NULL,
  changed_at TIMESTAMPTZ DEFAULT now(),
  reason TEXT,
  idempotency_key TEXT UNIQUE
);

This is the event sourcing pattern applied at the table level. A single append-only log table for your most sensitive entities gives you a complete audit trail and makes "undo" a projection query.

Idempotency Keys Are Not Optional

Agents retry, and this is by design. Every orchestration framework operates on at-least-once delivery semantics. If a step fails, it runs again. Your write paths need to be designed for this.

An idempotency key is a stable identifier that an agent includes with every write. The database rejects duplicates silently with a unique constraint. The agent gets a successful response either way. Running the operation twice produces the same result as running it once.

-- The agent generates this key from
-- task_id + operation_type + target_id
-- It is deterministic for the same logical
-- operation, so retries produce the same key
ALTER TABLE order_state_log
 ADD CONSTRAINT uq_idempotency_key UNIQUE (idempotency_key);

In practice, the agent constructs the key like this:

import hashlib

def make_idempotency_key(task_id: str,
   operation: str, target_id: str) -> str:
    raw = f"{task_id}:{operation}:{target_id}"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

The task ID comes from the orchestration layer and is stable across retries of the same logical task. This means the agent can retry as many times as it needs to, and your database sees exactly one write per logical operation.

Assumption - Connections are Brief

Traditional connection pool sizing follows a straightforward mental model. Your application handles N concurrent requests. Each request needs one database connection for a brief period. You size your pool to slightly above your expected concurrency peak, add a little headroom, and you are done.

Agents break this model in three ways.

1. Agents hold connections longer

A multi-step reasoning task may issue a query, pause to process the result with the LLM, issue another query, pause again, and repeat. Each pause holds the connection open. The connection time per task is no longer "query execution time" - it is "query execution time + LLM inference time x reasoning steps."

1. Agents fan out

A single high-level agent task often spawns sub-agents to work in parallel. One task becomes five simultaneous database sessions. This can exhaust connections when concurrent agent workflows holding db.session open across long IO waits until Postgres ran out of connection slots.

1. Agents multiply unexpectedly

In development, you had three agents. In production, you have thirty. Nobody updated the connection pool configuration.

The fix is a dedicated connection pool for agent workloads, sized independently from your human-facing transactional application traffic

# Rule of thumb: (num_agent_workers * avg_concurrent_steps * 0.5)
# The 0.5 accounts for the fact that most agent steps
# involve LLM time, not DB time

agent_engine = create_engine(
    DATABASE_URL,
    pool_size=10,           # base pool for agents
    max_overflow=5,         # burst capacity
    pool_timeout=3,         # fail fast rather than queue
    pool_recycle=300,       # recycle connections every 5 minutes
    pool_pre_ping=True,     # validate connections before checkout
    connect_args={
        "options": "-c statement_timeout=5000 -c idle_in_transaction_session_timeout=10000"
    }
)

The pool_timeout=3 is deliberate. When an agent cannot get a connection within 3 seconds, it should fail fast and retry with backoff, not queue indefinitely. Queued requests under a saturated pool is how you get cascading failures.

For systems running many agents concurrently, add PgBouncer between your agents and Postgres. PgBouncer operates in transaction pooling mode, which means it returns a connection to the pool immediately after each transaction rather than holding it for the entire session. This is a significant multiplier on your effective connection capacity for agentic workloads.

# pgbouncer.ini
[databases]
mydb = host=postgres_host dbname=mydb

[pgbouncer]
pool_mode = transaction       # critical: release connection after each transaction
max_client_conn = 500         # clients (agents) can connect up to this number
default_pool_size = 20        # actual postgres connections (much smaller)
reserve_pool_size = 5         # emergency capacity
reserve_pool_timeout = 1.0    # fail fast if reserve is also exhausted

In transaction pooling mode, 20 actual Postgres connections can serve 500 agent connections, because each agent only holds a Postgres connection for the duration of a single transaction, not the entire multi-step task.

Assumption - Bad Queries Fail Loudly

In a human-operated system, a slow or incorrect query surfaces quickly. The dashboard loads slowly. The API times out. An engineer runs EXPLAIN ANALYZE and finds the problem. The feedback loop is tight.

Agents close that feedback loop. An agent that gets a slow query result just uses the result. An agent that gets an empty result set does not know whether the data genuinely does not exist or whether the query was wrong. It continues with its task, potentially writing decisions based on a bad read.

This is a different class of failure from application errors. An exception is observable. A semantically wrong query that returns rows is not.

The mitigation is building agent-specific observability into your database access layer. Standard slow query logs are not enough. You need to know which agent, which task, and which reasoning step produced a query. The most practical way to do this in Postgres is query comments

from sqlalchemy import text, event
from sqlalchemy.engine import Engine

@event.listens_for(Engine, "before_cursor_execute")
def add_agent_context_comment(conn, cursor, statement, parameters, context, executemany):
    agent_ctx = getattr(conn.info, "agent_context", None)
    if agent_ctx:
        statement = f"/* agent_id={agent_ctx['agent_id']}, task_id={agent_ctx['task_id']}, step={agent_ctx['step']} */ {statement}"
    return statement, parameters

# Usage: set context on the connection before executing
with engine.connect() as conn:
    conn.info["agent_context"] = {
        "agent_id": "fulfillment-v3",
        "task_id": "task-abc-123",
        "step": "check-inventory"
    }
    conn.execute(text("SELECT ..."))

These comments appear in pg_stat_activity, pg_stat_statements, and your slow query logs. A query that appears in your slow query log tagged agent_id=fulfillment-v3, task_id=task-abc-123, step=check-inventory is immediately actionable. Without this, you are doing archaeology.

Build a monitoring view that surfaces queries grouped by agent:

-- pg_stat_statements with agent context extracted from query text
SELECT
  (regexp_match(query, 'agent_id=([^,]+)'))[1] AS agent_id,
  (regexp_match(query, 'task_id=([^,]+)'))[1] AS task_id,
  count(*) AS call_count,
  round(mean_exec_time::numeric, 2) AS avg_ms,
  round(total_exec_time::numeric, 2) AS total_ms
FROM pg_stat_statements
WHERE query LIKE '%agent_id=%'
GROUP BY 1, 2
ORDER BY total_ms DESC;

When you see a single agent type accounting for 60% of total database time, you know where to look.

Assumption - Schema is a Contract With Engg

This is the assumption that most teams never think about until it breaks. Your schema was designed for developer ergonomics - named to make sense to the engineers, structured for query convenience, with nullable columns that "mean something" only if you read the original migration comment.

When an agent can see your schema - through Text-to-SQL, through tool definitions, through an MCP server wrapping your database - the schema becomes a contract with a language model. Column names, table structure, and nullability now affect whether the LLM generates correct queries or confident-sounding nonsense.

Consider the difference between these two column definitions

-- What most schemas look like
CREATE TABLE orders (
  id UUID PRIMARY KEY,
  usr_id UUID,           -- which user?
  stat_cd INT,           -- what does 2 mean? what does 7 mean?
  flg_1 BOOLEAN,         -- ???
  upd_ts TIMESTAMPTZ     -- updated at? but by whom?
);

-- What a schema legible to an agent looks like
CREATE TABLE orders (
  id UUID PRIMARY KEY,
  customer_id UUID NOT NULL REFERENCES customers(id),
  fulfillment_status TEXT NOT NULL CHECK (
    fulfillment_status IN ('pending', 'processing', 'shipped', 'delivered', 'cancelled')
  ),
  requires_signature BOOLEAN NOT NULL DEFAULT false,
  last_modified_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

The second schema generates correct LLM queries almost automatically. The first schema requires extensive prompt engineering to compensate for what should have been done at the schema level.

For schemas you cannot rename (legacy systems, high-migration-cost tables), build an agent-facing view layer

-- The raw table retains its legacy names
-- Agents query this view; they never touch the underlying table directly
CREATE VIEW agent_orders AS
SELECT
  id,
  usr_id         AS customer_id,
  CASE stat_cd
    WHEN 1 THEN 'pending'
    WHEN 2 THEN 'processing'
    WHEN 5 THEN 'shipped'
    WHEN 7 THEN 'delivered'
    WHEN 9 THEN 'cancelled'
  END            AS fulfillment_status,
  flg_1          AS requires_signature,
  upd_ts         AS last_modified_at
FROM orders
WHERE deleted_at IS NULL;  -- agents only ever see active rows

Write column comments as if they are docstrings - because for Text-to-SQL agents, they are:

COMMENT ON COLUMN agent_orders.fulfillment_status IS
  'Current state of the order in the fulfillment pipeline. '
  'Use this to filter orders that need action: pending and processing orders are active. '
  'Cancelled orders should never be modified.';

COMMENT ON COLUMN agent_orders.requires_signature IS
  'True if the delivery requires an adult signature. '
  'When true, the shipping agent must schedule a delivery window.';

Scoping Blast Radius

There is one more failure mode worth treating separately, because it cuts across all the assumptions above: the blast radius of a misbehaving agent is determined by the access it was granted.

Traditional applications share a database role, or at best have a few roles for different services. The assumption was that the application code was the guard rail. If the code only allowed users to update their own records, the database role did not need to enforce that - the application layer handled it.

Agents make this assumption dangerous. An agent that reasons itself into an incorrect state can issue queries that the application developers never anticipated. The agent is not a known, finite set of code paths - it is a general-purpose reasoner with access to a database connection. Application-layer guardrails do not bind it the way they bind deterministic code.

The fix is role-per-agent-type access, with the minimum necessary privileges defined at the database level:

-- Each agent type gets its own role
CREATE ROLE agent_fulfillment;
CREATE ROLE agent_customer_support;
CREATE ROLE agent_analytics;

-- agent_analytics: read-only, only the tables it needs
GRANT SELECT ON agent_orders TO agent_analytics;
GRANT SELECT ON customers TO agent_analytics;
-- Explicitly: no access to payments, credentials, PII tables

-- agent_customer_support: can update order status, cannot touch financials
GRANT SELECT ON agent_orders TO agent_customer_support;
GRANT INSERT ON order_state_log TO agent_customer_support;
-- Does not have UPDATE on orders -- changes go through the event log

-- agent_fulfillment: can read and update shipping-related fields only
GRANT SELECT, UPDATE (fulfillment_status, shipped_at, tracking_number)
  ON orders TO agent_fulfillment;

The question to ask in your access design review is not "what does this agent need?" but "what is the worst case if this agent's reasoning goes wrong, or if its credentials are compromised?" Reduce that blast radius at the database level, where it cannot be reasoned around.

Defensively Designed Data Layer

Pulling this together, here is what the data layer looks like for a team that has internalized these failure modes. None of it is exotic. All of it exists in battle-tested database tooling.

Every agent type has its own database role with the minimum necessary privileges, enforced at the database level with role-level timeouts. Agents connect through a dedicated connection pool, sized for agentic workload patterns and separated from human-facing traffic. PgBouncer runs in transaction pooling mode between agents and Postgres.

Tables that agents can write to use soft deletes with a deleted_by column that captures agent identity. High-stakes write paths use append-only event log tables with idempotency key constraints. Every write carries an agent ID and task ID so the audit trail is always traversable.

Schema objects that agents can see are named for legibility, not legacy convenience. A maintained view layer translates legacy column names to meaningful ones. Column comments are written as docstrings. Agents are granted access to views, not directly to underlying tables.

Every query issued by an agent carries a comment with the agent ID, task ID, and reasoning step. A monitoring dashboard aggregates this data so the on-call engineer can see "agent X consumed 40% of database time in the last hour" in real time.

The circuit breakers are defined: max writes per task enforced in the orchestration layer, max rows affected per statement enforced via statement complexity checks, max task duration enforced with a watchdog process that terminates stalled agent sessions.

None of this is new technology. Soft deletes, append-only logs, least-privilege roles, row-level security, idempotency keys, query tagging - these are patterns that have existed for years. The shift that agents force is that these patterns go from "best practice we keep meaning to implement" to "load-bearing infrastructure." Agents do not give you the luxury of deferring them.

The database was not designed for this caller. But the tools to make it safe are already there.

Conclusion and Footnote

Traditional database architecture rests on assumptions that agentic AI workloads systematically violate: deterministic callers, intentional writes, brief connections, loud failures, and schema as a developer contract.

Each of these assumptions held because a human was always somewhere in the loop. Agents remove that guarantee. The result is that patterns long treated as optional best practice - soft deletes, append-only logs, idempotency keys, least-privilege roles, query tagging - become load-bearing infrastructure.

None of this requires new technology. It requires treating the database as a defensive layer that assumes the caller might be wrong, might retry, and might not be watching the results.

Cloud high availability can no longer assume regions are safe, independent failure domains: sanctions, data localization laws, conflict zones, and submarine cable cuts can take out an entire region or make it noncompliant. Treat region-level disruption as a first-class risk, with multi-region, jurisdiction-aware data placement, control-plane separation, and dependency audits. The added cost and complexity should be justified with Annual Loss Expectancy modeling rather than assumed.

Data platform decisions should start with use cases, constraints, and operating requirements, not with Kafka, Spark, Snowflake, or Airflow. The key questions are latency, data freshness, cost, failure handling, and who will consume the system. Choose the simplest stack that fits the problem, team, budget, and timelines.

tda-mapper

tda-mapper is a Python library built around the Mapper algorithm, a core technique in Topological Data Analysis (TDA) for extracting topological structure from complex data. Designed for computational efficiency and scalability, it leverages optimized spatial search methods to support high-dimensional datasets. The library is well-suited for integration into machine learning pipelines, unsupervised learning tasks, and exploratory data analysis.

Further details in the documentation and in the paper.

Core Features

• Efficient construction

  Leverages optimized spatial search techniques and parallelization to accelerate the construction of Mapper graphs, supporting the analysis of high-dimensional datasets.

• Scikit-learn integration

  Provides custom estimators that are fully compatible with scikit-learn's API, enabling seamless integration into scikit-learn pipelines for tasks such as dimensionality reduction, clustering, and feature extraction.

• Flexible visualization

  Multiple visualization backends supported (Plotly, Matplotlib, PyVis) for generating high-quality Mapper graph representations with adjustable layouts and styling.

• Interactive app

  Provides an interactive web-based interface for dynamic exploration of Mapper graph structures, offering real-time adjustments to parameters and visualizations.

Background

The Mapper algorithm extracts topological features from complex datasets, representing them as graphs that highlight clusters, transitions, and key structural patterns. These insights reveal hidden data relationships and are applicable across diverse fields, including social sciences, biology, and machine learning. For an in-depth overview of Mapper, including its mathematical foundations and practical applications, read the original paper.

Step 1	Step 2	Step 3	Step 4
Choose lens	Cover image	Run clustering	Build graph

Quick Start

Installation

To install the latest version uploaded on PyPI

pip install tda-mapper

How to Use

Here's a minimal example using the circles dataset from scikit-learn to demonstrate how to use tda-mapper. This example demonstrates how to apply the Mapper algorithm on a synthetic dataset (concentric circles). The goal is to extract a topological graph representation using PCA as a lens and DBSCAN for clustering. We proceed as follows:

import matplotlib.pyplot as plt
from sklearn.datasets import make_circles

import numpy as np
from sklearn.decomposition import PCA
from sklearn.cluster import DBSCAN

from tdamapper.learn import MapperAlgorithm
from tdamapper.cover import CubicalCover
from tdamapper.plot import MapperPlot

# Generate toy dataset
X, labels = make_circles(n_samples=5000, noise=0.05, factor=0.3, random_state=42)
plt.figure(figsize=(5, 5))
plt.scatter(X[:,0], X[:,1], c=labels, s=0.25, cmap="jet")
plt.axis("off")
plt.show()

# Apply PCA as lens
y = PCA(2, random_state=42).fit_transform(X)

# Mapper pipeline
cover = CubicalCover(n_intervals=10, overlap_frac=0.3)
clust = DBSCAN()
graph = MapperAlgorithm(cover, clust).fit_transform(X, y)

# Visualize the Mapper graph
fig = MapperPlot(graph, dim=2, seed=42, iterations=60).plot_plotly(colors=labels)
fig.show(config={"scrollZoom": True})

Original Dataset	Mapper Graph

Left: the original dataset consisting of two concentric circles with noise, colored by class label. Right: the resulting Mapper graph, built from the PCA projection and clustered using DBSCAN. The two concentric circles are well identified by the connected components in the Mapper graph.

More examples can be found in the documentation.

Interactive App

Use our app to interactively visualize and explore your data without writing code. You can try it right away using our live demo, or run it locally on your machine.

To run it locally:

1. Install the app and its dependencies:

   pip install tda-mapper[app]

2. Launch the app:

   tda-mapper-app

Citations

If you use tda-mapper in your work, please consider citing both the library, archived in a permanent Zenodo record, and the paper, which provides a broader methodological overview. We recommend citing the specific version of the library used in your research, along with the paper. For citation examples, please refer to the documentation.

Installing and Loading

INSTALL whisper FROM community;
LOAD whisper;

Example

-- Transcribe an audio file
D SELECT whisper_transcribe('audio.wav', 'tiny.en');
┌──────────────────────────────────────────────────┐
│           whisper_transcribe(...)                │
│                   varchar                        │
├──────────────────────────────────────────────────┤
│ Hello, this is a test of the whisper extension.  │
└──────────────────────────────────────────────────┘

-- Get detailed transcription segments with timestamps
D SELECT * FROM whisper_transcribe_segments('audio.wav', 'tiny.en');
┌────────────┬────────────┬──────────┬────────────────────┬────────────┬──────────┐
│ segment_id │ start_time │ end_time │        text        │ confidence │ language │
│   int32    │   double   │  double  │      varchar       │   double   │ varchar  │
├────────────┼────────────┼──────────┼────────────────────┼────────────┼──────────┤
│          0 │       0.00 │     2.50 │ Hello, this is a   │       0.95 │ en       │
│          1 │       2.50 │     4.00 │ test of whisper.   │       0.92 │ en       │
└────────────┴────────────┴──────────┴────────────────────┴────────────┴──────────┘

-- List available models
D SELECT model_name, size_mb, is_downloaded FROM whisper_list_models() LIMIT 3;
┌────────────┬─────────┬───────────────┐
│ model_name │ size_mb │ is_downloaded │
│  varchar   │  int64  │    boolean    │
├────────────┼─────────┼───────────────┤
│ tiny       │      75 │ false         │
│ tiny.en    │      75 │ true          │
│ base       │     142 │ false         │
└────────────┴─────────┴───────────────┘

About whisper

A DuckDB extension for speech-to-text transcription using whisper.cpp, the C/C++ port of OpenAI's Whisper model.

Transcribe audio files directly from SQL queries in DuckDB, making it easy to process and analyze audio data alongside your other data.

Features

• Transcribe audio files (WAV, MP3, FLAC, OGG, and more)
• Live recording and transcription from microphone
• Voice-to-SQL: speak natural language questions, get query results
• Support for all Whisper models (tiny, base, small, medium, large)
• Detailed transcription segments with timestamps and confidence scores
• Automatic language detection or specify target language
• Works with file paths, BLOB data, or remote URLs

Quick Start

Download a Model

Models must be downloaded before use. They are stored in ~/.duckdb/whisper/models/.

mkdir -p ~/.duckdb/whisper/models
curl -L -o ~/.duckdb/whisper/models/ggml-tiny.en.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.en.bin

Check available models and download status:

SELECT * FROM whisper_list_models();

Transcribe an Audio File

-- Simple transcription
SELECT whisper_transcribe('audio.wav', 'tiny.en');

-- Get detailed segments with timestamps
SELECT * FROM whisper_transcribe_segments('audio.wav', 'tiny.en');

-- Translate foreign audio to English
SELECT whisper_translate('german_speech.mp3', 'small');

Example Use Cases

Transcribe Remote Audio

INSTALL httpfs;
LOAD httpfs;

SELECT whisper_transcribe(content, 'tiny.en')
FROM read_blob('https://example.com/audio.mp3');

Batch Transcribe Multiple Files

SELECT file, whisper_transcribe(file, 'tiny.en') as transcript
FROM glob('audio/*.wav');

Search Within Transcriptions

SELECT * FROM whisper_transcribe_segments('meeting.wav', 'base.en')
WHERE text ILIKE '%action item%';

Generate Subtitles (SRT Format)

SELECT
    segment_id + 1 as id,
    printf('%02d:%02d:%02d,%03d',
        (start_time/3600)::int, ((start_time%3600)/60)::int,
        (start_time%60)::int, ((start_time - start_time::int) * 1000)::int
    ) || ' --> ' ||
    printf('%02d:%02d:%02d,%03d',
        (end_time/3600)::int, ((end_time%3600)/60)::int,
        (end_time%60)::int, ((end_time - end_time::int) * 1000)::int
    ) as timestamp,
    trim(text) as text
FROM whisper_transcribe_segments('video.mp4', 'small.en');

Recording (requires microphone)

Setup Audio Device

Before using microphone recording or voice query features:

-- List all available audio input devices
SELECT * FROM whisper_list_devices();

-- Set the device ID (use a device_id from the list above)
SET whisper_device_id = 0;

-- Verify your microphone is working
SELECT whisper_mic_level(3);

Record and Transcribe

-- Record for 5 seconds
SELECT whisper_record(5, 'tiny.en');

-- Record until silence (max 30 seconds)
SELECT whisper_record_auto(30);

-- Record and translate to English
SELECT whisper_record_translate(5, 'small');

Voice-to-SQL (Experimental)

Speak natural language questions about your data and receive SQL query results.

Requires text-to-sql-proxy running locally.

-- Create test data
CREATE TABLE customers (id INT, name VARCHAR, revenue DECIMAL);
INSERT INTO customers VALUES (1, 'Acme', 100000), (2, 'Beta', 50000);

-- Get SQL from voice (doesn't execute)
SELECT whisper_voice_to_sql();

-- Execute voice query directly
FROM whisper_voice_query();

-- Include generated SQL in results
FROM whisper_voice_query_with_sql();

Available Models

Models with .en suffix are optimized for English.

Supported Audio Formats

The extension uses FFmpeg for audio decoding: WAV, MP3, FLAC, OGG/Vorbis, AAC/M4A, and many more. Audio is automatically converted to 16kHz mono as required by Whisper.

Function Reference

Transcription Functions

• whisper_transcribe(audio, [model]) - Transcribes audio and returns the full text
• whisper_translate(audio, [model]) - Translates audio from any language to English
• whisper_transcribe_segments(audio, [model], [language], [translate]) - Returns table of segments with timestamps

Recording Functions

• whisper_list_devices() - Lists available audio input devices
• whisper_record(duration_seconds, [model], [device_id]) - Records and transcribes
• whisper_record_auto(max_seconds, [silence_seconds], [model], [threshold], [device_id]) - Records until silence
• whisper_record_translate(duration_seconds, [model], [device_id]) - Records and translates to English
• whisper_mic_level(duration_seconds, [device_id]) - Check microphone amplitude levels

Voice-to-SQL Functions

• whisper_voice_to_sql([model], [device_id]) - Records voice and returns generated SQL
• whisper_voice_query([model], [device_id]) - Records voice, generates SQL, executes it
• whisper_voice_query_with_sql([model], [device_id]) - Same as above with SQL columns

Model Management Functions

• whisper_list_models() - Lists all available models and download status
• whisper_download_model(model_name) - Returns download instructions

Utility Functions

• whisper_version() - Returns extension and whisper.cpp version info
• whisper_check_audio(file_path) - Validates that an audio file can be read
• whisper_audio_info(file_path) - Returns audio file metadata
• whisper_get_config() - Returns current whisper configuration settings

Configuration

Configure settings using standard SET statements:

-- Model settings
SET whisper_model = 'small.en';
SET whisper_model_path = '/custom/path/models';
SET whisper_language = 'en';
SET whisper_threads = 4;

-- Recording settings
SET whisper_device_id = 0;
SET whisper_max_duration = 30;
SET whisper_silence_duration = 2;
SET whisper_silence_threshold = 0.005;

-- Voice query settings
SET whisper_text_to_sql_url = 'http://localhost:8080/generate-sql';
SET whisper_text_to_sql_timeout = 60;
SET whisper_voice_query_show_sql = true;

-- View all whisper settings
SELECT * FROM duckdb_settings() WHERE name LIKE 'whisper_%';

See the GitHub repository for full documentation.

Jaeger adopts OpenTelemetry at its core to solve the AI agent observability gap

Jaeger v2 uses OpenTelemetry and MCP to trace GenAI pipelines and facilitate collaboration between engineers and AI agents for observability.

As software architectures evolve, observability tools must adapt. When the industry moved to microservices, distributed tracing became a necessity. Jaeger emerged as a core tool for engineers to understand those fragmented systems. Now, as organizations integrate generative AI applications and autonomous agents into production, tracing requirements are shifting again. Mapping the execution path of an AI agent involves prompt assembly, vector database retrievals, and multiple external tool calls.

"By adopting the Model Context Protocol (MCP), Agent Client Protocol (ACP), and Agent–User Interaction Protocol (AG-UI), the project is building an environment where engineers and AI agents can collaborate."

Jaeger is evolving to address these new workloads. This transition involves two main phases. First, the project rebuilt its core architecture in Jaeger v2 to natively integrate OpenTelemetry. Second, Jaeger is expanding beyond standard data visualization. By adopting the Model Context Protocol (MCP), Agent Client Protocol (ACP), and Agent–User Interaction Protocol (AG-UI), the project is building an environment where engineers and AI agents can collaborate. This helps map the complex execution paths of AI pipelines that often stretch the limits of traditional tracing tools.

Setting the foundation: Jaeger v2

Managing AI workloads requires an efficient data collection pipeline. This need guided the architectural changes detailed in the CNCF blog post, Jaeger v2 released: OpenTelemetry in the core!

Jaeger v2 replaces its original collection mechanisms with the OpenTelemetry Collector framework. This approach consolidates metrics, logs, and traces into a unified deployment model. By natively ingesting the OpenTelemetry Protocol (OTLP), the system eliminates intermediate translation steps, improving ingestion performance. This OpenTelemetry integration provides the necessary data foundation for more advanced tracing features.

Human and agent collaboration

Building on Jaeger v2, the project is exploring new ways for teams to analyze distributed systems. The goal is to facilitate collaboration between engineers and AI agents during debugging. Contributors from the CNCF LFX Mentorship program and Google Summer of Code (GSoC) are actively driving this work.

To support AI integration, Jaeger is adopting three open standards: the Model Context Protocol (MCP), Agent Client Protocol (ACP), and Agent–User Interaction Protocol (AG-UI). MCP standardizes how AI models securely access external data sources. ACP provides a uniform method for user interfaces to communicate with AI agents and sidecars. Together, they allow Jaeger to function as an interactive workspace.

Building the backend protocol layer

The technical implementation starts in the backend. We are building an Agent Client Protocol layer to act as a stateless translator between the Jaeger frontend and external AI sidecars. The design and proof of concept are documented in Jaeger backend issue #8252 (Implement AG-UI to ACP Jaeger AI) and issue #8295 (Implement ACP-based AI handler).

graph LR
  J_UI["Jaeger UI"]
  AI_A["AI Agent"]
  subgraph JAEGER["Jaeger v2"]
    AGW["Agent Gateway"]
    JMCP["Jaeger MCP"]
  end

  J_UI -- "AG-UI Protocol" --> AGW
  AGW -- "ACP Protocol" --> AI_A
  AGW -- "MCP Protocol" <--> JMCP

Traditionally, incident responders build queries by manually filtering services and tags. The ACP integration allows the backend to parse natural-language constraints (such as identifying 500-level errors in a payment service with latency exceeding 2 seconds) and translate them into deterministic trace queries.

Organizations can configure this backend to use cloud-based large language models (LLMs) for complex reasoning or local small language models (SLMs) for strict data privacy. The depth of analysis depends on the chosen model, as noted in industry analyses of hosted versus local AI infrastructure. By restricting the AI to protocol translation and query generation, the architecture minimizes the risk of hallucinations associated with open-ended chatbots.

The collaborative UI workspace

The Jaeger user interface is also being updated to support this backend logic. As tracked in Jaeger UI issue #3313, we are migrating from legacy Redux to modern Zustand + React Query.

The frontend introduces an in-app assistant powered by assistant-ui + AG-UI. The UI uses streaming events to send trace context (such as error logs and key-value tags) to the backend gateway. This allows engineers to prompt the assistant to summarize a failure path within a specific span, reducing the need to manually review raw log lines during an incident.

Visualizing GenAI execution paths

Beyond using AI to analyze standard traces, the project is adding support for tracing the AI applications themselves.

Outlined in Jaeger issue #8401 (GenAI integration), this work focuses on visualizing the rapidly evolving OpenTelemetry GenAI semantic conventions. The OpenTelemetry community is currently drafting specifications to standardize telemetry for these highly dynamic workflows. Key initiatives include emerging drafts for Generative AI Agentic Systems (Issue #2664) to track tasks, memory, and actions, and conventions for AI Sandboxes (Issue #3583) to monitor ephemeral code execution environments.

"Jaeger will map these new standard operations in the UI to provide clear visibility into AI execution paths without locking teams into vendor-specific formats."

Developers building Retrieval-Augmented Generation (RAG) pipelines and autonomous agents need to measure embedding model latency, track external tool calls, and monitor token usage. Jaeger will map these new standard operations in the UI to provide clear visibility into AI execution paths without locking teams into vendor-specific formats.

Unified observability: from local testing to production

Maintaining consistency across testing and production environments is a practical challenge. Jaeger originally popularized an "all-in-one" executable to simplify local testing. Because Jaeger v2 is built on the OpenTelemetry Collector, developers run the exact same binary locally as they do in production.

During testing, engineers can run the Jaeger v2 container with a local SLM. This creates a private sandbox for testing generative AI traces or debugging ACP integrations without exposing data to external APIs.

In production, platform teams deploy the same unified binary, often using tools like the OpenTelemetry Operator for Kubernetes. Organizations can then replace the local SLM with a larger cloud-based LLM to handle production incident analysis. This ensures that tracing configurations remain consistent from development to deployment.

What's next

Tracing requirements are shifting to accommodate the complexity of AI applications. By establishing a solid OpenTelemetry foundation with Jaeger v2 and integrating MCP and ACP standards, the project is adapting its core capabilities. This technical path forward enables a practical workflow where human engineers and AI agents collaborate to diagnose distributed system failures.

Fixing What LLMs Get Wrong

A Field Guide to Verification, Repair, and Self-Improvement

Part I - The Problem Worth Solving

The Hallucination Tax

There is a particular kind of operational failure that only emerges when you deploy a language model against real, private, structured knowledge. The model is fluent. Its grammar is impeccable. Its tone matches your internal documentation. And it is wrong about the things that matter most.

Not wrong in a detectable way - not grammatically wrong, not logically incoherent, not obviously absurd. Wrong in the way that a confident, articulate person is wrong when they misremember a number or confuse two similar-sounding policies. The answer sounds authoritative because the model's generation process is entirely indifferent to whether the specific fact it's producing is true. It is optimized for plausibility at the token level, not for correctness at the fact level.

This is the hallucination tax, and every organization building LLM-powered systems against proprietary knowledge pays it.

Ask an internal assistant what the NovaPilot Starter plan costs and it may confidently return $5,000 per month when the correct figure, encoded in your own knowledge graph, is $2,500. Ask it when a key executive joined and it will synthesize a plausible-sounding date from the statistical texture of similar corporate biographies. It will tell you "NovaAI was founded in 2019" because 2019 is a frequent founding year among AI companies in its training distribution, not because it knows anything specific about NovaAI.

The cost is not merely embarrassment. In domains where specific numbers govern contracts, where policy thresholds determine eligibility, where dates establish legal precedence - the hallucination tax accumulates as real operational risk.

The question that shapes this entire discussion is not "why do LLMs hallucinate?" That question has been answered well enough.

The productive question is: given that they do, what does a trustworthy system built on top of them actually look like?

wait, why are we even talking about it in 2026? we have fine-tuning, we have 30+ types of production-used RAG architectures. that should be enough... or so we thought.

Why the Obvious Solutions Are Incomplete

The engineering response to hallucination has followed a familiar three-stage progression.

The first response was fine-tuning. If the model doesn't know your domain, teach it. Write knowledge into the weights. This approach has an appealing directness - the model that emerges should, in principle, know the facts you care about. In practice, it fails for three reasons. First, the computational cost of fine-tuning frontier models at organizational scale is non-trivial. Second, private-domain knowledge is not static - pricing changes, personnel turns over, policies update - and fine-tuning cannot track a living knowledge base without continuous re-investment. Third, and most subtly, fine-tuning conflates model capability with model knowledge. You are retraining everything to change a few things, which is the engineering equivalent of reprogramming a city's traffic system every time a road is renamed.

A 2026 paper on continuous knowledge drift says continual pretraining or fine-tuning is often effective only at a cost: it is "computationally expensive," requires repeated retraining as knowledge evolves, and is vulnerable to catastrophic forgetting.

That matters because enterprise facts are rarely static:

• prices change
• org charts change
• policies change
• product bundles change

A separate 2026 study, Why Supervised Fine-Tuning Fails to Learn, reports a persistent Incomplete Learning Phenomenon: even after convergence, models can still fail to reproduce part of the very supervised data they were trained on. The paper says incomplete learning is "widespread and heterogeneous," and that aggregate metrics can hide unlearned subsets.

Fine-tuning is good for behavioral adaptation and domain shaping, but it is a poor single source of truth for fast-changing factual knowledge. And even when used, it does not guarantee complete or durable fact internalization.

The second response was retrieval-augmented generation, or RAG. Rather than writing knowledge into the weights, inject it at inference time. Retrieve relevant context from your knowledge store and prepend it to the prompt. This is substantially better. The model can now be grounded in documents or structured data it has never seen before. But RAG has a structural limitation that is easy to miss: it augments generation, it does not verify it. The model with retrieved context still generates freely. If the retrieved context is incomplete, ambiguous, or misaligned with the query, the model fills the gap with its training priors. More critically, RAG does not teach the model anything. The next query starts from the same baseline. The same category of error recurs.

A 2025 paper, ASTUTE RAG, explicitly argues that "imperfect retrieval augmentation might be inevitable and quite harmful." In their controlled analysis, they report that under realistic conditions roughly 70% of retrieved passages did not directly contain the true answer.

That is a big deal. It means that even if you build a retrieval pipeline, the model is often operating on:

• incomplete evidence
• irrelevant evidence
• mixed evidence
• conflicting evidence

A 2026 paper, Stable-RAG, shows something even more uncomfortable: RAG systems are "far from hallucination-free," and merely reordering the same retrieved documents can change the answer. In some cases, the model ignored the gold document even when it was present.

That means the failure is not only retrieval quality. It is also evidence integration instability inside the generator.

A 2025 paper, FVA-RAG, identifies retrieval sycophancy: if the user asks from a false premise, the retriever may fetch documents aligned with that false framing, causing the model to "hallucinate with citations."

RAG is necessary in many real systems, but it is not a truth machine. It improves access to current information, but does not guarantee:

• correct retrieval
• correct selection among retrieved evidence
• correct reconciliation of conflicting evidence

The third response was static verification - a catch-and-correct pipeline. Generate an answer, decompose it into claims, check each claim against a knowledge source, flag what is wrong, rewrite the flagged content. This is architecturally sound and operationally useful. In a system built against a knowledge graph, the static verifier compares each extracted claim against the graph's triples and returns a structured verdict: SUPPORTED, CONTRADICTED, UNVERIFIABLE. When a claim contradicts the graph, a repairer rewrites it using the correct evidence.

Static verification catches errors. It does not prevent them. The model answers the next question with no memory of having been wrong on the last one. Each session starts with the same hallucination propensity, and the verification layer must work just as hard each time. The catch-and-correct pipeline is an error filter, not a learning system.

These three approaches - fine-tuning, RAG, and static verification - are not wrong. They are incomplete. Each addresses one layer of the problem while leaving another layer untouched. Fine-tuning embeds knowledge but cannot track it. RAG grounds generation but cannot verify it. Static verification catches errors but cannot accumulate lessons.

The gap these three approaches share is the same gap: none of them improve through experience.

The Productive Question

Here, then, is the constraint that shapes everything that follows:

• We have a model whose weights we cannot - or prefer not to - continuously update.
• We have a private knowledge base that is authoritative, structured, and current.
• We have a verification mechanism that can detect when the model is wrong.

The open question is whether the system as a whole can improve across sessions - not by retraining the model, but by restructuring how failure is processed and retained.

Phrased more precisely: can we build a pipeline where each hallucination, caught and corrected, produces a durable lesson that makes subsequent answers less likely to contain the same category of error?

Can a frozen-weight system learn through its architecture rather than through its parameters?

The Reflexion framework, introduced by Shinn et al. at NeurIPS 2023, is the first rigorous answer to this question. It is not the only answer - and subsequent approaches have extended, refined, and in some cases superseded it - but it is the foundational one, and understanding it deeply is prerequisite to understanding what came after.

Part II - Reinforcement Without Gradients

What Traditional RL Requires

Reinforcement learning, in its standard formulation, is an optimization algorithm. An agent acts in an environment, receives a reward signal, and updates its policy to maximize cumulative reward. The update mechanism is typically gradient descent through parameter space: the reward signal backpropagates through the model, nudging weights toward configurations that produce better outcomes.

This works spectacularly well in narrow, simulable domains - games, robotic control, token-level optimization. It works poorly in the context of language model deployment for several compounding reasons.

The sample complexity problem comes first. Gradient-based policy optimization requires a large number of trials before the signal-to-noise ratio in the reward gradient becomes useful. For a language model, each trial is an LLM inference call. At the scale of samples required for meaningful policy improvement, the cost in compute, latency, and dollars is prohibitive for most production environments.

The reward design problem follows closely. Defining a reward function that accurately captures what "a good answer" means is harder than it appears. A scalar reward - correct or incorrect - provides no gradient information about which part of a complex answer was wrong. The model improved its score, but we cannot easily attribute the improvement to any specific aspect of its behavior. This is the credit assignment problem: when a trajectory spans multiple decisions and produces a single terminal reward, distributing that reward backward through time so that each action receives appropriate credit is computationally expensive and theoretically fraught.

Finally, and most practically: traditional RL requires differentiable access to the model's parameters. For closed-weight APIs, this access does not exist. The model is a black box. You can observe its outputs; you cannot nudge its internals.

Traditional RL is the right algorithm if you have a simulable environment, abundant compute, a well-defined reward, and parameter access. In production language model deployment, you typically have none of these.

The Reflexion Hypothesis

Reflexion begins from a different premise. If gradient descent is unavailable, what else can carry the learning signal? The hypothesis: language itself.

A large language model is not merely a generator of text. It is a system that conditions its outputs on its inputs with remarkable sensitivity. When the system prompt says "always verify tier names against their specific price points," the model produces different outputs than when it doesn't. That difference is not mediated by weight updates. It is mediated by context. The instruction modifies behavior as surely as a gradient - with less precision, perhaps, but also with considerably less cost.

Reflexion formalizes this intuition. The core mechanism is a feedback loop: the agent acts, the environment evaluates, the agent reflects on the evaluation in natural language, and that reflection is stored and injected into future action prompts. The weights never change. The policy changes because the context changes.

Reflexion Hypothesis:

LLMs can improve across attempts if failure feedback is translated into natural-language reflections, stored as episodic memory, and reinserted into future prompts.

This is verbal reinforcement learning - reinforcement not through gradient updates but through the accumulation of linguistically encoded experience.

The mechanism is more subtle than it first appears. The reflection is not simply a correction of the specific wrong answer. It is an abstraction - a generalized lesson derived from the particular failure. The difference between:

• "NovaAI was founded in 2019 is wrong" and
• "the model tends to infer founding dates from executive tenure rather than explicit founding records"

is the difference between error correction and category learning. The former fixes one answer. The latter alters a behavioral pattern across an entire category of future questions.

Part III - Anatomy of a Reflexion Agent

The Tripartite System

A Reflexion agent is not a single model. It is three models - or, more precisely, three distinct roles that may be instantiated by the same model with different prompting. The distinction is architectural, not necessarily computational.

These three components are:

• the Actor,
• the Evaluator, and
• the Self-Reflection model.

Their interaction is precisely defined, and understanding each in isolation before examining how they work together reveals why this architecture is both elegant and practically grounded.

This is not RL in the classical sense of gradient descent over policy parameters. But it is still reinforcement in the broader sense, because:

• the agent acts
• receives external evaluation
• converts evaluation into a better future policy

The difference is where the policy lives.

The Actor: Trajectory Generator and Policy Carrier

The Actor is the policy engine. It is the component that takes the current state of the world - a question, a task, an environment - and produces an action: an answer, a decision, a sequence of tool calls. In the standard language model framing, the Actor is simply an LLM invoked with a system prompt.

But the Actor is more precisely described as a composite: the model's weights, plus its memory. In Reflexion's formulation, the policy is not stored in weights alone. It is parameterized jointly by the weights and by whatever resides in context. This is the key insight that makes the whole framework coherent: if the policy is the weights-plus-context, then you can modify the policy by modifying the context without touching the weights.

In a knowledge graph verifier, the Actor's first invocation has no memory. It answers questions about NovaAI using whatever it knows from pretraining, augmented by its system prompt's general instruction to be accurate and specific. It will produce a fluent answer that may or may not be grounded in the actual facts of the company. At this stage, it is V0 - the raw baseline.

After the first hallucination is caught and reflected upon, the Actor's subsequent invocations carry the accumulated lessons. The system prompt now contains something like:

You have previously made factual errors about NovaAI. Apply these lessons to improve your accuracy:

- The model tends to confuse pricing tiers when product names share tier suffixes. Always verify price amounts before stating them.

This is a modified policy. The same model, producing different behavior, not through weight change but through context change. The weights are a fixed substrate; the context is the mutable state space.

The Actor also produces, as a byproduct of its execution, what the paper calls a trajectory: the sequence of observations, thoughts, and actions taken during a single episode. This trajectory is short-term memory - the immediate working record of what happened during this particular attempt. It resets between episodes but persists within them, providing the raw material for evaluation and reflection.

The Evaluator: Credit Assignment Through Structure

The credit assignment problem is one of the oldest and most persistent challenges in reinforcement learning.

In a multi-step trajectory - a sequence of decisions leading to an outcome - which specific decisions deserve credit or blame for the final result? A simple scalar reward at the end provides a single number that must somehow distribute its signal backward through a sequence of potentially many decisions, most of which had nothing particular to do with why the outcome was good or bad.

Language agents face a specific variant of this problem. An LLM answer is not a single decision; it is an aggregation of claims, each of which may be independently true or false.

The answer "NovaAI was founded in 2021 by Dr. Mara Chen and Lucas Ferreira, and raised $210M in Series C funding in November 2024" contains multiple verifiable assertions. Some may be correct, some wrong. A binary "incorrect" verdict on the full answer tells the system nothing about which claim failed.

Reflexion addresses this through what amounts to claim-level credit assignment. The Evaluator decomposes the task of evaluation into atomic units: individual claims, each assessed independently against the knowledge source. The result is not a scalar but a structured verdict - a list of results, each specifying which claim is supported, which is contradicted, and which cannot be assessed given available evidence.

In a knowledge graph implementation, this evaluation pipeline is quite specific. An LLM answer is first decomposed into atomic claims - each claim asserting exactly one verifiable fact with an explicit subject, no pronouns, no aggregation.

"NovaAI was founded in 2021" is a proper claim. "It was founded in 2021" is not - the pronoun reference breaks the claim's self-containedness.

This atomicity is essential because the verification step requires mapping each claim to relevant KG triples via entity linking, and entity linking requires that the claim's subject be explicitly named.

Once claims are extracted, each is passed through an entity linker - a component that identifies the named entities and relation phrases in the claim, fuzzy-matches them against the KG's entity index, and retrieves the most relevant triples. The entity linking step itself is a critical architectural decision: LLM-based entity extraction produces better entity recall than pure fuzzy matching, but pure fuzzy matching is the correct fallback when the LLM extraction fails or returns no entity matches. The two-stage approach - LLM extraction with relation-score boosting, falling back to triple-view fuzzy scoring - balances recall against robustness.

The resulting evidence is then passed to the verification LLM, which returns a structured verdict:

• SUPPORTED,
• CONTRADICTED, or
• UNVERIFIABLE,

with a confidence score and a one-sentence reasoning trace.

Critically, if no relevant triples are found for a claim, the system returns UNVERIFIABLE without an LLM call at all - a short-circuit that avoids fabricating a verdict against empty evidence, and saves a token budget that would otherwise be spent on meaningless inference.

This structured output is credit assignment made explicit. Instead of "the answer was wrong" the Evaluator produces "this specific claim was wrong, here is the contradicting evidence, here is why." The failure has been localized, the responsible claim identified, the correct value surfaced. The gradient, if we want to use that metaphor, is pointing in a specific direction.

The Self-Reflection Model: From Error to Episodic Memory

Between the Evaluator's verdict and the Actor's next invocation lies the step that distinguishes Reflexion from mere error correction: the generation of verbal reflections.

The Self-Reflection model - typically the same LLM prompted differently - receives the full context of a failed episode: the original question, the wrong answer, the structured verdict identifying which claims were contradicted and what the correct values are, and the repaired answer. From this, it is asked to produce a generalized lesson.

The key word is generalized. The reflection prompt explicitly forbids generic advice ("be more careful") and demands category diagnosis: what class of information was hallucinated, why the model likely made that error, and what specific strategy would prevent recurrence. The output might read:

"The model confused the NovaPilot Growth and Enterprise pricing tiers - a common error when multiple product tiers share a naming pattern. The mistake likely arose from conflating the tier suffix ('Growth', 'Enterprise') with pricing magnitude heuristics rather than consulting explicit price triples. Strategy: when answering any question about product pricing, trace the specific product tier name to its exact price before generating a number."

This reflection performs semantic compression. It takes a particular failure about a particular product at a particular price point and extracts a behavioral principle applicable to any pricing question in the domain. The particular generates the general. This is exactly what learning should look like: not memorizing that "NovaPilot Growth costs $8,000/month" but internalizing "when answering pricing questions, verify tier-specific prices explicitly."

The reflection is then stored in episodic memory, which is the long-term component of the Reflexion memory architecture.

Part IV - Memory, Context, and the Execution Loop

Two Memory Systems

Reflexion implements a two-tier memory architecture that maps, with reasonable fidelity, onto how human memory is described in cognitive science.

Short-term memory - also called the trajectory buffer - holds the contents of the current episode. The question, the raw answer, the extracted claims, the verification verdicts, the repaired output, the latency. This is working memory: rich, detailed, and ephemeral. It is assembled anew with each question and discarded when the episode closes. It provides the immediate context for evaluation and reflection but does not persist to inform future episodes.

Long-term memory - the episodic store - holds the distilled reflections from prior failures. It persists across episodes, is bounded by a sliding window (typically three entries), and is injected into the Actor's system prompt at the start of each new episode. It is compressed, general, and durable.

The sliding window is a deliberate design constraint. Three reflections is not a magic number - it reflects a trade-off between memory coverage and context budget. Too few reflections and the agent forgets recent lessons; too many and the prompt grows bloated, the relevant lessons diluted by earlier ones that may no longer apply. The window always retains the most recent entries: as new lessons are added, the oldest are evicted. This recency bias is principled - errors made on earlier questions in a session are less likely to recur than the categories of error encountered most recently.

In our knowledge graph verifier, this memory is shared across an entire evaluation session of forty-five questions. A single `ReflexionMemory` instance persists throughout the run - not resetting between questions. This is the architectural decision that makes V2 a genuine learning system rather than a per-query corrector. Early questions generate reflections. Those reflections alter the system prompt for all subsequent questions. The agent that answers question thirty-eight is not the same agent that answered question one. Its weights are identical. Its context is not.

The Injection Pattern: Context as Policy State

The mechanics of memory injection reveal something important about how Reflexion achieves its effect. At the start of each episode, the Actor's system prompt is assembled conditionally: if the episodic memory contains reflections, a formatted memory block is appended to the base instructions.

The formatted block follows a strict template:

[PRIOR SESSION LEARNING - APPLY THESE LESSONS]
- Reflection 1
- Reflection 2
- Reflection 3
[END PRIOR SESSION LEARNING]

The boundary markers - `[PRIOR SESSION LEARNING]` and `[END PRIOR SESSION LEARNING]` - are not decorative. They signal to the model that this content is distinct from the task instructions: it is retrospective guidance, not prospective direction. The model is not being told what the answer is; it is being told what patterns of error to avoid. The distinction matters because it preserves the model's generative freedom while biasing it away from known failure modes.

This pattern is the operational core of context engineering. The prompt is not a static template that encodes the task. It is a dynamic state object that encodes the task, the constraints, the history, and the learned lessons from prior failures. As the session progresses and more reflections accumulate, the context evolves - and so does the effective policy.

When weights are frozen, context is the only tunable parameter. Reflexion is, among other things, an argument that this single degree of freedom is sufficient for meaningful learning in bounded domains.

The Execution Loop

The full execution loop, described abstractly in the paper and concretized in implementation, operates as follows.

An episode begins when the Actor receives a question. If the episodic memory is non-empty, the system prompt includes the accumulated reflections from prior failures. The Actor generates a raw answer conditioned on this prompt.

The raw answer is then decomposed into atomic claims by the Claim Extractor. Each claim is processed independently: entity linking identifies the relevant KG triples, and the Evaluator LLM judges each claim as SUPPORTED, CONTRADICTED, or UNVERIFIABLE. The Evaluator's output is a list of structured verdicts - one per claim.

If any claims are CONTRADICTED, the Repairer is invoked. It receives the original question, the raw answer, and the full verification results including the KG evidence for each contradicted claim. Its task is surgical: replace the wrong values with the correct ones from the KG evidence, leave supported claims untouched, soften unverifiable claims with epistemic hedges. The output is the final answer.

Whether or not repair occurred, if any CONTRADICTED verdicts were found, the Self-Reflection model is invoked. It receives the full failure context and generates a verbal lesson. That lesson is appended to the episodic memory, and the sliding window trims anything beyond the maximum retention count.

The next episode begins with this updated memory. The loop is now closed.

The loop runs for every question in the session. The V0 pipeline - raw LLM with no verification - has no loop: question in, answer out, nothing learned. The V1 pipeline - static KG verifier - adds verification and repair but still has no loop: the episode closes cleanly with no residue. V2 adds the reflection and memory update, and in doing so, adds the loop. The architecture that produces learning is precisely this closed loop between failure, reflection, and future context.

Part V - The Evidence and Its Limits

What the Numbers Say

Empirical validation is the necessary counterpart to architectural elegance. The Reflexion paper evaluates the framework across three task domains, each stress-testing a different dimension of agent capability.

In AlfWorld, a suite of text-based household environment tasks, Reflexion agents using the ReAct prompting strategy improved absolute performance by 22 percentage points over the baseline across 134 tasks in 12 iterative learning steps - completing 130 of 134 tasks total. The improvement arose through the agent's accumulated ability to diagnose two specific failure modes: hallucinated possession of objects (the agent acts as if it holds an item it hasn't retrieved) and inefficient planning (searching for items in a fixed order regardless of prior evidence). Reflections on these failures produced lessons about state tracking and search strategy that transferred across tasks.

In HotPotQA, a multi-hop question-answering benchmark, Reflexion improved accuracy by 20% over baseline. Crucially, the paper isolates the contribution of reflection from the contribution of episodic memory alone through ablation: adding only the most recent trajectory (episodic memory without reflection) produces an 8% improvement; adding the verbal reflection step on top of that produces the full 20%. Reflection contributes 12 percentage points independently of memory. This is the quantitative argument that verbal reflection is not merely a fancy way of logging history - it extracts something from failure that history alone does not provide.

In code generation on HumanEval, Reflexion achieved 91% pass@1 accuracy, surpassing GPT-4's 80% - at the time of publication, a state-of-the-art result. The mechanism is worth examining: the coding agent generates self-written unit tests alongside its implementation, runs them, and reflects on failures. The unit test suite acts as the Evaluator, providing grounded, executable feedback rather than LLM-generated judgment. The reflection then diagnoses why the implementation failed specific tests and proposes a corrected approach for the next attempt.

The ablation study on HumanEval Rust is particularly revealing. Four conditions are compared: base model alone (60% accuracy), test generation without self-reflection (60% - no improvement), self-reflection without test generation (52% - performance degrades), and the full Reflexion combination (68%). The counter-intuitive result - that self-reflection without test-driven grounding actively harms performance - is an important finding. Without reliable evaluation, the agent is reflecting on incorrect premises. It may conclude it failed for reasons it didn't fail, and generate lessons that steer it toward worse behavior. The quality of reflection is bounded by the quality of evaluation.

The Boundary Conditions

The WebShop experiment is the most instructive failure in the paper. WebShop is an e-commerce navigation task: the agent must search for products matching client specifications in a simulated online store. After four trials with no improvement, the experiment was terminated. The agent failed to generate useful reflections - its lessons were either too generic or misidentified the source of its failures.

The diagnosis is precise: Reflexion works when failures can be articulated as correctable behaviors. In WebShop, the failure modes were structural. The search space required diverse exploration strategies, not incremental correction of specific wrong moves. When the agent searched for "blue cotton shirt" and failed, the lesson wasn't "don't search for blue cotton shirts differently" - it was that the e-commerce search engine interpreted natural language queries non-deterministically. This is a failure the agent cannot reflect its way out of. The environment is irreducibly stochastic in a way that verbal lessons cannot address.

This points to the general boundary condition: Reflexion is effective when failure is diagnosable, correctable, and recurrent within recognizable categories. It is less effective - potentially harmful - when failures are random, structurally unavoidable, or require diverse exploration rather than careful correction.

A second boundary condition concerns model capability. The authors find that the ability to generate useful self-corrections is an emergent property of stronger, larger models. When the framework is applied to a weaker model (StarChat-beta in the appendix), performance does not improve at all - the reflection accuracy on HumanEval Python is statistically indistinguishable between baseline and Reflexion. The self-reflection step requires genuine reasoning capability to produce meaningful category diagnoses. A model that cannot accurately diagnose its own failures will generate lessons that are either trivially generic or actively misleading.

Finally, Reflexion does not escape the local minima problem inherent to all iterative optimization. If early reflections encode an incorrect causal theory of the failure - if the agent blames the wrong aspect of its behavior - subsequent attempts will optimize against the wrong target. There is no external error signal to correct a bad reflection; the system's self-assessment is authoritative.

Part VI - The Deeper Architecture

Verbal Feedback as Semantic Gradient

The most persistent insight in Reflexion, beyond any specific empirical result, is the framing of verbal feedback as a semantic analog to the gradient.

In gradient descent, the gradient is a vector in parameter space that points in the direction of steepest loss increase. The update rule moves parameters in the opposite direction - toward lower loss. The gradient carries directional information: not just "this was wrong" but "adjust these weights in these directions by these magnitudes to make it less wrong."

A scalar reward, by contrast, carries no directional information. It says "this was better or worse than expected" without specifying what to change or how. The gradient that backpropagates from a scalar reward to the parameters that produced it is computed through the model's computational graph - a process that requires differentiable parameter access and produces updates that may be hard to interpret.

A verbal reflection is a semantic gradient: it does not point in parameter space, but it points in behavior space. "When answering pricing questions, trace the specific product tier name to its exact price before generating a number" is directional information about behavior. It does not tell the model which weights to adjust; it tells the model which generation patterns to adopt. The directional information is encoded in natural language rather than real-valued vectors, and it is consumed by the model's attention mechanism rather than by a parameter update rule.

The insight that language models respond to linguistic direction as effectively as gradient descent responds to numerical gradients - within the bounded scope of in-context learning - is what makes Reflexion theoretically coherent rather than merely pragmatically useful. It is not a workaround for the unavailability of gradient access. It is an alternative signal modality that carries complementary information.

The Episodic Memory Lifecycle

In the full Reflexion framework, the lifecycle of a reflection follows a precise arc: from failure to diagnosis to storage to injection to effect.

Failure: the Actor produces an answer that is evaluated as containing contradictions.

Diagnosis: the Self-Reflection model produces a generalized lesson - the reflection - from the failure context. The lesson encodes what failed, why it likely failed, and what behavioral strategy would prevent recurrence.

Storage: the reflection is appended to the episodic memory. The sliding window trims entries beyond the retention limit, maintaining recency.

Injection: at the start of the next episode, the memory is formatted and appended to the Actor's system prompt, inside the session learning block.

Effect: the Actor's generation for the next episode is conditioned on this memory. The model's effective behavior is modified without any weight update.

The lifecycle is self-contained and modular. Each stage is independently observable and testable. If reflections are failing to prevent recurrent errors, the failure can be diagnosed at any stage: Are the reflections being generated with sufficient specificity? Is the injection format being consumed appropriately by the model? Is the category of failure amenable to verbal diagnosis at all? The modularity of the pipeline is not merely an engineering convenience - it is what makes the system debuggable.

What This Opens Up

Reflexion is, in the framework we're building here, the first answer to the central question. It is not the final one.

The framework it establishes - frozen weights, structured failure feedback, verbal reflection, episodic memory injection - is a specific architectural choice among several possible choices. The memory is a flat list. The reflections are free-form text. The sliding window is a simple recency heuristic. The evaluation is synchronous and claim-level. These are not the only ways to implement any of these components.

What happens when the memory is not a flat list but a graph? When reflections are not free text but structured templates with typed fields for error category, confidence, and scope? When evaluation is not post-hoc but interleaved within the generation process? When memory retention is not governed by recency but by estimated future relevance?

These are not rhetorical questions. They are active research directions, and each represents a distinct architectural approach to the same fundamental problem: how does a frozen-weight system improve its behavior through experience?

Reflexion establishes the hypothesis: it can. The specific mechanism - verbal reflection stored in episodic memory - is one answer. Subsequent work has proposed others, each with different characteristics, different strengths, and different failure modes.

The hallucination problem with which we began is not solved by Reflexion. It is meaningfully addressed. The residual gap - what Reflexion does not handle, where it fails, what capabilities it leaves unrealized - is precisely what the subsequent answers in this series will attempt to fill.

References

Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Neural Information Processing Systems (NeurIPS) 2023.

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations (ICLR) 2023.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., & Zhou, D. (2022). Chain of Thought Prompting Elicits Reasoning in LLMs. arXiv:2201.11903.

Sutton, R. S. & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.

Chen, M., Tworek, J., Jun, H. et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374.

Measurement Engineering: The Part of Data Science That Will Thrive in AI

What good judgment actually looks like in data science needs a new name

Am I measuring the right thing?

If you've been in data long enough, you know that question matters more than any model you've ever built. Yet most discussions center on "what's going on with this metric" before considering if we're measuring what we intend to.

We've spent 15 years building a field around execution. Learn Python. Learn SQL. Build a model. Ship a dashboard. Get good at Kaggle. The assumption was always that if you could execute, the judgment would come with experience. You'd just absorb it.

I don't think that was ever true. And now, with AI taking over more of the execution layer every month, the gap between people who can do the work and people who can tell you whether the work was worth doing is becoming the defining split in our field.

I've started thinking about the latter group as measurement engineers. Not because they need a new title, but because the data science title is too broad and ambiguous, and they need to be recognized for what they actually do, which is fundamentally different from what most data science job descriptions ask for.

The split we're already feeling

You know this in your gut even if you haven't named it yet. There are two kinds of questions in data work:

Execution questions (AI can do these, and it's getting better fast):

• Write the query
• Build the pipeline
• Train the model
• Generate the chart

Judgment questions (AI cannot do these, at least not well):

• Are we measuring the right thing?
• Does the metric actually capture what we think it captures?
• Should we trust this A/B test, or is something confounded?
• The data says X, the users say Y. Who's right?
• We have 300 metrics. Which 4 actually matter?

We interview for the first list, but the second list generates most of the impact. We get. That disconnect is the problem.

What good measurement engineering looks like

Here are three situations you've probably lived through, or will soon.

The 100+ metric problem

Your team runs an experiment. It's measured against hundreds of metrics. Some go up, some go down, some don't move. The PM cherry-picks the three that support what they already wanted to do. Another PM on a different team runs a different experiment against a different subset of the same metrics and reaches the opposite conclusion. Both teams call themselves "data-driven."

The execution was fine. The queries were correct. The dashboards were accurate. The problem is that nobody did the harder work: deciding which metrics actually predict the outcomes the business cares about and having the courage to retire the rest.

That's judgment. The person who solves this doesn't write a query. They sit in a room with product leaders and make uncomfortable decisions about which numbers matter and which ones are noise dressed up as signal. They kill metrics that teams have been watching for years. It might be the single highest-leverage thing a measurement engineer does.

The eval that lied

An AI team builds evaluations for their model. The eval scores say quality is improving every week. Charts go up and to the right. Everyone feels good. They ship.

Users hate it. Support tickets spike. Satisfaction drops. The model is measurably "better" on every internal eval and measurably worse in the real world.

What happened? The evals were precise and reproducible, but they measured the wrong thing. They tested whether the output was fluent. Users cared whether the output was useful. Those are correlated, but they're not the same construct. The eval suite was systematically missing the dimension that mattered.

A data scientist would look at the scores and say, "The model improved." A measurement engineer would ask: When was the last time we validated that these scores predict what users actually experience? The answer, in most organizations, is never.

The fix isn't a better model or a better pipeline. It's redesigning what you measure. And that requires a skill most of us were never taught: understanding the difference between "the thing we're measuring" and "the thing we think we're measuring." Those are not the same. The gap between them is where organizations make their most expensive mistakes.

The ambiguous result

Your team runs an A/B test on a new feature. The primary metric is up 3%, statistically significant. The guardrail metric, long-term retention, is down 1.5%. Not significant, but trending wrong. The PM wants to ship. Everyone looks at you.

What do you do?

A data scientist can tell you what the numbers are. A measurement engineer can tell you what the numbers mean. And sometimes what they mean is: we don't have enough evidence to decide, and pretending we do is more dangerous than waiting.

That willingness to say "the data is ambiguous and here's what I'd do about it" rather than just presenting a p-value is the skill that separates measurement engineers from analysts. It requires understanding power analysis well enough to know when your test couldn't detect the effect that matters. It requires understanding the business well enough to know whether a 1.5% retention drop compounds into a crisis over six months. And it requires the confidence to walk into a room full of people who want a green light and say, "I'd wait."

Judgment is not intuition

I don't believe judgment is just "experience" or "gut feel." That framing lets us off the hook for not teaching it deliberately. You can learn judgment. It comes from specific disciplines that the data science curriculum almost entirely ignores because they're from fields we don't talk to enough. Some examples:

Construct validity comes from psychometrics. Before you measure something, you ask: Does this measurement actually capture the thing I care about? If you're measuring "engagement" with time-on-page, are you measuring interest or confusion? Most data scientists have never been taught to ask this question. Most ML eval suites have never been subjected to it.

Measurement reliability is the difference between a thermometer that gives you a different reading every time and one that reads 72 degrees regardless of the actual temperature. Consistent and wrong. Most ML eval suites have this problem. They're reproducible, and they're reproducibly measuring something that doesn't matter.

Decision theory under ambiguity is what happens after the number comes back, and it's unclear. When two metrics disagree. When the confidence interval is wide. When the cost of being wrong in one direction is 10x the cost of being wrong in the other. Almost all data training teaches you to produce the number. Almost none teaches you what to do when the number doesn't give you a clean answer.

Why this matters right now

Three things changed, and they all point in the same direction.

First, AI is handling the execution. If a language model can write the query, build the pipeline, and generate the chart, what's left for us? The part the model can't do: deciding whether the query answered the right question, whether the pipeline measured the right thing, and whether the chart tells a true story or a convenient one.

Second, AI itself needs to be evaluated. And AI evaluation is the hardest measurement problem most organizations have ever faced. Traditional software either works or it doesn't. AI outputs are non-deterministic, context-dependent, and subjective. "Is this response good?" is not an engineering question. It's a measurement question. And most teams are improvising because nobody on the team was trained in the science of evaluation.

Third, the cost of bad measurement is scaling. An incorrect number on a dashboard leads to a poor quarterly decision. An incorrect eval score on an AI system can lead to a model that hallucinates in production, even when the metrics say everything is fine. The consequences of getting measurement wrong are outpacing our ability to get it right.

What I'd change

In hiring, stop testing for SQL speed and model-building. Start testing for judgment. Give candidates an ambiguous dataset and a question with no clean answer. The best candidates will tell you what the data can and cannot support. The rest will just give you a number.

In training, every data science program should require measurement theory. Construct validity. Inter-rater reliability. Item Response Theory. Not as an elective. As a core requirement. If you're going to build systems that measure things, you should know the science of measurement.

In org design, build a measurement function that owns the question "is this working?" across analytics, data science, and ML. Give it the authority to kill bad metrics and flag bad evals. Most data teams have the skills distributed across individuals. Few have the organizational structure that gives those individuals the mandate to actually use them.

Where this leaves you

If you're reading this and recognize yourself as someone who already does this work, the thing I want you to hear is this: the skill you have is about to become the most valuable on any data team. The people who can judge whether a measurement is valid, whether an eval is predictive, whether a result is trustworthy, those people were always important. In a world where AI handles the execution, they become essential.

And if you're reading this and realizing you've been building your identity around the execution layer, the SQL, the models, and the pipelines, just know it's not enough anymore. The people who make the shift toward judgment, toward measurement, toward owning the question "is this actually working," they're the ones who will define what data teams look like for the next decade.

The transition is uncomfortable. A lot of what made us feel technical and respected is moving to the execution layer that AI handles. But the part that stays human is the part that always mattered most.

Data systems evolved to decouple storage and compute, making it cheaper and easier to scale.

Security patches and provider updates stopped last week.

A US federal court granted iyO a preliminary injunction blocking OpenAI and Jony Ive's venture from using the “io” name, siding with iyO in a trademark dispute. The case began after OpenAI and Sam Altman announced the “io” AI hardware brand, prompting iyO to sue over infringement, consumer confusion, and later trade secret theft. Judge Trina Thompson ruled iyO is likely to win and could suffer irreparable harm, rejecting OpenAI's claim that dropping the name makes the case moot since the injunction prevents any future use.

This Adobe Sneak Uses AI to Rethink the Personalized Web

Cookies have long been the default tool brands reach for when personalizing the web. For over 30 years, small text files stored in browsers have given companies a window into how users move across the web—and a way to tailor ads and content accordingly. But privacy regulations such as GDPR and CCPA, restrictions in browsers like Firefox and Safari, and shifting consumer expectations have gradually eroded their effectiveness, pushing brands to search for alternatives.

What emerged to fill that gap, though, has its own set of tradeoffs. First-party data, contextual targeting, and login-based personalization all attempt to tackle this problem, but they share the same flaw—they personalize for segments, not the individual.

However, a new Sneak from Adobe called Project Page Turner is taking a different approach—and it's powered by artificial intelligence. Adobe previewed it exclusively with The AI Economy ahead of its public unveiling this week.

Meet Project Page Turner

What if you could use a large language model to transform your website so it feels tailor-made for every visitor? It's what Adobe is exploring with Project Page Turner, a working idea that helps brands provide a more one-to-one experience.

Paolo Mottadelli, the Adobe Experience Manager (AEM) engineering director and the creator of this project, says the timing is right—inference speeds have improved to the point where personalized pages can be generated in real-time. Models are now fast enough to deliver a first response in under 100 milliseconds, quicker than you can blink, and with the full page loading in under a second—well below the two-second threshold most users expect.

He points out that websites haven't really changed: "Everybody goes to the same page, and then there is the marketer-driven personalization that requires effort, and it's limited." What Project Page Turner does is empower the page to "imagine itself based on what the site knows about the visitor." In other words, Mottadelli imagines a future in which websites are no longer "static" or even exist anymore.

"They will create themselves under the browsing experience of the user," he says.

Conceived From Customer Feedback

The idea behind Project Page Turner didn't appear out of nowhere. Mottadelli reveals it originated from customer feedback—in working with more than 90 AEM customers in the past year, he was repeatedly asked for ways to provide better, more meaningful personalization. He recalls working on this project back "when technology to do it was not around yet." Now, it is.

As the company behind one of the web's leading content management platforms, Adobe has a direct stake in how websites evolve. Mottadelli says that after seeing Google provide dynamic visual answers that took three minutes to generate, his team began asking what would happen if that same experience could be delivered "in a page delivery timeframe, which happens to be under two seconds."

After some successful trialing, Mottadelli believes "this might be one of the directions the world [wide] web is going."

Unsurprisingly, Project Page Turner sits on top of AEM, utilizing it in two ways: AEM Assets—where brands already store images, video, and content—is the content library this system draws from; and AEM Sites—Adobe's content management system—provides the layout elements. Project Page Turner applies a new indexing layer that makes both rapidly accessible to a fast LLM, allowing it to assemble a personalized page on the fly.

Mottadelli notes that Project Page Turner's inference and underlying models are currently provided by Cerebras, primarily using GPT—though any fast model can be swapped in—reflecting Adobe's broader model-agnostic approach, as evidenced by its Firefly platform.

When implemented, brands need to provide the LLM with brand guidelines, product knowledge, and content rules. All of this goes into instructing it on what to recommend and when, similar to how you'd brief a new employee on company policy. Mottadelli predicts that, eventually, "training the website is like training a human."

How Project Page Turner May Work

One possible way to implement Project Page Turner could be through a website's search feature. Say you're a personal trainer looking for a new blender that can make multiple protein shakes for your clients. Instead of visiting Amazon or Google, you go directly to a website like Vitamix to do your research. There, you start by entering your query in the search feature.

Traditionally, information is surfaced by keyword or by a referenced product name. However, Project Page Turner analyzes the natural language query and returns a fully customized results page just for you. So, you can specify that you're curious about how different models compare in terms of heavy daily use and cleanup, and that speed matters a lot.

What gets generated isn't a generic results page where you have to click through to find piecemeal information. Rather, you're provided a fully assembled experience built around your specific request. Information about high-volume blenders, self-cleaning instructions, thick protein shake recipes, model comparisons, and purchase options all surface together, organized around what you actually asked for.

"The prompt is basically zero-party data…it doesn't require cookies," Eric Matisoff, Adobe's principal evangelist, tells The AI Economy. "It wouldn't require… second-party or third-party data to that user in order to personalize [the search results]."

He adds, "The most interesting thing to analyze across an entire website is internal search data, because there's no other time that someone is telling you exactly what they're looking for than when they're doing an internal search on your website. This takes that to the next step, while also building the personalized experience on the fly."

Beyond the explicit intent of the search prompt, Project Page Turner can also learn from how you browse—observing which products you view, which categories you explore, and what content you engage with. Mottadelli adds that if you arrive at a supported site from ChatGPT, Google, or another platform, the system could carry that intent over as well. But regardless of the signals available, he emphasizes that consent is central to the vision.

Mottadelli says that his ambitions for Project Page Turner extend far beyond the search bar. "What we're really envisioning is the seamless browsing experience that the website is…becoming your second skin without you even noticing." He imagines sites adapting so closely to us that it feels natural and invisible, easily picking up intent and signals and quietly reshaping content and layout around us in real time. How this idea is realized isn't a technology question but rather a question of how companies want users to experience their websites.

ChatGPT, But With the Brand Experience

People approach traditional search engines and AI chatbots like Perplexity, OpenAI's ChatGPT, Google's Gemini, or Anthropic's Claude very differently. The former is about discovery—you're browsing and exploring without a clear destination—while the latter is about intent; you have a specific goal in mind and expect a tailored answer.

Brands want to capitalize on intent signals without sending visitors to a third-party platform. Yes, ChatGPT answers your question, but it strips away the brand experience—the visuals, the product narrative, the path to purchase. A brand website has all of that, but can't respond to your specific intent in real time. Project Page Turner is Adobe's attempt to sit in between: a brand website that behaves like a chatbot.

Companies can already extend their brands to platforms like ChatGPT. Booking.com, Canva, Coursera, Expedia, Figma, Spotify, Zillow, Upwork, and even Adobe have integrated their services with the popular chatbot. But that means operating on someone else's turf and surrendering control of the brand experience. Project Page Turner flips that dynamic, keeping AI's intent-driven responsiveness on the brand's own website.

"This is putting two things together: immediate reaction to the intent with breadthful brand experience," Mottadelli says.

Whether Adobe can defend Project Page Turner against the likes of OpenAI, Google, Anthropic, Salesforce, and others—some of which are moving aggressively to disrupt established players—remains to be seen. But Mottadelli argues that Adobe has something they mostly do not: 4,000 enterprise customers, decades of content management infrastructure, and direct relationships with the marketing teams responsible for brand experiences. "AEM has the right technology to implement [this solution]," he says. The advantage isn't the technology, Mottadelli points out—it's the trust.

What Are Adobe Sneaks?

Project Page Turner is one of seven so-called Sneaks being unveiled this week at Adobe's AI Summit. None are committed to the product roadmap, though Matisoff notes that historically, 30 to 40 percent have made it into production in some form. That said, the goal of Adobe Sneaks is to highlight innovation.

Matisoff says that AI has contributed to an increase in ideas, all jockeying to be featured on stage: "Last year, we had a little over 150 [submissions]. This year, we had well over 500." He adds that even after identifying the projects that will be featured at the event, "we've seen the ideas continue to advance at a more rapid pace…because of how easy it is for our teams to ideate and expand and improve what they're delivering."

Adobe Sneaks isn't unique to Summit—the showcase also happens at Adobe Max. Think of it as Silicon Valley demo day with a comedic twist: engineers present their ideas on stage before a live audience of thousands, Matisoff, and a celebrity co-host. This year, that's comedian, actress, and producer Iliza Shlesinger.

And for those who want a say in what gets built, new this year is a concept that lets audience members vote for their favorite Sneak. The results will help influence the product roadmap. Now, everyone can voice their support in hopes that their favorite idea becomes reality.

For Adobe, the stakes around Sneaks extend beyond any single idea. The company faces mounting pressure—not just from longtime rivals like Figma and Canva, but also from AI-native challengers like Anthropic rewriting the rules of the software industry. Matisoff sees this showcase as proof that Adobe hasn't lost its founder mentality—that innovation still bubbles up from anywhere in the organization. He points to a line from one of Adobe's founders that still guides the program: "Great ideas can come from anywhere in the company." In 2026, Adobe needs that to be true more than ever.

SpaceX said it has struck a deal with Cursor to develop a next-generation "coding and knowledge work AI," which includes a surprising provision — an option to buy the popular software development platform for $60 billion later this year.

Partnering with and potentially purchasing a leader in the hottest AI product category can only be seen in the context of SpaceX's much-anticipated public offering. Investors seeking more value in the IPO might see its engagement with Cursor as another way to extract value from Elon Musk's increasingly sprawling tech conglomerate.

The deal won't shock those who follow the industry closely. Last week, it was reported that xAI would begin renting computing power from its data centers to Cursor, with the coding startup using tens of thousands of xAI chips to train its latest AI model. And last month, two of Cursor's most senior engineering leaders, Andrew Milich and Jason Ginsberg, left the company to join xAI, where both report directly to Musk.

SpaceX described the partnership as a project combining Cursor's "product and distribution to expert software engineers" with SpaceX's Colossus supercomputer, which the company claims has the equivalent compute power of a million Nvidia H100 chips.

SpaceX also said that at some undisclosed point later this year, it will either pay Cursor $10 billion for its work or acquire the company for $60 billion. Last week, TechCrunch reported that Cursor was eyeing a $50 billion valuation in an upcoming private fundraising round. That figure itself reflects an astonishing series of leaps. Cursor was valued at just $2.5 billion in January of last year, climbed to $9 billion by last May, and was assigned a $29.3 billion post-money valuation when it closed on $2.3 billion in Series D funding in November.

Either figure would represent a significant expense for SpaceX, which is widely seen to be losing money following the acquisition of xAI and the social media network X and is planning extensive capital investment. The brief statement did not say if either deal could be paid in SpaceX stock.

In the meantime, the move could shore up weaknesses at each company, but it also reveals them. Neither Cursor nor xAI has proprietary models that can match the leading offerings from Anthropic and OpenAI — the same companies now competing directly with Cursor for the developer market.

Cursor still uses and sells access to Claude and GPT models even as both firms roll out their own coding tools, an awkward arrangement that this new SpaceX partnership may be designed to eventually escape.

In a rush to embrace AI, the industry is redefining what it means to be a UX designer, blurring the line between design and engineering. Carrie Webster explores what's gained, what's lost, and why designers need to remain the guardians of the user experience.

In early 2026, I noticed that the UX designer's toolkit seemed to shift overnight. The industry standard "Should designers code?" debate was abruptly settled by the market, not through a consensus of our craft, but through the brute force of job requirements. If you browse LinkedIn today, you'll notice a stark change: UX roles increasingly demand AI-augmented development, technical orchestration, and production-ready prototyping.

For many, including myself, this is the ultimate design job nightmare. We are being asked to deliver both the "vibe" and the "code" simultaneously, using AI agents to bridge a technical gap that previously took years of computer science knowledge and coding experience to cross. But as the industry rushes to meet these new expectations, they are discovering that AI-generated functional code is not always good code.

The LinkedIn Pressure Cooker: Role Creep In 2026

The job market is sending a clear signal. While traditional graphic design roles are expected to grow by only 3% through 2034, UX, UI, and Product Design roles are projected to grow by 16% over the same period.

However, this growth is increasingly tied to the rise of AI product development, where "design skills" have recently become the #1 most in-demand capability, even ahead of coding and cloud infrastructure. Companies building these platforms are no longer just looking for visual designers; they need professionals who can "translate technical capability into human-centered experiences."

This creates a high-stakes environment for the UX designer. We are no longer just responsible for the interface; we are expected to understand the technical logic well enough to ensure that complex AI capabilities feel intuitive, safe, and useful for the human on the other side of the screen. Designers are being pushed toward a "design engineer" model, where we must bridge the gap between abstract AI logic and user-facing code.

A recent survey found that 73% of designers now view AI as a primary collaborator rather than just a tool. However, this "collaboration" often looks like "role creep." Recruiters are often not just looking for someone who understands user empathy and information architecture — they want someone who can also prompt a React component into existence and push it to a repository!

This shift has created a competency gap.

As an experienced senior designer who has spent decades mastering the nuances of cognitive load, accessibility standards, and ethnographic research, I am suddenly finding myself being judged on my ability to debug a CSS Flexbox issue or manage a Git branch.

The nightmare isn't the technology itself. It's the reallocation of value.

Businesses are beginning to value the speed of output over the quality of the experience, fundamentally changing what it means to be a "successful" designer in 2026.

The Competence Trap: Two Job Skill Sets, One Average Result

There is potentially a very dangerous myth circulating in boardrooms that AI makes a designer "equal" to an engineer. This narrative suggests that because an LLM can generate a functional JavaScript event handler, the person prompting it doesn't need to understand the underlying logic. In reality, attempting to master two disparate, deep fields simultaneously will most likely lead to being averagely competent at both.

The "Averagely Competent" Dilemma

For a senior UX designer to become a senior-level coder is like asking a master chef to also be a master plumber because "they both work in the kitchen." You might get the water running, but you won't know why the pipes are rattling.

• The "cognitive offloading" risk. Research shows that while AI can speed up task completion, it often leads to a significant decrease in conceptual mastery. In a controlled study, participants using AI assistance scored 17% lower on comprehension tests than those who coded by hand.
• The debugging gap. The largest performance gap between AI-reliant users and hand-coders is in debugging. When a designer uses AI to write code they don't fully understand, they don't have the ability to identify when and why it fails.

So, if a designer ships an AI-generated component that breaks during a high-traffic event and cannot manually trace the logic, they are no longer an expert. They are now a liability.

The High Cost Of Unoptimised Code

Any experienced code engineer will tell you that creating code with AI without the right prompt leads to a lot of rework. Because most designers lack the technical foundation to audit the code the AI gives them, they are inadvertently shipping massive amounts of "Quality Debt".

Common Issues In Designer-Generated AI Code

• The security flaw: Recent reports indicate that up to 92% of AI-generated codebases contain at least one critical vulnerability. A designer might see a functioning login form, unaware that it has an 86% failure rate in XSS defense, which are the security measures aimed at preventing attackers from injecting malicious scripts into trusted websites.
• The accessibility illusion: AI often generates "functional" applications that lack semantic integrity. A designer might prompt a "beautiful and functional toggle switch," but the AI may provide a non-semantic <div> that lacks keyboard focus and screen-reader compatibility, creating Accessibility Debt that is expensive to fix later.
• The performance penalty: AI-generated code tends to be verbose. AI is linked to 4x more code duplication than human-written code. This verbosity slows down page loads, creates massive CSS files, and negatively impacts SEO. To a business, the task looks "done." To a user with a slow connection or a screen reader, the site is a nightmare.

Creating More Work, Not Less

The promise of AI was that designers could ship features without bothering the engineers. The reality has been the birth of a "Rework Tax" that is draining engineering resources across the industry.

• Cleaning up: Organisations are finding that while velocity increases, incidents per Pull Request are also rising by 23.5%. Some engineering teams now spend a significant portion of their week cleaning up "AI slop" delivered by design teams who skipped a rigorous review process.
• The communication gap: Only 69% of designers feel AI improves the quality of their work, compared to 82% of developers. This gap exists because "code that compiles" is not the same as "code that is maintainable."

When a designer hands off AI-generated code that ignores a company's internal naming conventions or management patterns, they aren't helping the engineer; they are creating a puzzle that someone else has to solve later.

The Solution

We need to move away from the nightmare of the "Solo Full-Stack Designer" and toward a model of designer/coder collaboration.

The ideal reality:

• The Partnership: Instead of designers trying to be mediocre coders, they should work in a human-AI-human loop. A senior UX designer should work with an engineer to use AI; the designer creates prompts for intent, accessibility, and user flow, while the engineer creates prompts for architecture and performance.
• Design systems as guardrails: To prevent accessibility debt from spreading at scale, accessible components must be the default in your design system. AI should be used to feed these tokens into your UI, ensuring that even generated code stays within the "source of truth."

Beyond The Prompt

The industry is currently in a state of "AI Infatuation," but the pendulum will eventually swing back toward quality.

The UX designer's nightmare ends when we stop trying to compete with AI tools at what they do best (generating syntax) and keep our focus on what they cannot do (understanding human complexity).

Businesses that prioritise "designer-shipped code" without engineering oversight will eventually face a reckoning of technical debt, security breaches, and accessibility lawsuits. The designers who thrive in 2026 and beyond will be those who refuse to be "prompt operators" and instead position themselves as the guardians of the user experience. This is the perfect outcome for experienced designers and for the industry.

Our value has always been our ability to advocate for the human on the other side of the screen. We must use AI to augment our design thinking, allowing us to test more ideas and iterate faster, but we must never let it replace the specialised engineering expertise that ensures our designs technically work for everyone.

Summary Checklist for UX Designers

• Work Together. Use AI-made code as a starting point to talk with your developers. Don't use it as a shortcut to avoid working with them. Ask them to help you with prompts for code creation for the best outcomes.
• Understand the "Why". Never submit code you don't understand. If you can't explain how the AI-generated logic works, don't include it in your work.
• Build for Everyone. Good design is more than just looks. Use AI to check if your code works for people using screen readers or keyboards, not just to make things look pretty.

Heavy reliance on AI tools is pushing designers to outsource critical thinking, favoring speed and rapid output over deeper evaluation and idea development. To counter this, an AI collaborator (Thia) was created to engage in real-time, critical dialogue during sketching—supporting, rather than replacing, human thinking. The key takeaway is that while AI accelerates execution, strong design still depends on deliberate reasoning and “cognitive friction,” which must be preserved to avoid building fast but flawed solutions.

Working in open source—especially on large-scale projects like ODK—pushes designers beyond simply shipping features toward designing transparently, collaborating closely with users, and thinking long-term about maintenance and impact. Open collaboration, public roadmaps, and constant community feedback improve decision-making while forcing clarity, accountability, and trust. A key lesson is embracing the “un/learning loop”: slowing down, staying open to uncertainty, and learning in public leads to better, more inclusive solutions—especially in complex or high-stakes environments—while prioritizing stewardship over speed.

AIMAC: The AI Model Accessibility Checker

AI is writing more code than ever. But is it accessible to People with Disabilities?

We prompted the top AI models to build web pages across 28 categories and audited them for accessibility. We published every generated page, side by side, so you can see how different models tackled the same design challenge. We even measured emdash usage.

AIMAC is an initiative by the GAAD Foundation, in partnership with ServiceNow. Updated Apr 08, 2026.

Leaderboard

Legend: 🅿️ Pareto Optimal | 🏆 Lowest Debt

1 #1 and #2 and #3 tied with an AIMAC Debt of 0.00. Tiebreaker: #1 averaged fewer violations (0.94 vs 1.02 vs 1.31).

* GPT 5.3 Codex shows a median AIMAC Debt of 0.00. This means at least half of the 28 categories had zero accessibility violations, but some categories still had minor issues (20 total violations across all categories).

Analysis

Introduction

95.9% of the top million websites fail basic accessibility checks. WebAIM has tracked it for seven years. After six years of marginal improvement, 2026 reversed the trend: errors per page jumped 10% to 56.1 and the failure rate climbed back to 95.9%.

AI is writing more of the world's code every day. Vibe Coding was the Collins Dictionary Word of the Year. If AI keeps writing code as poorly as the developers it learned from, nothing changes. But if it prioritizes accessibility, the web gets its first real chance to improve.

Our one ambitious goal

Ensure that AI models write accessible code by default.

Which Model is Best?

GPT 5.3 Codex by OpenAI holds the top spot with a median AIMAC Debt of 0.00, generating just 20 accessibility violations across all 28 categories. GPT 5.4 Mini joins it at #2, also achieving zero debt for just $0.89. GPT 5.4 takes #3 (also 0.00) and GPT 5.4 Pro sits at #4 (1.00). Arcee AI's Trinity Large Preview (free) rounds out the top five at #5 with a debt of 3.98. OpenAI holds four of the top five spots.

Three models now achieve a median AIMAC Debt of 0.00. The cheapest is GPT 5.4 Mini at $0.89. The most expensive model on the benchmark is GPT 5.4 Pro at $212.75.

The Pareto Frontier

AIMAC Debt vs Cost

Choosing a model isn't simply about which model is most accessible. Some models are very expensive. Benchmarks commonly use Pareto Frontier analysis to compare models on quality vs cost dimensions. Pareto optimal models (teal diamonds) are the efficient picks: to lower the AIMAC Debt grade, you'd pay more; to pay less, your AIMAC Debt grade rises. A gold ring marks the lowest AIMAC Debt.

Top 3 Winners

1. OpenAI dominates the top of the leaderboard. Three of their models achieve a median AIMAC Debt of 0.00: GPT 5.3 Codex (#1), GPT 5.4 Mini (#2), and GPT 5.4 (#3). A median of 0.00 means at least half of the 28 categories had zero violations, though a few categories still had minor issues. GPT 5.4 Pro rounds out the top four at 1.00. They hold four of the top five spots (#1, #2, #3, #4), and their open-source gpt-oss-120b lands at #11 for just $0.10.

2. Arcee AI is a 30-person San Francisco startup that built Trinity Large Preview from scratch for $20 million. It is a 400-billion-parameter sparse model with only 13 billion active per token, Apache 2.0 licensed, and ranks #5 for free. It is the largest permissively-licensed open model from a US company. Their Trinity Large Thinking variant lands at #29 (debt 7.97, $0.25).

3. Alibaba/Qwen fields the deepest roster outside OpenAI, with eight models spanning the leaderboard. Qwen3 Max leads the pack at #7 (debt 4.14, $1.07), and Qwen3 Coder Plus follows at #8 (debt 4.36, $0.75). Their lineup covers proprietary flagships, code-tuned variants, and open-weight releases, giving developers more accessibility-tested options than any other non-US provider.

Google's Comeback

Google's Gemini 3 Pro Preview once finished dead last at #39 with an AIMAC Debt of 10.65. We were disappointed because Google has the tools and talent to do better.

Its replacement, Gemini 3.1 Pro Preview, now sits at #9 with an AIMAC Debt of 4.40. Google is no longer at the bottom. This is exactly the kind of progress we hoped this benchmark would encourage.

What About Claude?

Anthropic has real developer mindshare, and Claude is often the default choice when teams want strong coding help. In this benchmark, their best result is Claude Haiku 4.5 at rank #10 (4.47 for $2.14). Claude Opus 4.6 (Fast) ranks #23 (5.94 for $112.18), Claude Opus 4.6 ranks #26 (6.90 for $18.34), and Claude Sonnet 4.6 sits at #36 (9.83 for $12.64). Sonnet's regression from 4.5 remains sharp: from #19 (debt 4.78) to #36 (debt 9.83).

Other major providers improved their accessibility results. Anthropic is the only one whose models got worse.

Sonnet 4.6 generates 1,186 accessibility violations, nearly three times the field average of 403. Opus 4.6 produces 788, nearly double. The same gap shows in Anthropic's frontend-design skill, where accessibility is barely mentioned while the guidance is overwhelmingly visual.

Anthropic is a Public Benefit Corporation whose stated values include acting "for the global good" and being good to their users, whom they define broadly as "anyone impacted by the technology we build." People with disabilities are impacted every time Claude generates inaccessible code. We hope Anthropic will bring the same energy they apply to AI safety to the accessibility of their models' output.

The broader trend is encouraging. Models are training against automated tools like axe-core, and scores are improving. We are working on new benchmarks that test models that go beyond axe-code testing.

Beyond the Rankings

As AI models get better at visual design, they face a tradeoff between optimizing for beauty or for accessibility. You can achieve both, but it requires deliberate effort.

If you're picking a model for your workflow, start with a category page that matches your use case. Each category shows how ~40 models handled the same prompt, alongside their AIMAC Debt. For example, visit the Sports category in grid mode to compare sports designs side by side.

If you already have a model in mind, use the model detail page to see what its pages look like across all 28 categories. Then drill down into the categories that match your needs. See GPT 5.4 Pro in grid mode for an example.

Value vs Premium

The #1 model on this benchmark costs $3.02. But the #2 model costs just $0.89. GPT 5.4 Mini achieves zero debt for under a dollar, while GPT 5.4 Pro costs $212.75 for a debt of 1.00. Twenty-eight of 43 models cost under $1, including Grok 4.1 Fast at #14 for 8 cents, OpenAI's gpt-oss-120b at #11 for 10 cents, and Gemma 4 31B at #40 for 8 cents.

Closed vs Open (Source/Weights)

A closed model holds the top spot. GPT 5.3 Codex (proprietary) remains #1, and three more closed OpenAI models fill spots #2 through #4. Trinity Large Preview (open-source, Apache 2.0) holds #5 for free.

The upper-middle tier remains packed with value picks across both closed and open-weight releases. Qwen3 Max and Coder Plus rank 7th and 8th. Kimi K2 Thinking sits at #12 for 82 cents. OpenAI's open-weight gpt-oss-120b ranks 11th for 10 cents.

The smaller models are mostly from China. Qwen, Z.ai, DeepSeek, and MoonshotAI account for 15 of 43 models. Alibaba's Qwen line has eight entries. Despite US chip restrictions, Chinese labs keep shipping.

DeepSeek made headlines in January 2025 when their R1 model triggered a $600 billion single-day loss in Nvidia's market cap by claiming similar performance to Western models at a fraction of the cost. But on accessibility, their lineup sits near the bottom: R1 0528 scores 10.17 (rank #37), V3.2 lands at 8.18, and their best result (V3.2 Speciale) still comes in at 7.21. R1 0528 regressed from the original R1 it replaced (which scored 8.08). The efficiency breakthrough has not translated to accessible output.

Mistral struggled on this benchmark. Medium 3.1 is their lone mid-pack result at 4.70 (rank #15), while the rest cluster near the bottom: Large 3 2512 at 8.33 (rank #32), Codestral 2508 at 8.40 (rank #33), and Devstral 2 2512 at 9.01 (rank #34). Their newest release, Mistral Small 4, finishes dead last at #43 with a debt of 16.96, a sharp regression from the Small 3.2 24B it replaced (which scored 8.48). Only Medium 3.1 cracked the Top 20, and it is not open source. Their open-source Small 4 finishes last.

What Trips Models Up

Low contrast text dominates both AI-generated and human-built websites. Both columns report the share of pages with at least one instance of each issue. AIMAC uses axe-core across AI-generated pages; WebAIM uses WAVE across the top 1,000,000 home pages.

AIMAC: Top 6 issues

Share of pages with >= 1 error

• Low contrast text: 84.8%
• Empty links: 31.0%
• Missing form labels: 19.7%
• Empty buttons: 6.9%
• Links distinguished only by color: 3.9%
• Target size too small: 3.7%

WebAIM Million 2026: Top 6 issues

Share of home pages with >= 1 error

• Low contrast text: 84%
• Missing alt text: 53%
• Missing form labels: 51%
• Empty links: 46%
• Empty buttons: 31%
• Missing document language: 14%

Emdash Benchmark

We tracked emdash usage as a small writing-style signal, because punctuation can affect how text is read aloud.

Screen readers handle emdashes differently. Some announce "em dash" at every occurrence, others treat it as a pause, and some ignore it depending on voice and settings. Ricky Onsman explored the issue and found this is largely a screen reader behavior difference, not a content authoring bug.

We sanity-checked this with screen reader users, and none of our friends reported emdash-heavy text as a practical problem in day-to-day reading. So this turned out to be more interesting than impactful, and it does not affect AIMAC rankings.

Emdash usage varies widely. Across 43 models, counts range from 0 (Codestral 2508) to 754 (Claude Sonnet 4.6). 1 model uses zero emdashes.

Models Tested

This analysis covers 43 models from 15 companies:

1. Claude Haiku 4.5
2. Claude Opus 4.6
3. Claude Opus 4.6 (Fast)
4. Claude Sonnet 4.6
5. Codestral 2508
6. DeepSeek V3.2
7. DeepSeek V3.2 Speciale
8. Devstral 2 2512
9. Gemini 3 Flash Preview
10. Gemini 3.1 Pro Preview
11. Gemma 4 31B
12. GLM 4.7 Flash
13. GLM 5.1
14. GPT 5.3 Codex
15. GPT 5.4
16. GPT 5.4 Mini
17. GPT 5.4 Pro
18. gpt oss 120b
19. Grok 4.1 Fast
20. Grok 4.20
21. KAT Coder Pro V2
22. Kimi K2 Thinking
23. Kimi K2.5
24. MiniMax M2.7
25. Mistral Large 3 2512
26. Mistral Medium 3.1
27. Mistral Small 4
28. Nemotron 3 Super (free)
29. Nova 2 Lite
30. o3
31. o4 Mini
32. Olmo 3.1 32B Instruct
33. Qwen3 Coder Flash
34. Qwen3 Coder Next
35. Qwen3 Coder Plus
36. Qwen3 Max
37. Qwen3 Max Thinking
38. Qwen3.5 397B A17B
39. Qwen3.5 Flash
40. Qwen3.6 Plus
41. R1 0528
42. Trinity Large Preview (free)
43. Trinity Large Thinking

Update History

April 9, 2026

Models Removed

• GLM 5: was Rank #5, AIMAC Debt 2.66, Cost $2.03
• GPT 5.1 Codex Mini: was Rank #6, AIMAC Debt 3.78, Cost $0.39
• Qwen3.6 Plus Preview (free): was Rank #12, AIMAC Debt 4.44, Cost $0.00

Models Added

• Qwen3.6 Plus: Rank #22, AIMAC Debt 5.41, Cost $0.99
• Claude Opus 4.6 (Fast): Rank #23, AIMAC Debt 5.94, Cost $112.18
• Gemma 4 31B: Rank #40, AIMAC Debt 12.59, Cost $0.08
• GLM 5.1: Rank #42, AIMAC Debt 16.38, Cost $2.47

Notes

• GLM 5.1 replaces GLM 5 as Z.AI's latest flagship. GLM 5.1 was retrained for coding/agentic tasks and achieved SOTA on SWE-Bench Pro, but dropped from #5 to #42 on accessibility.
• GPT 5.1 Codex Mini retired as OpenAI consolidates around the 5.3/5.4 generation.
• Qwen3.6 Plus is the GA release of the preview. Free access ends; the paid model costs $0.99.
• Claude Opus 4.6 (Fast) is Anthropic's fourth model on the benchmark, filling the premium-fast slot.
• Gemma 4 31B is Google's first open-weight model to pass the 10KB HTML threshold.

Impact

• The version-upgrade regression pattern strikes again: GLM 5 (#5) to GLM 5.1 (#42) is the largest rank drop from a version upgrade we have tracked (37 ranks). It joins KAT V1 to V2 (-28) and Sonnet 4.5 to 4.6 (-18).
• Z.AI drops out of the Top 3 Winners. Alibaba/Qwen enters, with Qwen3 Max at #7 and eight models across the leaderboard.
• Trinity Large Preview (free) moves to #5, a free model in the top five.

April 1, 2026

Models Added

• Trinity Large Thinking (Arcee AI): Rank #30, AIMAC Debt 7.97, Cost $0.25