Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

Full article content is not available for inline reading.

Read the original article →

Some thoughts on the Navier–Stokes Millennium Prize Problem

On the Navier–Stokes Millennium Prize Problem introduces an impressive result from OpenAI, who used an unreleased model to produce a resolution to the Navier–Stokes existence and smoothness problem, one of the seven Millennium Prize Problems that have been subject to a $1,000,000 prize since May 24th, 2000.

The discovery is somewhat overshadowed by accusations of skulduggery from Tristan Buckmaster, an NYU mathematics professor who was collaborating on related problems with Levent Alpöge, an accomplished mathematician who currently works for Anthropic.

Tristan’s complaint accompanied a hastily published version of their own results. Here’s the PDF describing what happened. The very short version is that Tristan and Levent worked on the problem for almost a year, making extensive use of Claude and Codex (mainly GPT-5.6 Sol), then had a breakthrough on August 15th. The mathematical rumour mill kicked into gear and Tristan and Levent heard that OpenAI had heard that Anthropic had resolved “a major open problem”, so they reached out and learned that OpenAI had a team working on a related problem, with a similar approach. Quoting Tristan:

I asked when the first prompt had been sent by them. This question was not answered directly by OpenAI for some time. Eventually it was agreed that it had been sent in the past few days, after information about our work had reached OpenAI.

I asked whether the model had been trained on, or had access to, our sessions in Codex, into which we had been putting all our drafts for the whole of this project. I was told the model did not look up user data. I asked again, about training, and I did not get an answer.

It gets more complicated from there. The OpenAI team offered to wait for Tristan to publish, or to have him author a paper about their result, but were clear that Levent would not be invited as a co-author due to OpenAI’s competitive relationship with his employer.

Here’s how OpenAI described their work:

On Tuesday, September 1, we heard rumors that two Millennium Prize problems had been resolved. Inspired by these rumors and by the step change in performance of our internal model, we launched an effort to evaluate it on all open Millennium Prize problems and a few other high-impact problems. [...]

The agents arrived at their resolution on Saturday, September 5, about 88 hours after the first agents were launched. Lean formalization and verification took an additional 17 hours via GPT‑6 Astra.

Across all attempted problems, the agents sent 4.9 million messages and used about 300 billion output tokens. In the process of resolving the Navier–Stokes problem, the agents sent 2.7 million messages and used approximately 130 billion output tokens.

(We don’t know the cost structure of the internal model they used, but 300 billion output tokens at public API prices for GPT-6 Astra would cost $15,000,000.)

Here’s where they provide their perspective on Tristan and Levent’s work (emphasis mine):

Our effort began on September 1st after hearing a rumor which we later realized was related to Levent Alpöge, an Anthropic employee, and Tristan Buckmaster, a math professor at NYU. After the completion of our full project and Lean verification (on September 6th), believing from the rumor they also had a solution of Navier–Stokes, we reached out to them to offer a concurrent release of our result and to recognize their priority in a joint announcement. [...]

We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem. While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models. However, our proofs differ significantly and even the precise results proved are different in the Euler case (forced vs unforced).

My interpretation of what happened here is that OpenAI heard that some Millennium Prize problems had been solved using LLMs and saw this as an opportunity to demonstrate the power of their latest model, without thinking too hard about the optics of scooping a team who had been using OpenAI’s own models to work on this problem for the best part of a year.

This situation appears to mirror what’s happening in the world of computer security right now. Anil Madhavapeddy recently pointed out that Just a rumour of a bug is enough to find a security exploit these days, because if someone knows that some software has an unpatched vulnerability, they can set their agents the task of finding it. Is the same now true of mathematics? Just knowing that there is an unpublished solution to a problem might trigger millions of dollars in LLM spending to get there first.

This also highlights one of my ongoing frustrations about how all of this works. When an AI lab says that my data is “used to improve model performance”, what does that actually mean?

My two favourite hypothetical questions regarding this used to be:

• If I’m running Codex and one of my API keys accidentally gets consumed in the context, what are the chances that someone else might ask for an API key in the future and get mine back? (I asked someone at OpenAI once and they called this the “regurgitation” problem and assured me that they take great pains to prevent that... but wouldn’t describe how.)
• If I brainstorm with ChatGPT about potential new directions for my company, what’s the chance that information might be exposed to a competitor in six months’ time who asks “what might company X plan to do next”?

My new preferred hypothetical for this is:

• If I use ChatGPT to help me partially solve a Millennium Prize problem, what are the chances that my work will influence training such that a later model helps someone else solve it first?

Built for Reliability: How American Express Processes Payments at Scale

A customer making a card payment expects it to get approved or declined almost instantly. This single requirement shapes everything about how a payment is built, because the entire processing of the payment has to finish within this short window.

We recently spoke with Ben Cane, a Distinguished Engineer at American Express, to understand how their platform handles it.

To understand how things work, let us picture what happens when a transaction arrives at American Express. It is routed into one of several independent processing units and moves through a chain of microservices. However, partway through, one of those services begins to fail, and the customer at the checkout terminal is still waiting.

Just retrying the transaction inside the failing unit carries obvious risk. Also, moving the transaction to another server is an even harder problem. This is because the second unit would need to know how far the first transaction went. Sharing such knowledge creates a dependency that can turn two independent units into a fragile system.

The American Express engineering team handles this with a cell-based architecture that helps keep the impact of a disruption to a minimum. In our chat with the engineering team of American Express, we learned that most of the engineering effort goes into keeping that isolation intact under pressure. In this article, we will try to understand how the transaction runs through such a cell-based architecture and how the payments are processed even when some services are failing.

Core Payments Ecosystem

In a typical payment flow, American Express sits in the middle of a chain. A merchant holds a relationship with an acquiring bank that sends the transaction to American Express, and American Express delivers it to the card issuer that holds the account balance. This middle hop is the core payment network’s job.

In 2018, American Express began modernizing this platform onto cloud-native infrastructure. Immediately, it was clear that one assumption had to change. While the older systems ran on hardware engineered to stay running, the cloud infrastructure behaved differently. Servers could disappear for reasons outside any one team’s control. You have to assume failures completely outside your control will happen more often, and build the application around that.

Two familiar patterns were considered:

• Event-driven processing fit poorly. The workload needs to run at scale with low latency and a real-time response, leaving little room for delay. Asynchronous work exists, but the answer itself still has to arrive fast.
• A monolith is where most teams start, but it fits the scaling requirements poorly.

The design American Express engineering team ultimately chose was driven, in part, by decades of internal thinking. Ben pointed out that back in the SOA era, long before microservices existed as a concept, his teams were already trying to reduce the impact of a single failure on a specific system. The patterns were present even before the more formal term of cell-based architecture was coined.

Cell Boundaries

In a cell-based architecture, a cell is a complete, self-sufficient copy of the payment processing stack. In other words, everything required to process a transaction lives inside one boundary. This includes microservices, databases, DNS, and supporting infrastructure.

There are five key properties of a cell:

• It deploys independently and processes payments on its own.
• It owns its microservices, databases, and other components.
• It forms a single failure domain, so problems inside stay inside.
• It can be pulled out of rotation for maintenance or during an incident, with the rest of the platform carrying on.
• It does not have synchronous cross-cell dependencies in the critical path.

Reference data still replicates across cells, observability data can aggregate across cells, and router instances communicate with each other across cells. What stays out of the critical path is any blocking call between cells during transaction processing.

To put it differently, a cell is defined by its failure boundaries rather than by a specific infrastructure construct. Cells stay within a single region with everything needed for processing local inside that boundary.

Deciding on the appropriate size of a cell requires judgment. The engineers at American Express think of it as a balancing act. You don’t want to put the entire enterprise within a cell. Ideally, it is better to draw the boundary around the journey being supported. In the case of payments, this means the minimum components required to produce an immediate answer. In other words, the real-time journey of a transaction. The after-the-fact journey can still be handled in a different cell.

To be clear, cells are not the same as microservices. While microservices divide a system by function, cells divide it by failure. One cell can contain many microservices. The boundary matters because whenever we leave the cell, we enter dangerous territory and need the right resiliency capabilities, processes, and business logic to handle the potential failures.

Data Locality

A cell remains self-sufficient only when the data it needs already resides within it. The American Express engineering team has three strategies to deal with data, chosen according to how often the data changes:

• Immutable data is set up once and stays fixed thereafter.
• Semi-static data changes anywhere from every couple of hours to once a year. For example, exchange rates, merchant category codes, and country codes.
• Dynamic data changes with every transaction.

For the first two categories, American Express distributes the reference data to every cell before any transaction needs it. The alternative would be a fall-through read, meaning a lookup that misses the local cache and travels to a central system of record while the transaction waits. Pushing the data ahead provides multiple benefits:

• The first transaction avoids paying for a cold cache.
• The critical path avoids a synchronous call leaving the cell.
• The replication work runs entirely outside the transaction path.

Ben contrasted this with the more common pattern. He said that they lean toward push and distribute rather than the traditional pull and then cache, which spares that first transaction from building up caches.

However, the same approach falls short for dynamic data. Replication runs quickly and asynchronously. But it still leaves a window where a cell may hold stale state when a transaction arrives. Sending a transaction to a cell with stale data would add latency and risk a processing failure.

To deal with this scenario, American Express inverts the problem. Rather than moving the data to the transaction, the platform moves the transaction to the data.

This works through deterministic routing, meaning the decision follows from the transaction’s own contents rather than from cell load or availability. Some example attributes for this transaction content are partner, market, and payment type. A component called the Global Transaction Router makes the decision at the front door. We will look at it in more detail in the next section.

Deterministic routing is one of two modes. The router also supports priority-based routing in which cells carry a specific ordering and traffic goes to the highest-priority healthy cell available. The choice of the mode that is used depends on the use case.

However, this routing is not a universal rule. American Express applies deterministic routing selectively, where transactions require strong consistency between them. This depends on the transaction type. Some types carry minimal data requirements and can be routed freely.

Message-based replication between cells continues throughout, so failover data exists in more than one place. Timing is the important part here, because every in-flight transaction proceeds without waiting for replication to finish.

Global Transaction Router

The Global Transaction Router routes traffic and enforces the cell boundary at the same time.

Every transaction enters a cell through the router, and any transaction moving to a different cell travels back through it as well. Cells lack any ability to communicate directly, which makes the router the single path between them. The result of this setup is a payments mesh that connects cells globally while also keeping them independent.

The router handles external traffic on the same terms. When a cell finishes its work and the transaction needs to reach a card issuer, the cell returns it to the router. The router then makes that outbound call. Cells don’t talk outside their own boundary, whether the destination is another cell or an outside institution.

Concentrating that much responsibility in one component raises the obvious question: How does a component that every transaction depends on avoid becoming the platform’s weakest point?

The answers from American Express engineers can be split into parts.

The first is keeping the router deliberately simple. They keep business logic out of the router and give it just enough message parsing capability to extract certain values and route on them. The knowledge of what those values represent falls outside its job. The inputs vary by transaction type, sometimes a header value and sometimes a field inside the card transaction message.

The second direction is reducing what the router depends on:

• Dependencies stay minimal. The closer a component sits to the edge, the fewer dependencies it carries.
• State stays out of persistent storage. Router state lives in non-persistent stores, which keeps instances as close to stateless as the design allows.
• Remaining dependencies run asynchronously. Logging uses an asynchronous logger with a buffer truncation policy, so a full buffer drops log records rather than blocking transaction processing.
• Configuration loads into memory and updates asynchronously, so an unreachable config service leaves the router running on last known good values.
• Instances run in parallel across regions. Multiple instances, multiple regions, and multiple connections mean an unavailable path always has a backup.

During the discussion, Ben also gave a clear preference on deployment posture. He said he would avoid active-standby as much as possible and run something like this as active as possible, since active-standby earns its place mainly when you have state to manage.

Credit Card Authorization Flow

The journey of a transaction starts outside the American Express systems. A card gets used at a merchant’s point-of-sale terminal. From there, the sequence goes as follows:

• The Global Transaction Router receives the request first.
• The router applies deterministic or priority-based routing.
• Inside the cell, a collection of microservices performs validation, enrichment, transformation, and issuer determination.
• The cell returns the transaction to the router.
• The router sends the request to the appropriate card issuers for authorization.
• The issuer response arrives back at the router.
• The router uses deterministic routing to reach the cell holding the context of the transaction for authorization.
• The cell validates the response.
• The transaction travels back to the merchant’s acquiring bank.
• The final confirmation of the payment is sent back to the POS terminal.

One thing to note is that the cell never contacts the card issuer directly. This preserves the rule that cells should not communicate with anything beyond their boundaries.

Mid-Transaction Failure

Payments processing at American Express uses an orchestrated microservices architecture. This means one orchestrator microservice manages the workflow and calls the other microservices in turn. This orchestrator also monitors the health of those microservices continuously and detects failures.

Here is the full sequence of events that happen once a required service starts failing:

• The orchestrator detects the failure and halts processing.
• It sends the transaction back to the Global Transaction Router.
• The router selects a healthy cell.
• Processing restarts in that cell using the original transaction data.

The last step is where the design diverges. American Express discards the partial work. Every microservice call the failing cell completed gets thrown away, and the second cell starts from the beginning with the same input. This rerouting covers both new transactions arriving at the failing cell and transactions already in flight inside it.

The reasoning behind this approach goes back to the boundary problem. Resuming a transaction would require the second cell to read state from the first. However, that link would create shared state between cells. Shared state brings synchronization problems and consistency risks during failover, which is the exact problem this architecture seeks to avoid.

Recovery Semantics

Restarting a transaction elsewhere can be done up to a specific moment. Think of it like a point of no return. The exact position of that moment is a design decision that a team needs to make. For example, rerouting is safe while a transaction remains inside the core payments ecosystem, and once it has gone to an external system such as a card issuer, it stays where it is.

For payment types where late rerouting of the transaction is not possible, safety comes from idempotency. Each transaction carries a unique identifier consistent across every retry and reroute, and downstream systems use it to suppress duplicates. Failback is controlled with canary capabilities using percentages.

There are three mechanisms here that narrow the window:

• Speed: Card payments move so fast that by the time a failed cell returns, the transaction has usually completed elsewhere.
• Idempotency identifiers: Duplicate suppression happens downstream, using the identifier that travelled with the transaction.
• Paced recovery: Percentage-based traffic control governs when and how much work a recovering cell receives.

Design Tradeoffs

• Duplicated services: Enforcing the boundary sometimes produces duplicate implementations of the same service across cells.
• Dropped log records: The buffer truncation policy means that under sustained pressure, the platform keeps processing transactions while losing part of the record of what it did.
• Delayed global visibility: Each cell writes logs, metrics, and traces locally first, with aggregation to global dashboards happening asynchronously.
• Rejected transactions: When a transaction requires strong consistency, but the required data cannot be validated or turns out to be inconsistent, American Express may reject that transaction to preserve data integrity.

Conclusion

The payment platform built by the American Express engineering team survives a cell failure because the design doesn’t allow one transaction to depend on two cells at once.

• Two data strategies, one goal: Reference data that changes rarely gets pushed to every cell ahead of time, while data that changes constantly stays put and the transaction travels to it.
• A thin component at the chokepoint: The Global Transaction Router carries enormous responsibility and very little logic, which makes it dependable.
• Restart in place of resume: Discarding partial work costs a few hundred milliseconds and removes the shared state that would link every cell to every other cell.
• A recovery window with a chosen position: Placing the point of no return late in the sequence widens the range in which failure remains survivable.
• Recovery semantics from the domain: The pattern provides structure, and business logic provides the rules for what a partially completed transaction requires.

A deploy rolls out to 60 pods. Each one starts, connects to Redis, warms a small cache, and registers with service discovery. The rollout is staged and takes four minutes.

Nine minutes later, every pod hits the database at the same instant. Nobody scheduled that. The pods were started at different times, they are not coordinated, and there is no shared clock driving them.

They synchronised themselves, because every one of them set a five minute TTL from a start time that the rolling deploy had already bunched into a narrow window, and after two refresh cycles the small differences had been absorbed by the fixed interval.

That is the entire failure. Independent things with a fixed period converge, and once they converge nothing pulls them apart again.

Fixed intervals are attractors

The intuition that trips people up is that clients starting at random times stay at random times. They do not, because most systems have events that reset everyone’s clock at once.

A deploy restarts every instance within a few minutes. A network partition disconnects every client and they all reconnect when it heals. A cache flush empties everything simultaneously. A leader election completes and every follower re-registers. After any of those, a fixed interval keeps the alignment forever.

The cache version of this is well known enough that I have written about TTL expiry as a scheduled simultaneous failure. What took me longer to notice is how many other things in a normal service have exactly the same shape and none of the attention.

Retries are the worst offender

Exponential backoff is standard advice and it is only half the fix.

Consider a thousand requests failing when a dependency returns 503. Every one of them waits one second, retries, fails, waits two seconds, retries, fails, waits four. The backoff is doing its job in the sense that the total rate decreases. It is doing nothing at all about the fact that all thousand clients are still moving in lockstep.

The dependency sees a thousand requests, then silence, then a thousand requests, then silence. Those spikes are what prevent it from recovering, because recovery usually requires a period of survivable load rather than a period of zero load followed by a wall.

import random

def backoff_no_jitter(attempt, base=1.0, cap=30.0):
    # Every client waits the same amount. Spikes persist.
    return min(cap, base * (2 ** attempt))

def backoff_full_jitter(attempt, base=1.0, cap=30.0):
    # Uniform over the whole window. Clients decorrelate immediately.
    return random.uniform(0, min(cap, base * (2 ** attempt)))

def backoff_equal_jitter(attempt, base=1.0, cap=30.0):
    # Guarantees a floor while still spreading. Useful when a very fast
    # retry would itself be a problem.
    ceiling = min(cap, base * (2 ** attempt))
    return ceiling / 2 + random.uniform(0, ceiling / 2)

AWS published measurements comparing these in their architecture blog on backoff and jitter, and full jitter came out ahead on both total work performed and time to completion under contention. The result is slightly counterintuitive, because full jitter sometimes retries almost immediately, and that turns out to matter less than never having a synchronised wave.

This is the piece that pairs with bounding retries so the resilience layer does not become the outage. Bounding limits how much amplification you get. Jitter limits how concentrated it is. Both are needed and the second one is usually missing.

The places nobody thinks to look

Retries and cache TTLs get discussed. These generally do not.

Cron jobs default to the top of the minute, the hour, or midnight. A fleet of services each running a nightly reconciliation at 0 0 * * * produces a thundering herd against whatever they all read, at the exact hour when nobody is watching. Spreading them by a random offset within the window costs nothing, because almost no nightly job actually cares whether it starts at 00:00 or 00:17.

Health checks and heartbeats fire on a fixed period from process start, so a rolling restart aligns the entire fleet. Sixty pods checking a dependency every 30 seconds is fine when spread and is a burst of sixty when aligned.

Reconnect logic after a dropped connection is the sharpest one, because the disconnect event itself is what synchronises everyone. A load balancer restart drops every connection at once, and every client reconnects after its fixed delay, at once, to a service that has just restarted and is at its least able to absorb a spike.

Token refresh is a quiet version. Credentials issued during a deploy expire together and get refreshed together, which puts a synchronised burst on the identity provider rather than on your own service, so it shows up as somebody else’s incident.

Scheduled cache warming, metrics flushes, and log shipping intervals all belong on this list for the same reason.

The fix is genuinely one line

def jittered(interval, spread=0.15):
    # Plus or minus 15 percent. Average rate unchanged.
    return interval * random.uniform(1 - spread, 1 + spread)

What I like about this is the cost profile. The average rate does not change, so capacity planning is unaffected. There is no coordination, no new dependency, no state to keep. The p99 of the interval moves by 15 percent, which for a health check or a cache refresh is not a number anybody cares about.

For a fleet of size N with an interval T, going from aligned to spread turns a peak of N requests into roughly N divided by the number of distinct slots in the jitter window. At 60 pods, a 30 second check, and 15 percent jitter, the peak drops from 60 concurrent to a handful.

The one place to be careful is that jitter on the first interval matters more than jitter on subsequent ones, because the first is the one anchored to the synchronising event. Some libraries only jitter after the initial delay, which leaves the worst spike intact.

Where it does not help

Jitter spreads load that was already going to happen. It does nothing about load that should not happen.

A retry storm caused by an unbounded retry policy is still a retry storm with jitter, just a smoother one. If every client retries forever, jitter converts a series of spikes into a sustained overload, which is arguably harder to diagnose because the graph looks like a traffic increase rather than an obvious wave pattern.

Jitter also does not help a genuinely single hot key, where there is one value and one expiry and no population to spread. That needs coalescing.

And it does not fix cold start. Restarting a cache tier gives you a 100 percent miss rate with nothing to jitter, because no key has a TTL yet.

What I would do

Grep for fixed intervals. Every setInterval, every scheduleAtFixedRate, every cron expression ending in 0 * * * *, every Thread.sleep inside a reconnect loop. Most of them will be fine. The ones that are not are the ones where the fleet is large or the target is shared.

Set the default in whatever wrapper your services use for scheduling and retrying, so it is applied by construction rather than remembered. A retry helper that jitters by default and a scheduler that offsets by a hash of the instance id will cover most of this without anyone thinking about it again.

The framing that made this click for me is that a distributed system does not need a coordinator to behave like one. Give a thousand independent processes the same constant and they will find each other, and the thing that finds them is arithmetic rather than anything you designed.

Full article content is not available for inline reading.

Read the original article →

ByteDance is preparing an AI model for real-time spatial video generation, with founder Zhang Yiming personally overseeing the work and a launch possible as soon as next month, Bloomberg reported, citing people familiar with the matter who asked not to be identified.

One of those people cautioned that the timing is not settled and the plans could change. TNW has not independently verified the account, and a ByteDance spokesperson did not respond to Bloomberg’s request for comment.

The model would be built on Seedance, ByteDance’s existing video generation system, and would let users create interactive virtual worlds for live streams, short-form dramas and games.

The reported specification is the interesting part: on-demand video at around 20 frames per second with latency of roughly 0.05 seconds, generated in the cloud rather than on the device.

That last detail is the strategy. Rendering spatial content remotely takes the computational load off the headset, which lowers what the hardware has to do and therefore what it has to cost.

ByteDance owns Pico, its extended reality arm, and the model is reportedly meant to generate worlds that respond to Pico users’ voices and movements.

If it works, the contest over XR moves away from hardware specifications and towards models, cloud capacity and content distribution, three things ByteDance already has.

None of this arrived from nowhere. 36Kr reported earlier this year that ByteDance had set four AI priorities for 2026, with world models at the top of the list, ahead of holding Seedance’s lead in video, improving coding, and commercialising Doubao.

World models were said to command the company’s largest data budget of any model direction, an eight-figure sum in renminbi that 36Kr’s sources put at three to four times what rivals were spending.

The same reporting set the target explicitly: ship at least one world model by the end of the year and measure it against Google’s Genie, which now lets users walk around Street View imagery rendered in real time.

Internal testing early in 2026 put ByteDance about 10% behind the global state of the art, on the same account. A launch next month would be ahead of that schedule.

The company is pursuing two routes at once, according to 36Kr: a vision-language-action approach aimed at embodied intelligence and robotics, and 3D simulation for entertainment and games.

The spatial video model belongs to the second, which is also the one with an existing user base attached to it.

A world model, in the sense everyone is now using, is a system that learns how environments behave well enough to render one that responds coherently to what a user does in it.

The reason video companies keep turning up in this field is that the training material is video, and the firms with the most of it, and the most experience compressing it into something that renders fast, start from an unusual position. ByteDance has spent a decade building exactly that pipeline for a different purpose.

Bloomberg frames Zhang’s involvement as putting him among researchers such as Fei-Fei Li and Yann LeCun, who have argued that models grounded in visual and physical understanding, rather than language alone, are the route to systems that can act in the world.

That case is now being tested commercially by companies whose actual product is entertainment.

The money behind it is not in doubt. ByteDance secured a $30bn loan last week, Bloomberg reported, and has been weighing capital expenditure of as much as $70bn on its AI build-out.

Seedance already underpins CapCut and Doubao, and the company remains, on its own domestic terms, a challenger against Alibaba, DeepSeek and Moonshot AI.

For Meta and Apple, the implication is awkward rather than immediate. Both have spent heavily on headsets that have not gone mainstream, and Europe’s own XR specialists have built their businesses on high-end hardware rather than cheap devices fed from a data centre.

A cloud-rendered world model does not beat a Vision Pro on fidelity. It just makes the fidelity somebody else’s problem.

Apple today introduced its first foldable device, named the iPhone Duo, at its Surprise and Shine event held in Cupertino. The phone is the first marquee device launched under the company’s new CEO, John Ternus.

Unlike early iterations of foldables, which looked like vertical slabs, Apple adopted a wider form factor that could aid in a better aspect ratio while watching videos when the device is unfolded.

The unfolded display is a 7.6-inch Retina display, and the outer display is 5.4 inches. The company said that the new device was “inspired by the versatility of iPad” and showed how people can use it unfolded in a vertical mode. The inner display also features an “under-display” camera that can give users an all-screen feel.

Apple said that it used a custom nanotexture finish to hide the crease and reduce glare. Most foldables still have a visible crease, but companies like Oppo have made strides toward an almost seamless phone. The company claimed that its hinge was created using grade 5 aluminum and over 100 special components for smooth opening and closing and durability.

Notably, the device only has Touch ID, with Face ID missing. You can unlock the device using the Apple Watch, though.

The device is powered by the new A20 Pro chip and C2 model, which allows for faster uploads. Just like last year’s iPhone Air, this particular model is eSIM-only.

On the battery front, the company said that with the inner display, users can get 31 hours of video playback, and with the outer display, 44 hours of video playback.

The phone features a 48-megapixel fusion main camera and a 48-megapixel ultrawide camera with 2x optical zoom. This dual-camera setup skips a telephoto camera. Just like other foldables, users can take rear-camera selfies and look at previews on the outer display. It also has a Google Pixel-like feature that plays an animation on the outer display, which is useful when you are trying to get kids to focus while taking photos. There is also a smart-take feature that uses an on-device model and snaps an image when all subjects within the frame are posing.

The company has made iOS 27 adaptable to the new foldable — essential controls like Wi-Fi move to the side and dock on the home screen vertically oriented. Users can use apps in split screen and also pair the apps. If someone is using the phone vertically, they can pin a video on top and use other apps in the bottom half. One of the fascinating features is that users will be able to use the iPhone half folded or even set it on a table. Apple noted that later this year, the device will support Apple Pencil.

The iPhone Duo is priced at $1,999 with 256GB storage and comes in two colors. Preorders for Apple’s new foldable begin on October 16, and the device will be available on October 23.

Rivals like Samsung and Xiaomi have already launched their version of wide foldables before Apple.

According to analytics firm Counterpoint, foldables contribute to less than 2% of shipments in the overall smartphone market. However, Apple’s new phone could be an instant hit, as the analytics firm said the Duo could garner up to 25% market share by the end of the year.

We read and analyze data with implicit business context that helps us interpret it. We know what data points mean and how they fit together. We know broader business objectives, which help us zoom out and see the bigger picture. We can recognize when something doesn’t smell right and dig in further before proceeding.

Agents just have data points. If you give agents a bunch of ones and zeros, they won’t automatically know their significance or how to connect them. Even worse, they think they understand the data and make a series of bad decisions from that assumption.

Agents need to understand what you’re feeding them before they can provide value. They need context and need to be pointed at specific use cases and data sets to start.

“AI can’t squint” is a great way of saying it doesn’t inherently have the context to sniff out the good from the bad.

Context: How you teach agents to think like humans

Let’s say you feed your agent sales data that shows a sudden spike in orders for a product. A human immediately realizes that the spike aligns with a limited-time promotion or a seasonal event and treats it as temporary.

But your agent doesn’t automatically have that context. It interprets the spike as the beginning of a long-term trend and recommends higher production. You end up with overstock and squandered resources. This could have been avoided had your agent understood the “why” behind your numbers.

When you define your metrics and the context around them, agents start to understand more of your universe and accurately connect the dots across your data points.

Your agents need both quantitative and qualitative context to do good work. Without qualitative data, you’re only getting half the picture.

Quantitative data can tell you what’s happening, but your qualitative data explains the drivers behind it. Voice of the customer (VoC) channels, session replays, and other qualitative touchpoints create tons of rich, unstructured data. Agents need to learn context from all these inputs.

How do you communicate the context of your data to agents? Via your semantic layer.

Building your semantic layer

A semantic layer is your company’s data translator. It uses plain language to explain what your columns of data mean. It also tells agents how to read the data and gives them everyday terminology to use when they talk back to your team.

Let’s say you have an event in your data called “GHI completed.” Humans on your team know that stands for “guided home in installation” and that it happens at the end of your onboarding flow. But if you ask your agent how many users completed onboarding, they won’t know to look at the “GHI completed” event. There’s nothing connecting the dots.

That's what context solves. It defines what an event means, when it occurs, and why it matters so everyone can interpret it correctly.

Last year, the Amplitude data team worked on getting our marketing attribution product data ready for AI. We had to build our semantic layer to do that, so our data team:

• defined every column being used by the business across Salesforce objects
• captured business definitions using real words people use in Slack and meetings to talk about their data
• flagged deprecated fields, duplicates, and added synonyms

This semantic layer provides all the context our agents need to produce accurate outputs. It also lets anyone ask questions about our data and get valid answers.

With a fully-built semantic layer, our agent answered questions correctly 80 to 90 percent of the time. The team tuned the last 10-20 percent using custom instructions and enriched the semantics even more based on user testing and feedback.

Rather than starting with a blank page, our data team used AI to speed up the process of building the semantic layer. AI generated the first draft of the definitions. Then data owners and stewards reviewed and edited them.

You can build your semantic layer in a data catalog, DBT YAML, using database metadata, or a system that makes the most sense for your business. Even a spreadsheet works if that’s where your team works.

Keep these things in mind as you build your semantic layer:

• Write business definitions using the words your team actually uses in Slack and meetings, not technical field names
• Add synonyms, including the UI label context, not just the API name
• Map data lineage so agents can trace where data comes from and understand the downstream impact of changes
• Tag sensitive data and give data a certification status so agents know what's trusted and what isn't
• Assign clear ownership: data owners, stewards, and technical contacts for each area
• Flag deprecated fields, duplicates, and add synonyms so agents don't get confused by outdated or overlapping terminology
• Use AI to generate a first draft of definitions, then have data owners review and refine them rather than starting from scratch
• Connect quality signals so agents know how much to trust the data when they answer questions

How Amplitude solves for context

Don't rely on tribal knowledge. Build domain knowledge directly into Amplitude so both your team and your AI agents can understand your data.

• Organization Level: Define company-wide standards, including business models, KPI definitions, fiscal calendars, and your North Star Metric in the Org Context field (10,000-character limit). Mark core assets as "Official" to guide AI agents toward trusted sources.
• Project Level: Layer in product-specific details and overrides—such as custom funnel logic, local terminology, or project-specific metrics—without cluttering global settings.
• Event & Property Level: Define what each event and property represents, while letting Amplitude automatically map event schemas, usage volumes, and catalog structures so you only need to document what can’t be inferred.
• Automated & Workspace Context: Amplitude automatically reads active chart settings, dashboard filters, and session replays, instantly pairing your documented rules with live user interactions.

AI Context in Amplitude makes sure every person and AI agent in your company has a shared, accurate understanding of your data.

Start with context and contained agent inquiries

If your kid needs to learn about dog training for a school project, you wouldn’t drop them at the front door of the library and let them wander around pulling every book that has “dog” in the title. You’d tell them to look in the animal section.

But before your kid can search that section, the librarians need to curate it. They need to comb through their books and identify the right ones to make that search as simple and direct as possible. Data teams need to do the same when organizing data sets for agents. Start by picking the data that makes sense for the specific questions you're trying to answer.

A solid semantic layer does double duty here. It not only embeds context and definitions into your data. It also makes your data easy to get around. It gives agents the business context they need to read your data correctly, and it gives data teams a place to say which datasets apply to which questions. Control the scope, give agents the right context, and you'll get answers that make sense.

For example, let’s say you have a question about customer churn for mobile app users in Europe. Feeding the agent your entire global customer database, including unrelated product lines, regions, and inactive segments, would be a waste of time and energy and could divert or dilute your agent’s attention. It’s introducing too much room for error.

Instead, start by defining a specific question: Why are European mobile app users churning in Q2? Then you'd pull together a focused dataset just for that group: recent usage logs, support conversations, satisfaction scores, and cancellation notes. This keeps the agent from drawing the wrong conclusions from data that doesn't belong, like behavior in other markets or channels that have nothing to do with this.

Have humans double-check (even with solid context)

Context is great, but you still shouldn’t give your agents the thumbs up without someone double-checking the work. You can automate and accelerate verification by giving your agents a certified result or sample dataset that they can measure against. These provide “the rules of the game” for value ranges, statistical thresholds, and business logic rules. Then, automation can check, flag, and send to humans for review.

Treat each incorrect agent answer as a chance to fine-tune your approach and close gaps. Maybe it didn’t have enough context of the problem you were trying to solve, so you need to provide more information.

Having humans double-check results keeps things accurate, lowers the risk of mistakes, and builds trust in what these agents can do. The more people see your agents getting it right, the more they'll trust them with real work. For data teams, that means fewer manual checks, faster decisions, and more time for the analysis that actually matters, instead of QA and data wrangling.

AI is becoming a more common part of business operations, from customer support and data analysis to software development and internal processes. However, adopting AI does not always lead to the results companies expect. In many cases, the challenge is not the technology itself, but how businesses choose use cases, prepare their data, integrate AI into existing workflows, and measure its actual impact.

1. Starting with AI instead of the business problem

One common mistake is to begin with the technology. A company may decide that it needs a chatbot, an AI assistant, or an AI agent because these technologies are becoming popular. The business then looks for a way to use them, rather than first identifying a specific problem that needs to be solved.

This can lead to projects that work technically but have limited business impact. For example, an AI assistant may answer employee questions well, but if employees rarely use it or the information it provides is not connected to their daily work, the actual value may be small.

A better starting point is to identify where the business is losing time, money, or opportunities and then consider whether AI is an appropriate solution.

2. Data is often a bigger problem than the AI model

AI systems depend heavily on the quality, accessibility, and governance of the data they use. However, many organizations still have data spread across different systems, inconsistent formats, duplicate records, or information that is difficult to access.

IBM identifies data quality and readiness, fragmented data environments, governance, and security as important challenges organizations face when scaling AI beyond experimentation.

This is particularly important for businesses building AI systems around their own internal information. A more capable model cannot fully compensate for incomplete, inconsistent, or poorly governed data. As a result, preparing data, defining access rules, and connecting relevant sources can be as important as selecting the AI technology itself.

3. AI is not always connected to existing workflows

Another challenge is the gap between an AI tool and the way employees actually work. An AI application might perform a task well on its own, but employees may still need to move information manually between the AI tool and their CRM, ERP, document management system, or other business software.

Designing an effective AI workflow can help businesses connect AI capabilities with existing processes and systems, reducing unnecessary manual steps and making the technology more useful in day-to-day operations.

This is why simply adding AI to an existing process may not be enough. In some cases, the process itself needs to be adjusted so that AI can provide useful support at the right point. Research from McKinsey has also pointed to workflow redesign as an important factor in turning generative AI adoption into business impact.

4. Expectations can be higher than what AI can realistically deliver

AI capabilities have improved quickly, which can make it easy for businesses to expect too much from a new system. Some organizations may expect AI to automate an entire process from beginning to end, while the technology may be better suited to handling only certain steps. Other businesses may expect immediate cost savings without considering implementation, integration, monitoring, and ongoing maintenance.

This does not mean AI is ineffective. It means that the scope of an AI project needs to match the business problem and the technology’s current capabilities. Starting with a smaller, measurable use case can sometimes provide a more realistic path to wider adoption.

5. Success is difficult to judge without clear metrics

Another issue is that businesses do not always define what success should look like before starting an AI project. It is easy to measure technical indicators such as response time, accuracy, or the number of users. But these metrics do not necessarily show whether the project is helping the business.

For example, a customer service chatbot could have a high answer accuracy but still fail to reduce support workload. Similarly, an AI coding assistant may be widely used but have little effect on development costs or delivery time.

Clear business metrics are therefore important. Depending on the use case, these could include reduced processing time, lower operating costs, faster response times, higher conversion rates, or fewer manual errors.

6. What a more practical AI approach looks like

There is no single approach that works for every organization. However, several principles can help businesses reduce unnecessary risk:

• Choose a business problem with a clear potential benefit.
• Assess the quality and accessibility of the relevant data before development begins.
• Consider how AI will fit into existing systems and workflows rather than treating it as a separate tool.
• AI systems need to be evaluated after launch. Business requirements, data, user behavior, and AI models can all change over time, so ongoing monitoring and improvement are often necessary.

7. An expert perspective from PowerGate Software

From a software development perspective, the main challenge is often not simply choosing an AI model. It is making sure the technology fits the business context.

According to Mr.Chung Tran – a technology expert at PowerGate Software, AI projects tend to create more value when they are connected to a clear business objective and an existing workflow. Companies do not always need to start with a large-scale implementation. A focused use case, reliable data, and clear measures of success can provide a more practical starting point.

This view is consistent with a broader shift in the AI market. Businesses are increasingly moving beyond small experiments, but scaling AI requires attention to data, processes, infrastructure, governance, and user adoption, not just the AI model itself. IBM similarly notes that organizations moving toward broader AI adoption are facing challenges around data, governance, ROI, skills, and workflow integration.

8. AI adoption is becoming more about execution

As AI adoption continues, businesses are likely to pay more attention to how these technologies fit into their actual operations. The challenge is no longer simply deciding whether to use AI, but understanding where it can solve a real problem and how its impact can be measured.

This does not mean every company needs a large AI strategy or a complex implementation. In many cases, a smaller use case can be a better starting point. Businesses can test whether AI improves a specific process, learn from actual user feedback, and expand the solution if the results are meaningful.

AI can offer real value to businesses, but successful adoption usually requires more than adding a new AI tool. Companies need to start with practical business needs, use reliable data, set realistic expectations, and track meaningful results. As AI adoption continues to grow, these factors will become increasingly important for businesses that want to turn AI investment into measurable outcomes

What will our economic future look like?

We don’t know yet how AI will reshape the economy. Will it lead to unprecedented growth? Widespread unemployment? Neither, or something else? How can we tell?

Anthropic’s Economics team built a model of how AI might affect jobs, growth, and unemployment in the US in coming years. Read about possible economic futures and make your own predictions about AI capabilities to see the economy they imply.

We study how AI is reshaping the economy because we’re committed to ensuring that this transition is beneficial for society, including workers. By providing better visibility into our possible economic future, we can take steps to make sure that everyone benefits from it.

While our Economic Index measures how AI is being used across the economy right now, this scenario explorer is about looking ahead. Based on our technical report, Economic Scenarios for Transformative AI (Korinek et al., 2026), this explorer gives you a chance to find out what the economy might look like as AI continues to get more capable.

In scenarios ranging from business as usual to an economy where AI increases growth to about twice the normal rate, unemployment stays within the historical range and wages remain flat or rise depending on the industry. But in scenarios where growth is faster than anything in economic history, there are adverse impacts on wages and job prospects for knowledge workers. In those scenarios, society is far wealthier, so the challenge is making sure that the gains are broadly shared.

As you scroll down, you’ll see an overview of how AI affects the economy. Then, you can plug in your expectations for how capable AI will be, and how extensively it will be used across the economy in the future. The model will show you what the economy in 2030 might look like if your predictions come true—and how your predictions compare to others.

The economy is made out of tasks

This model represents all the jobs people do in the economy as bundles of tasks. AI can help people do a given task better or faster. It can automate the task. It might not affect the task at all. And it can lead to new tasks.

Think of a day in the life of a nurse

You can think of any job as a bundle of tasks that someone does. She does rounds to check on a sick patient. She draws blood. She triages incoming patients, charts patients’ vitals, and orders supplies for the ward. That’s just the start of the list.

Each of the tasks listed are based on the US Department of Labor’s O*NET taxonomy, listing the tasks for each occupation.

Jobs change over time

Tasks leave the bundle (hardly anyone hand-writes paper charts anymore) and new tasks arrive (30 years ago, no one monitored patients remotely). The bundle of tasks isn’t static, and the job changes as tasks change.

Only humans can do some tasks

For instance, AI can’t bathe a patient.

Some tasks will get augmented

AI helps a human do them better, or faster. AI helps the nurse draft discharge instructions, monitor patients remotely, and plan the shift’s care schedule.

Some tasks may get fully automated

For instance, AI may chart a patient’s vitals, or order the ward’s supplies.

And new tasks will appear

Historically, new technologies have also created new tasks for workers. For a nurse, that might be checking how well an AI triages patients, or reviewing an AI-proposed care plan.

The result: the nurse’s job changes

As the nurse incorporates AI into her job, the nurse is able to oversee and accomplish more. She can spend more time talking with patients and helping them understand diagnoses. Productivity increases.

Every task happens millions of times every day, across the country. Nurses are doing their work on every ward and on every shift. As more nurses use AI, AI supports a higher percentage of these millions of instances of each task.

From tasks to the economy

Today, if you add up every single instance of tasks performed in the US, by people and by the machines and software they work with, you get the US economy: over $30 trillion of value created over the past year. So how will AI shape the economy of the future?

The answer depends on how AI affects all the tasks that make up the economy, the new tasks it creates, and how fast AI takes on this work. Will AI lead to more task augmentation or automation? How much more productive will it make us? How quickly will it be adopted by workers and companies? The answers to these questions have direct effects on GDP, the labor market, and the share of the pie taken home by workers.

There are many possible futures, but we’re highlighting three scenarios

The future will depend on how AI’s capabilities advance, and how industries and workers adopt those capabilities. The three scenarios we share capture distinct kinds of impact.

In the modest scenario, it’s hard to see the effect of AI in macroeconomic data: its economic impact is something like the internet’s. In the substantial scenario, AI makes a bigger impact than the internet, or the railroad. And in the extreme scenario, AI drives a completely transformed, unprecedented economy, likely driven by recursively self-improving AI systems and a faster rate of AI adoption.

Small economic gains

In the modest scenario, AI has roughly the same kind of impact as the internet did. It drives real economic gains, but they’re within the historical norm for new technologies, and they arrive gradually.

A revolution in knowledge work

In the substantial scenario, AI is capable of doing half of all knowledge work by 2030, the majority of it autonomously, but it’s not adopted for all of that work: most knowledge work tasks are still done without AI. The economy grows at twice its normal rate. Wages for knowledge workers don’t rise, but other workers see gains.

A profound economic transformation

In the extreme scenario, AI is more productive than humans at the vast majority of knowledge-work tasks. It does nearly all of them autonomously, and it creates essentially no new knowledge tasks for people. This scenario would likely require recursively self-improving AI, adopted quickly for knowledge work.

As AI diffuses, annual GDP growth rates reach 15% a year, leading the economy to double in size every 4.5 years. As a society, we’re far richer than we’ve ever been, but many fewer workers have jobs in knowledge work, and unemployment has risen beyond typical recessionary levels.

People’s expectations about future AI capabilities vary.

In August, we surveyed more than 10,000 Americans about their views on present and future AI capabilities, adoption, and the ease of finding new work if they have to change occupations.

The typical respondent’s answers imply outcomes close to the “substantial change” scenario: GDP is 10% higher by 2030 than it would be without AI, and the overall unemployment rate has risen to around 5%. Around 10% of respondents have views in line with the extreme scenario.

Finding 1: GDP growth

AI drives GDP growth in all scenarios, although the scale varies enormously depending on the scenario.

But growth isn’t the only economic dynamic we care about. What would these potential futures mean for how much of this growth workers receive in their paychecks, or how many people have to find new jobs?

This model isn’t a complete map of reality, but it shows us some interesting findings. The country’s GDP will grow, but a larger share of that prosperity might go to the resources and technology used to create more wealth (capital) compared to workers, even if society as a whole is much wealthier.

And in most scenarios, job reallocation and unemployment both stay within ranges history has seen before, with one exception. In the extreme scenario, if we see recursive self-improvement and rapid adoption, unemployment could spike to historic levels.

Finding 2: Job reallocation

In more transformative scenarios, more workers have to change occupations. That may mean higher unemployment. There is always some churn in the job market—people losing jobs and finding new ones. In normal times, this process can be painful, but works relatively well from a macroeconomic perspective. Most job seekers find new jobs fairly quickly.

In our substantial and extreme scenarios, knowledge workers may see a lot of automation and displacement. At the individual level, it means coders and call service center agents may have to switch to jobs like electrician and nurse, which are less exposed to AI.

But changing occupations entirely is hard, and it takes many people a long time to land a new job. The more of this switching a scenario requires, the more people will be between jobs.

Finding 3: Wages

Across the three scenarios, average wages rise, but this increase is concentrated in occupations outside of knowledge work. That’s because it takes time for workers to switch to occupations where demand is rising. If there’s less demand for human knowledge work, that puts downward pressure on wages. Meanwhile, as AI increases productivity within knowledge work, the demand for manual work that benefits from that productivity will increase. For example, more quickly producing designs and permitting for physical infrastructure could increase the number of construction projects, resulting in rising demand for construction workers, which pushes those wages higher. In the substantial scenario, wages for knowledge workers are essentially flat. In the extreme scenario, they fall by more than 10% by 2030.

Finding 4: Labor vs. capital share

The pie will grow, but a larger share might go to capital. Today, of each dollar the economy produces, about 60¢ goes to workers and 40¢ go to capital. If the economy grows, but AI automates more tasks, more of each dollar might go to capital. This can happen even when wages for all workers rise substantially. If capital becomes more useful for more things, it will be in higher demand, which raises its price. In that world, more of the gains from a growing economy flow to owners of capital.

We find that the labor share falls noticeably in the substantial and extreme scenarios, and the capital share rises. Average wages rise—non-knowledge workers are paid much more—but wages for knowledge workers stagnate or decline alongside worsening unemployment.

In the extreme scenario, the gains from a rapidly expanding economy are unevenly distributed. Most knowledge workers face either lower wages or unemployment, and workers overall get a smaller fraction of the larger pie. Total labor income is barely changed by 2030.

In this scenario, the main challenge is not achieving economic growth, but making sure the benefits are broadly shared and the costs aren’t unequally dispersed.

The future is not predetermined.

Ultimately, what the economy looks like in 2030 depends on many factors, like what AI can do, and how companies and workers choose to adopt it. It also depends on how the financial benefit of this technology is shared.

Like any economic model, this one has limits. For example, we did not include scenarios where humanity develops hyper-capable robots. The model draws on our research and external review, and we’ll keep adding to it as the evidence develops.

Run any model, on any backend

100+ model families ported to pure Keras 3, with weights converted from the original checkpoints. The same code runs on JAX, PyTorch and TensorFlow, and nothing from transformers or torch is needed at run time.

Two calls to a prediction

Build the model with from_weights, then feed it whatever its processor produces. Every model in the library follows this shape, so moving between a detector, a depth estimator and an LLM costs you nothing.

pip install -U zeromodels

Weights come from the zeromodels org on the Hub, and the same identifier builds both the model and its processor, so the resolution and normalization always match the checkpoint.

import os

os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.models.detr import DETRDetect, DETRImageProcessor

model = DETRDetect.from_weights("zeromodels/detr-resnet-50")
processor = DETRImageProcessor.from_weights("zeromodels/detr-resnet-50")

image = Image.open("photo.jpg").convert("RGB")
output = model(processor(image)["pixel_values"], training=False)

results = processor.post_process_object_detection(
    output, threshold=0.9, target_sizes=[(image.height, image.width)]
)[0]

One call, three sources

from_weights dispatches on what you hand it: a preconverted Keras repo on the Hub, a bare variant name that converts an upstream checkpoint on the fly, or any compatible Hugging Face repo behind the hf: prefix. Architecture details, including the class count of a fine-tune, are read from the repo config.

from zeromodels.models.qwen3 import Qwen3TextGenerate
from zeromodels.models.segformer import SegFormerSemanticSegment

# Preconverted Keras weights (zm_config.json)
SegFormerSemanticSegment.from_weights("zeromodels/segformer_b0_ade_512")

# Bare variant: converted from upstream on the fly
Qwen3TextGenerate.from_weights("qwen3-8b")

# Any Hub repo with a matching model_type
SegFormerSemanticSegment.from_weights("hf:nvidia/segformer-b0-finetuned-ade-512-512")

# Architecture only, randomly initialized
SegFormerSemanticSegment.from_weights(
    "zeromodels/segformer_b0_ade_512", load_weights=False
)

Measured outputs, not illustrative ones

Every figure and every printed result on a model page comes from actually running the snippet beside it on the image or audio clip shown. Nothing is hand-written to look plausible, so what you read is what you get when you run it yourself.

Any backend, either data format

Set KERAS_BACKEND before importing Keras and the rest is unchanged. Models read keras.config.image_data_format() when they are constructed, so set that first too if you want channels_first; processors take a per-instance data_format argument.

import os

os.environ["KERAS_BACKEND"] = "jax"  # or "torch" / "tensorflow"

import keras

keras.config.set_image_data_format("channels_first")

Large checkpoints, as they ship

• GPT-OSS 120B loads at bfloat16 with its MoE experts left packed in MXFP4 and dequantized on the fly, so it stays near 66 GB instead of the ~130 GB an fp32 expansion would cost.
• Weight-only int8, int4, fp8 and mxfp4 are arguments to the same from_weights call, on any model.

from zeromodels.models.gpt_oss import GptOssTextGenerate

# Experts stay packed in MXFP4, dequantized in the expert layer's call
model = GptOssTextGenerate.from_weights("zeromodels/gpt-oss-120b")

# Quantize weight-only on the way in, for a smaller footprint again
model = GptOssTextGenerate.from_weights("zeromodels/gpt-oss-120b", quantization="int8")

Where to start

Vision

Detection, segmentation, depth, and self-supervised backbones.

Text

Encoders and decoder LLMs, dense and mixture-of-experts.

Multimodal

Vision-language generation and grounding.

Speech

Transcription and speech-aware language models.

Ready to use ZeroModels?

One install, 118 model families, three backends.

Connections: Managed credentials and per-caller identity for Managed Deep Agents

Key Takeaways

• Keep credentials out of your project. A connection lives in your LangSmith workspace, not in .env and not in the build. Rotate or revoke it without touching code or redeploying.
• Give each caller their own identity. A user-owned connection resolves to whoever is asking, so the ticket your agent files carries their handle rather than a bot’s.
• Skip the OAuth plumbing. Managed Deep Agents runs the authorization round-trip. No callback route, no token store, no refresh logic, no consent screen in your project.

Connections are available now in Managed Deep Agents v0.7.0+.

Every agent eventually needs to act on someone's behalf — search the web, file a ticket, open a pull request. Today that usually means one API key hard-coded across every deployment, and every action showing up under a service account. A key in .env answers what the agent may do. It has no way to answer who asked.

That is what Connections fixes. A connection is a named credential in your LangSmith workspace that your tools read at run time, by slug, through one call.

Two axes, not one

A connection has an owner and a credential type, and they are independent.

The owner is either the agent or the caller. An agent-owned credential belongs to the deployment, and every caller shares it. A user-owned credential resolves per person, at run time.

The credential is either a static secret or an OAuth grant: an agent can hold an OAuth grant, and a user can hold a secret.

Ownership is fixed when you create the connection using mda connections create, and connections.get() only selects among credentials that already exist.

Agent-owned secret

An agent-owned secret is used in situations where you need one credential shared by every caller. This is the right approach for a capability that does not differ per person: web search, a geocoder, a pricing feed.

In this example, let’s configure a connection to Tavily to add a generic web search tool to an agent:

uv run mda connections create tavily-agent --secret-from-env TAVILY_API_KEY

tavily-agent is the slug. It is your name for the connection and the name your code uses, and nothing checks it against a provider list. The value came out of TAVILY_API_KEY and went into your LangSmith workspace. It is not part of the build, and mda deploy does not sweep it in the way it sweeps .env into deployment secrets.

The tool that reads it is an ordinary LangChain tool with one new line leveraging connections.get:

# tools/search_web.py
import httpx
from langchain.tools import tool
from managed_deepagents import connections

@tool(parse_docstring=True)
async def search_web(query: str) -> str:
    """
    Search the web.

    Args:
        query: Search query.
    """
    api_key = await connections.get("tavily-agent", {"type": "agent"})
    async with httpx.AsyncClient(timeout=30.0) as client:
        response = await client.post(
            "https://api.tavily.com/search",
            json={"api_key": api_key, "query": query, "max_results": 5},
        )
        response.raise_for_status()
        return response.text

If you need to rotate your key, you can update the secret stored at tavily-agent, and any future agent requests will automatically use the new key.

User-owned OAuth, with your own app

Shared tokens are useful, but allowing your agent to act on behalf of your users means you can securely provide more capabilities to your agent. GitHub ships in the connections catalog alongside 22 other services, so you bring a client ID and a secret and nothing else — no authorization URL, no token URL, no auth method to look up.

You can quickly reference the catalog connections with mda connections catalog, but you can connect to any provider which offers OAuth if you bring your own metadata.

For example, to configure a connection to a custom Github OAuth app:

uv run mda connections create github-issues \
 --oauth github \
 --client-id "$GITHUB_CLIENT_ID" \
 --secret-from-env GITHUB_CLIENT_SECRET \
 --scope repo

In this example, github-issues is the slug, which is yours and which your code uses. github is the catalog service, which only decides which endpoints get filled in.

• -scope repo replaces the catalog default rather than adding to it. GitHub's default is read:user, which cannot open an issue, so whatever you pass becomes the whole list.

The tools read the token through a helper. In this example, the key line is:

access_token = await connections.get("github-issues", {"type": "user"})

With a single call to connections.get, a deployed agent can automatically either invoke an OAuth flow for a new user or fetch a cached OAuth token for a user who has previously authenticated against the OAuth provider.

We can leverage this access token to make arbitrary API calls to Github:

# tools/github.py
async def _github(method: str, path: str, **kwargs) -> dict:
    access_token = await connections.get("github-issues", {"type": "user"})
    async with httpx.AsyncClient(timeout=30.0) as client:
        response = await client.request(
            method,
            f"{GITHUB_API}{path}",
            headers={
                "Authorization": f"Bearer {access_token}",
                "Accept": "application/vnd.github+json",
                "X-GitHub-Api-Version": GITHUB_VERSION,
            },
            **kwargs,
        )
        response.raise_for_status()
        return response.json()

Notice that we set {"type": "user"}. The agent-owned connection stored a value at create time. This one stored no value at all, only the app registration. The credential arrives per caller, at run time — and if the caller has never authorized GitHub, or their token has expired, connections.get() pauses the run and asks for a grant instead of failing.

That word appears once, inside _github. A search_issues tool and a create_issue tool both inherit per-caller identity from the helper, and a third GitHub tool would cost no auth code at all.

The payoff shows up in two places. search_issues already differs per caller before anything is written, because private repositories one person can see and another cannot change the results — same query, same deployment, different answers. And when create_issue runs, the issue lands in GitHub opened by the person who asked. user.login in the response is their handle, not a bot's.

User-owned OAuth, with no app to register

Some MCP servers register the OAuth client themselves. When they do, the whole setup is a URL.

uv run mda connections create linear-mcp --mcp https://mcp.linear.app/mcp

# tools/mcp.py
from managed_deepagents import connections, define_mcp

mcp = define_mcp(
    servers={
        "linear": {
            "transport": "http",
            "url": "https://mcp.linear.app/mcp",
            "connection": connections.get("linear-mcp", {"type": "user"}),
        },
    },
)

No client ID, no client secret, no app registration. Because the server advertises its OAuth metadata and a client gets registered for you, you don’t need a scope either, and the connection came out with read and write on it, negotiated from the server's own metadata.

Compare that with the GitHub flow: one needed your own app and one needed nothing, and the line of code reading them is the same. The tool code is the part that disappears here — GitHub took a helper and two functions, this takes a server URL, and the tools arrive from the MCP server.

One pause, every missing grant

Ask an agent for something that spans both services and the run pauses before the first model turn, with a single interrupt listing every connection the caller has not granted. Authorize them and the run resumes where it stopped.

There is no callback route in the project, no token store, no refresh logic, no consent screen. The caller never opens LangSmith.

Do the same thing as a second caller and you get a second issue with a different author, from the same agent, the same slug, and the same workspace entry. Contrast that with the Tavily key, which is the same for everyone by design. You can check the connections you or other developers added to LangSmith with:

uv run mda connections list

Getting started

Connections ship in the Managed Deep Agents prerelease, and the OAuth catalog ships inside the binary, so the version you have decides what --oauth accepts:

uv tool install managed-deepagents
uv run mda connections catalog

An agent-owned credential belongs to a deployment, so scaffold and deploy once before creating one. After that, each connection is three steps — create it, read it with connections.get(), redeploy to ship the code that reads it.

Local development works the same way. Agent-owned connections resolve from MDA_DEV_<SLUG> in .env, uppercased with hyphens as underscores. User-owned connections resolve the signed-in developer to a real principal under mda dev, so the authorization interrupt fires locally and the grant it stores is a real one.

Beyond the three flows above, --authorize stores one OAuth grant for the deployment, so every caller acts as a single shared account — the fourth cell of the owner-by-credential model, and the right answer when you want a dedicated team account rather than per-person identity. --allowed-scope caps what later authorizations may ask for, and --authorize-url with --token-url covers any provider outside the catalog.

For more detail and examples, documentation for Connections can be found at https://docs.langchain.com/langsmith/python/managed-deep-agents-connections

I Never Want to Use Third-Party Software Again

Welcome to the era of digital hyperpersonalization

i.

In 1975, Andy Warhol made the observation that “a Coke is a Coke, and no amount of money can get you a better Coke than the one the bum on the corner is drinking.”

Marc Andreessen referenced this in 2023 and threw in an extra line: "Same for the browser, the smartphone, the chatbot!"

That’s kind of cool in some ways, right? The richest and most powerful people in the world are opening the same Slack as we do. They’ve got the same smart phones. Their Duolingo owl sends them the same guilt-tripping Your streak is in peril! notifications.

Sure, with more money, you can buy a private chef, a private jet, a private yacht. But when you sit down at your laptop, you’re using roughly the same software as every other fresh-faced college intern.

That was before. Going forward, I no longer believe this holds.

ii.

Let me back up a bit.

If you have multiple young kids, like I do, you will be familiar with famous challenge known as the “Mommy, who do you love more?”-barrage. After copious trial and error, I have learned that the best method is to sidestep the question and actually answer a different one: tell each what I love about them specifically. It seems to work: after all, we humans love what’s unique to us. In 1959 a psychologist named Neville Moray played two streams of speech into people's ears. They were told to just focus on one of the streams, but a third of them still heard their own name in the stream they were supposed to be ignoring. Dale Carnegie once wrote that a person’s name is, to that person, the sweetest sound in any language.

Personalization has been commercialized ever since. Coke, if you can believe it, had a period where they printed your first name onto their bottles. Nike lets you pick the color of your own swoosh; Louis Vuitton advertises more than 200 million monogram combinations.

And it’s not just names and colors; entertainment is now personalized to your tastes. In the 1980s, a single show could still command over 70% of the TVs in use. Today, streaming is almost half of all TV time and sliced into the thousands of niches that make up the long tail. A good three-fourths of YouTube watching starts from a personalized recommendation.

AI knows us better than ever. That trend is only continuing.

iii.

And yet, most software today, with the exception of personalized social, feels rather uniform. Duolingo, with its 58 million daily users, personalizes how hard the next sentence is, but not what those sentences are about. My seven-year-old loves the app but let me tell you, he is never going to tell someone “My, your kitchen looks so lovely!” in any language.

Where software played with personalization was largely in the settings menu. We got dark mode, color themes, maybe a font size slider here and there. But settings operated like a rule book. Somebody decided in advance the small handful of things one would be allowed to change. Designer Jared Spool once asked several hundred people to send in their Microsoft Word settings files and discovered fewer than 5% had changed a single setting. And if what you wanted wasn’t in there, you’d send a support ticket or beg for it on the forums.

This model made sense because software used to be expensive to make, so it had to be made for a lot of people at once. Clay Shirky wrote in 2004 that building an application for a few dozen users was "an absurd target population." Twenty-one years later, it’s time to rethink what’s possible.

iv.

Here is the story that changed my mind.

I run a lot of AI agents at once, usually across multiple projects both personal and professional. I work in the command line for the flexibility and power of it. My typical set-up is eight terminal windows open on my home machine, and I arrange them by hand like a poor game of Tetris. While on the go, I used an app called Mosh to control my terminal windows remotely, but the entire setup was six clicks deep. Meanwhile my husband was using the remote control version of Claude and Codex, and I jealously watched as he attached screenshots and two-tapped between his agent tasks. I was gearing up to trade my homemade setup for the more polished third-party version, despite having to give up some of the flexibility of my setup.

Then a thought crept into my mind: could I build the thing I was envying?

Half an hour later, I got a mostly-working version that answered my question with a resounding yes.

With the core questions answered, I spent the next few days elatedly designing for myself, adding in little moments of delight that would make sense to very few others.

I wanted to know how many subagents were working at a time by looking at the number of sparkles. I reinstated Claude’s status verbs with my own twist. I wanted more clarity on what subagents were doing. I called this app Mog Squad. If you know, you know, kupo.

After this, I thought to myself, I never want to use third-party software again. I don’t mean it boastfully. I don’t think my version is better than the official apps. But it works better for me.

For one thing, it lets me use both Claude and Codex, and neither official app does that. It’s hyper-optimized for my workflow, with my eight agents arranged in a grid. (Why eight? Because it’s the ideal number for me. Fewer, and I get the itch to start something new. More, and I start losing track of ongoing projects.) It has none of the bloat from features I don’t use. Because I created it, I know exactly how every button and drawer and keyboard shortcut works. When I find a little friction point, I remove it. When I get a new idea, I implement it. The feedback loop could not be more efficient. Because I am my best customer, I care immensely about making myself happy. I will never complain about my own software!

Economists even have a name for this feeling, the IKEA effect: we love the things we assemble ourselves more than the ones that come pre-packaged for us.

v.

I’ll share one more example of hyperpersonalization.

My kids love Duolingo, so over the summer I built them a Duolingo-inspired math and reading app.

Now, like many kids, my kids are not especially motivated to answer math and reading questions. And I didn’t want to resort to state-of-the-art gamification techniques like streaks or “you got a 2x boost for the next 15 minutes!!!” So I tried a different tack: the stories would be about their lives: their real friends, their real life news, and their real hobbies. My daughter’s stories are about fashion and theater. My son gets Minecraft, Geometry Dash, and a large dose of Pokemon. When we take a trip, those stories show up in next week’s lessons. And just as I used to sneak veggies into their meals, I make sure the stories carry a sprinkle or two of our family's values in what the characters choose to do.

There is a “feedback” button at the end of every lesson so the kids can tell me if it was too easy or too hard, or if they want a particular story (my daughter has used it more than once to request a story in which she meets and gets to sing with Idina Menzel.)

One request was not about the stories at all but about the rewards. They wanted something to collect. So I made holographic cards of their favorite stuffies. The rarer the card, the shinier it is when you tilt it!

There is some research beneath this: in 2013, a study gave 145 ninth graders algebra problems. Half were standard and half were rewritten around each student's own interests. The personalized group solved them faster and more accurately, with the gain being largest for the students who were struggling. Best of all, it stayed after the personalization was taken away!

This makes sense to me: even as an adult, I pay more attention and do better work on the things I’m actually interested in.

vi.

So what did I learn from these experiments?

Hyperpersonalization is the next great trend. And there are two major reasons for it:

The first reason is workflow utility: anything you do fifty times a day is a place where friction compounds. Two clicks instead of one. A menu with 20 items rather than 2 that your eyes have to process. Two screens you need to flip through to remember the context. When I wrote last week about how to pull off an AI transformation, I made the point that workflow improvements are great early wins. And workflows are deeply personal, influenced by our unique brains (are you an audio or visual learner? Do you like tl;dr or details?), lifestyles (are you constantly on the go, or do you sit in front of a computer all day?), and the constellation of tools at our disposal. Even where best practices exist for a single tool, there are hairy seams between tools because vendors are not typically incentivized to make sure every possible combination of tools works beautifully. The only people who can describe a perfect workflow are the ones living it.

The second reason is expression. I have a closet full of shirts that perform the same utility in keeping me warm, but I’ll still have favorites that I think express myself better. Software has tended towards utility: most software (especially enterprise software) is bland, most UIs converge (can you even tell the difference between the main chat screens of ChatGPT or Codex or Gemini?), and I get it, bland is safe, bland is least-likely-to-offend when you’re creating for millions. Turn your agents into a flock of winged cat-creatures and a lot of people will reasonably be like WTF? But hey, it puts a smile on my face. And I’d hope your own software tickles you too.

I think this is where interface design goes next. When I wrote about conversational interfaces last year and talked about the golden opportunity of personalization, I was thinking from the perspective of a company learning you well enough to customize what it shows you. I now think the inverse is also true: enabling you to tune the thing for yourself, whether that’s UI or content.

A lot of people will not do this, at least for some time, and that’s fine. Just like not everyone cares about fashion or eking out that last ounce of optimization, not everyone will care about perfecting their workflows or expressing themselves through software.

But enough will. And no software is safe from the personal remixes to come.

vii.

I think about software now in levels.

At the left are the use cases. This is where most software lives today, because use cases are tangible. An app that creates photo collages or helps you tune your guitar or lets you watch movies solves for very specific problems that need only a yes-no answer: Does this solve a problem I care about?

One level up from that is a platform builder like Shopify, Webflow, Roblox or Lovable, that helps you create a personalized version of a certain type of thing: whether an online store, a website, a game, or an app. Here, you need some imagination to picture how it might look and feel, and what functionality is important. But there are still guardrails around how much you can build or change.

One level beyond that is the anything builder: a coding LLM with a full set of tools and permissions to create whatever your mind conjures up.

Each step from left to right requires more imagination and delivers more freedom. The far right is where few live today. But slowly but surely, everything to the left will start shifting more and more to the right. That’s why the Replits and Lovables of the world are growing the way they are. If operating systems are smart, they’ll get in on this. It’s hard to imagine that an AI-first operating system of the future will look like a grid of apps.

If you build software today, I’d love for you to consider four things:

• Make your interface malleable. A UI should be something a person can take apart and put back together in the way that best suits them.
• Let preferences reach past the rule book. A descriptive vision is worth more than fifty toggles.
• Let feedback change the product. Give people a chance to tell you what they really want, and immediately make it so.
• Treat the seams as part of the experience. Nobody lives in just one tool, so a better workflow is a more integrated experience across the tools a person already uses.

viii.

Cokes remain popular today. They are still cheaper and more efficient than making your own drink. Perhaps one day, when personal robots can prepare ambrosia to our specific tastes, this will no longer be true. Until then, we’ve got software to play with. Happy remixing.

The Apple Watch Series 12 and Ultra models have new Audio Intelligence tools that use ambient listening to take high-level notes of wearers' conversations throughout the day. The devices will also have a Live Recap function that can show a transcription of what was said in a live conversation in the last 15 seconds and be able to listen to important sounds, such as a crying baby, doorbells, or sirens. These features will not create or store recordings, and Apple will not be able to access the raw audio. They will be available to test in English later this year. The improved Siri is being rolled out on watchOS 27 in beta on September 14.

Multimodal models need video. We froze time to give it to them

Contents

• A session replay might look like a video, but it's not (and we needed it to be)
• The obvious way to build a rasterizer
• Two processes, one clock
• Solving our problems by manipulating time
• Roads? Where we're going, we don't need roads

Almost none of the session recordings we ingest are watched. The majority are stored, never opened, and expire without receiving any attention.

It's no wonder, really: a moderately busy product results in tens of thousands of recordings per day. You'd need an army of humans doing nothing but watching to get through them all.

As a result, most of what the Session Replay team has shipped over the past couple of years is some form of search: filters on events and properties, relevance sorting, collections, session summaries. All are attempts to help our users find the needle in the haystack and maximize the value they can get with the limited amount of human attention available to them.

Replay Vision started from the thought that we should stop rationing attention and start automating it. Recent multimodal AI models are good enough at watching video that you can hand them a recording and get a sensible account of what the user did and where it went wrong. Apply this to session recordings, and you get details on all of them, not just the ones someone had time for.

This post covers the first big problem we hit building it: models couldn't actually see a replay.

A session replay might look like a video, but it's not (and we needed it to be)

A replay looks like a screen recording when you play it, but there's no video anywhere in the pipeline. What our SDK captures, using rrweb, is a serialized copy of the DOM at the moment recording starts, and then a stream of events describing what changed:

• This node was inserted under that parent
• This attribute changed value
• The mouse moved to these coordinates
• The page scrolled

When you press play, a browser rebuilds the DOM from the snapshot and applies the mutations one at a time, with the page's real stylesheets, using the browser's real layout engine.

Handing this data to an LLM proved fruitless. It is verbose, often many megabytes of JSON per recording, and even the most recent models have limited context windows, so they simply cannot hold a whole recording in context at once.

The bigger problem is that the model has no idea what it's looking at. An rrweb stream is a changelog against a tree. To know what the screen showed at second 42, it would have to hold the whole tree in context, apply every mutation up to that point, and then run layout and paint as well. The model will happily tell you that a div gained a class called is-disabled. It cannot tell you that the checkout button just went gray and slid underneath the cookie banner, because that fact only exists after layout has run.

Modern multimodal models like Gemini need actual video in order to make sense of a recording. A good mental model for this is vector versus raster graphics. The rrweb data is a series of instructions that lets the browser perfectly recreate the original session, in the same way an SVG is a mathematical description that lets a renderer recreate an image.

What we needed was the rasterized version of the recording: an MP4, a compressed stream of pixels. Which meant the first piece of Replay Vision was a rasterizer. rrweb goes in, an MP4 comes out. Load the player in a headless browser, record the screen, encode it.

The obvious way to build a rasterizer

The rasterizer would need to handle a large volume of recordings to keep up with what we ingest. The service consists of a number of moving parts:

• A very small frontend bundle that can play back an rrweb data stream, built specifically for video capture
• A pool of running headless Chromium instances with a warm browser cache
• A Node.js process controlling the Chromium instances via Puppeteer
• A Temporal task queue to manage incoming workloads and retry on failures
• Dedicated infrastructure to run all of these components

Upon receiving a new task, the service would load the frontend bundle in one of the browsers. The browser would download the recording data from one of our internal services and play it back at 8x speed, while Chrome pushed a screencast frame to the Node.js process over the Chrome DevTools Protocol (CDP) every time it repainted. The frames were piped into ffmpeg, which encoded them into a single MP4 file.

The rasterizer worked well during initial testing, but once we started running multiple instances of the service on the same hardware, problems began to appear.

Two processes, one clock

We saw videos that skipped over sections of the recording, and others that froze for several seconds at a time. The same recording would often produce videos of different lengths.

It became clear that the rasterizer was very sensitive to CPU contention. Essentially we were running two independent processes, one rendering the recording and one capturing the frames. If rendering slowed down because another process was taking all the available CPU, capture would keep streaming on schedule and ffmpeg would just repeat the last frame it was given. That's the freeze. Or conversely, capture could be starved while rendering continued, and eventually frames would pile up and some would get dropped. That's the skip.

We needed a way to ensure determinism across these processes, regardless of the available resources. The same input data should always result in the same output data.

Solving our problems by manipulating time

Another way to frame the problem is time. The capture process expects frames to arrive in real time, and the render process expects to be captured in real time. Each depends on the other keeping up. We were hoping these processes would always stay in agreement, but on a modern preemptive OS this was a fool's hope.

Regardless of CPU contention, we need these processes to wait for each other and only progress in lockstep. Turns out Chrome has a mode for exactly this scenario: HeadlessExperimental.beginFrame. It stops the compositor from producing frames on its own. Instead, it waits for an explicit beginFrame command over CDP, renders exactly one frame, and returns once that frame is complete.

In this mode we can ensure that Chrome does not render the next frame until the capture process has taken the previous one and added it to the ffmpeg stream. In case of CPU contention, rendering would simply halt until the capture process is ready for another frame.

However, the JS-based rrweb player does not wait for the browser. If the browser stops invoking the requestAnimationFrame callbacks as often, the player compensates by skipping forward based on the time passed since the last tick. We need the rrweb player to believe it is running in real time, and only ever advance it by exactly one frame interval per tick.

The trick is to freeze time inside the JS virtual machine. Before playback starts, we override Date.now, performance.now, requestAnimationFrame, setTimeout and setInterval inside the page with versions that don't track the wall clock at all. Time only moves when we call beginFrame, and it moves by exactly one frame interval. The player asks for an animation frame and gets one, it reads the clock and sees it ticking forward at a steady rate, and it never notices that a single tick might have taken 20 milliseconds or two seconds of real time to produce.

Roads? Where we're going, we don't need roads

With beginFrame gating the compositor and the virtual clock gating the player, the two failure modes from earlier simply stop existing. If rendering is starved, the clock waits for the frame. If capture is starved, the renderer waits for the next beginFrame call. Either way, the next frame in the video is the next frame in the recording, and CPU contention only changes how long the job takes, not what comes out of it. We can run as many jobs per pod as we like, and the same recording produces the same video every time.

The version of our rasterizer described in this blog post has been in production since March 2026. It runs as hundreds of concurrent jobs across many pods and powers all of Replay Vision, as well as some features in Session Replay itself. At the time of writing it has rasterized more than 370 years of recordings into roughly 3.5 million videos.

The rasterizer was the unlock we needed to get recordings in front of a multimodal model at all. Getting useful answers out of the model turned out to be a separate problem: what to show it, what to leave out, and how to stop it from fixating on irrelevant details. That's a post of its own, coming soon.

Prompt Evals Alone Are Useless

Over the last couple of nights I have been building a text adventure RPG, mostly to see what Astra could do. I have been giving the AI an end of day prompt, leaving it to run for hours, and seeing what it has done in the morning.

Playing the game is simple. It is told through a diary style interface, almost like reading an ebook: you type your character’s actions, you roll some dice if needed, and the AI narrates what happens with little illustrations, keeping notes and weaving a story as it goes. I will write more about the game itself soon. This post is about how I am building an eval system to improve it.

At the moment, the game is fun but inconsistent. The diary drifted out of tense, a safe action was treated as if it were risky, and the town in the prologue felt deserted. I noticed all of them as a player, and my instant thought for fixing them came from Kaijo, the AI email product I built to fix unreliable AI, and from the eval framework I described when I wrote up how to build a robust LLM application: save the example that went wrong, have an AI judge score it, rewrite the prompt, and try again.

It has grown into much more than that.

Astra built the prompt evaluation layer first. If I want to complain about what happened, a little flag button and a dialog let me save the complaint with the exact request we sent to the model. Astra can then pick it up in development, and a replay command sends that same request again with a changed prompt so I can compare the two answers. The agents rebuilt my three complaints as the first saved cases.

Then Astra started fixing things. The interesting part was that it was not just fixing the prompts. Almost every fix changed the shape the model has to fill in when it answers, the code around it, or the record of the story so far, and only a few lines of instruction changed with them. A prompt eval, a test that checks a prompt on its own, cannot tell me whether this game works, and neither can the ordinary tests AI writes for the code. I need to stress the whole system at once.

The Faults Were Not in the Prompts

The first fix came from a storm. The rules stopped my character setting out in foul weather, which was right, but the story then said “The attempt consumed the phase”, which is the game’s internal bookkeeping leaking into the prose. Every mechanical check passed, and Astra still rejected the run because the story was wrong. Astra changed the required answer format so that every story paragraph must name the game event it describes.

The second fault was a loose end in the story. Early on, a character called Mara lent me a cup, and the game is supposed to remember that I owe it back until I return it. If my character carried the cup around or checked their kit, the game counted that as progress towards returning it, even though nothing had happened. The fix was one sentence beside the answer format: “Continued possession, routine use, cleaning, checking equipment and reminders of a promise are not progress.” To place that sentence, Astra had to trace how the game recorded progress on the debt. That needed a whole new subsystem, with ordinary tests of its own, and also an end to end check to make sure the whole thing worked.

On an ordinary turn the game sees only the last three passages of the story, and everything older lives in a ledger, a running record the code keeps so that loose ends like the cup survive. The code also rolls the dice, and a separate check looks at every picture before the player sees it. What the player reads is never the prompt on its own. It is the prompt, the answer format, the dice, the ledger and the picture check all working together, and a replay of one saved request can never tell me whether they did. All of that surrounding software is the harness. If the harness decides whether work lands, the eval has to test the harness. So I stopped treating evals as rows of data and started writing them as code.

All My Evals Are Code

This feels to me like an extension of Behaviour Driven Development, which I wrote about at length in earlier days, driving the whole system from the outside with end to end scenarios, only here the thing being checked is far more subjective and nuanced. So Astra and I drew on this and started putting a new system together.

I organise my tests and my evals into one pyramid. Ordinary software tests sit underneath, covering the game rules, the dice and the saving of progress, so the evals above them only ever have to question the model and the harness around it. The replay cases, which only check prompts, sit in the middle. Each saves the exact request and the model’s answer, and the replay command sends that request to the model again, refusing to run if the game has changed since the case was saved.

At the top sit scenarios written as code that test the whole game. Each one specifies the action the player types, a sentence saying what should happen, the starting conditions, the dice results to fix in advance, and the checks to run afterwards. Each scenario runs the game, calls the model and saves progress into a temporary copy of the database, with pictures switched off because the picture check has its own evals. I fix the dice because the model’s answers already vary from one run to the next, and adding the game’s own randomness on top would make a run impossible to judge.

{
  action: "Lost in thick mist, I trek east from Whitebank toward the mapped land there, accepting that I may drift off course.",
  purpose: "The interpreter declares an eastward Trek and explicit blind navigation; held die 2 resolves the left-hand actual edge, whose encounter state drives the continuation.",
  validateInterpretation(interpretation) {
    assert(interpretation.travel?.navigation?.lostOrMist === true, "The explicit mist was not classified as blind navigation");
  },
}

The purpose sentence tells whoever reads the result what to expect: the player heads east, accepts they might get lost, and drifts the way the fixed die says. The checks then inspect what the game recorded, and a report keeps those results beside the story the model wrote, so the reason for running the case survives with the output. Even a case with every mechanical check passing still needs someone to read the story.

The Agent Is the Judge

The combat report leaves the story column marked “Not graded; coordinator inspection is required.” and hands it to the coordinator, which is Astra in Codex or Fable in Claude Code. The coordinator compares the story and the recorded results with the expected outcome, then traces any mismatch through the saved game, the answer format and the code to find where the fix belongs. The cause can be an event several turns earlier, which is why the judge also needs a longer session.

This morning I needed to know whether the AI checking the game’s maps was judging them properly. Astra made me a review form with the whole map and enlarged regions to inspect. It asked me to describe what I could see before revealing the geography the game required, then kept the earlier AI judgements hidden until I had made my own assessment.

I found this extraordinary. Astra had made me a form to help it work out whether it was judging another AI correctly. The AI spotted the problems but did not know how to fix them, so I gave it some pointers from my game-development experience. I have never worked in such a light-touch, high-level way before.

The Long Session Is the Real Test

The model’s memory is trimmed as a story grows, and that trimming can lose the details that make a story worth following long before any single reply looks broken. One six turn scenario tests whether the ledger keeps the cup debt alive after the loan drops out of the three passages the model can see. The game must remember the debt until the cup is returned, and that test is what exposed routine actions being counted as progress.

A separate Day 120 case checks whether the game still remembers unfinished business from day one when the three recent passages have nothing to do with it. Once the agent can judge a whole session, the screen is the last part of the player’s experience left outside the loop.

Right Through to the Screen Itself

The agent drives the browser and takes screenshots of the whole user experience as it works, noting page errors, failed requests, the text on screen and anything that spills outside its box. Reading its own screenshots lets it inspect what the player sees, so there is no separate automated screen test. With Fable and Astra, this loop has finally closed for me. Once the agent has had a good go at the screens, I try them, and they mostly work first time with a few tweaks.

This builds on how I use galleries to give UX tickets a visual specification, which gives the agent something concrete to aim at and inspect. My feedback now leads through a code change, a scenario, a session and a browser check before I try the result again, so the eval covers the whole path from a complaint beside Go to the screen where I can see whether the fix helped.

What Evals Are Becoming

The system improving this game now changes code, tools and answer formats, along with memory and screens, guided by failures from whole runs. Research projects such as Self-Harness and Weco’s AIDE2 test the same idea, one agent improving another by changing the software it uses. Andrej Karpathy’s autoresearch runs a similar loop against a single number, and Shopify has generalised it into pi-autoresearch, which will chase any metric you can measure. This game has no such number. Whether a story is good is a judgement, so the judge has to be an agent reading the whole run. The whole eval setup came together in about two days, with me directing Fable and Astra and their delegated workers doing the building.

A system that improves itself needs guardrails, and it needs to run in a sandbox. Otherwise, as the comic shows, you never know what your agent might do while you are out of the room. Every scenario here runs against a temporary copy of the database, and since the second evening every agent reruns the saved scenarios and reads the results before handing work back. The cases live with the game code, where the agents can read and change the tests too.

For your own product, take a complaint from using it and write down what a run that reproduces it should show. Give that run enough of the real system to reach the fault, preserve its failures, and let the coding agent read the output alongside the code. Keep replay cases where a single call answers the question, then extend the scenarios through lost memory and onto the screen. I will keep playing the game and pressing the flag when something feels wrong, giving the agents another concrete example of what good should look like.

1. The storm fix had three parts: every story paragraph must name the game event it describes, the model provider enforces that requirement through the answer format, and one prompt instruction bans the bookkeeping phrase. Astra then reran the earlier scenarios and read the results to make sure nothing that used to work had broken, and the three failed attempts stay on record beside the passing one so the history is not lost.

2. On an ordinary turn the model also sees a list of established facts and a repeatable selection of unfinished storylines, including ones that have not appeared recently. The ledger holds a record for each thread as it is set up, progresses, pays off and closes, with a short summary and a quote copied by code from the passage it came from.

3. Dan North’s original introduction to BDD describes writing each test as a sentence about behaviour, driven from the outside of the system in. Every scenario in this post follows that shape, with a model in the loop.

4. Evals like these need habits that ordinary tests do not. If the model provider fails mid run, the run says nothing about quality and must not be counted as a pass. Failed attempts stay on record, because rerunning until one passes would hide the behaviour I am trying to see. Changes to saved cases are recorded, and tests of the whole game need updating more often than prompt tests because they depend on more of the software.

5. Self-Harness, by Hangfan Zhang et al., describes agents mining failure traces, proposing minimal harness edits and accepting them only after regression tests. Weco’s AIDE2 write-up describes an outer loop rewriting the inner agent’s harness, including its search policy, context compression, an evaluation bug fix and reward hack reduction. Both sets of results are self reported by their authors.

6. Andrej Karpathy’s autoresearch gives an agent a small training setup, a five minute budget per experiment and one metric, then leaves it overnight to keep the changes that improve the score and discard the rest. Shopify’s engineering team generalised the loop into pi-autoresearch, an extension for the pi coding agent that works on any measurable target, and reports using it on more than forty metrics across the company.

Tom Lane, a committer for 25 of Postgres' 30 years, walks through the architectural bets that held: process-per-connection for code simplicity, write-ahead logging in 8.0 that turned it from a plaything into a serious database, and MVCC pushing maintenance work into background vacuum. He credits Berkeley's liberal license for the project surviving where its contemporaries did not, names version 13 the buggiest release, and says he would delete partitioning given the chance.

Cloudfloe is a web interface I made a while back for querying Apache Iceberg data on S3 with DuckDB. A user saves a connection, writes SQL, runs it, and can see the results. In the hosted version, a query session starts a container containing native DuckDB to run the queries. It’s not overly innovative but it works well.

As an experiment I wanted to try running DuckDB inside celld and see if there were any interesting results, knowing that it’s probably not a typical use case for celld.

Celld is a self-hosted implementation of Cloudflare Workers and Durable Objects. A named Durable Object is a cell with its own SQLite database. Application code runs in a V8 isolate. Long-lived state is replicated through an object store, which also acts as the coordination layer for ownership and failover.

At first glance, celld sounds like a faster place to start a query worker. That was the hope anyway. Celld does not run containers. It can’t start the native DuckDB binary that Cloudfloe uses. Trying celld means compiling DuckDB to WebAssembly and running it inside a single-threaded V8 isolate. That’s how it started. A number of interesting things cropped up from there.

DuckDB was already halfway there

I started with Ducklings, a compact DuckDB WebAssembly build for browsers and serverless runtimes. It runs without threads and uses Asyncify to let DuckDB pause for an asynchronous fetch() and resume when the data arrives.

The part I expected to get stuck on was Iceberg. Reading Parquet files is only part of reading an Iceberg table. Something also has to interpret the table metadata and Avro manifests to work out which files belong to the snapshot being queried.

I thought I would have to compile DuckDB’s Iceberg extension into the module myself, or do that metadata work in JavaScript and hand DuckDB a list of Parquet files. But the Ducklings Workers package I checked already included httpfs, Avro, and Iceberg. Someone had done the first option already.

That gave me a much shorter route to the first test: could celld load this build and run a query?

Before getting to Iceberg, I needed to know whether the runtime would accept a database engine this size. The DuckDB WASM file alone was 42.4 MB uncompressed. These are deployment sizes, not the memory DuckDB needs to run a query. A bundle-size limit could stop the experiment before it started.

Celld loaded the module, and DuckDB returned its version and a SQL result. That answered the first question: the existing build could run. Next up was a query against an actual Iceberg table.

A real Iceberg query worked

I created a 37,537-row Iceberg v2 table with PyIceberg and stored it in MinIO. There was one small setup issue: DuckDB expected a version-hint.text file when I gave it the table directory, but PyIceberg had not created one. Giving DuckDB the exact location of the metadata JSON file fixed that.

From there it worked. DuckDB inside the celld cell read the table metadata, decoded the Avro manifests, read the Parquet data, and returned the correct row count. The count query took 148 ms, with celld and MinIO running on the same machine. Cloudfloe’s basic query path was now running inside a celld cell.

An equality-delete test returned the expected remaining rows too. That gave me a little more confidence than a count alone, though there were still Iceberg features I had not tested.

The next question was how much time I was spending getting DuckDB ready to do any work. The 148 ms result alone would not tell me that. Later tests would show a trivial SELECT 42 taking over 300 ms in a fresh celld process, compared with about 3 ms when the cell kept DuckDB open. Getting the engine ready would turn out to be a large part of the wait.

What happens when the cell goes to sleep?

For Cloudfloe, there would be gaps between a user’s queries. I wanted to know whether a cell could go to sleep between requests and still behave correctly when it woke up.

First I tested what happened if celld was asked to evict a cell while DuckDB was waiting for data. I delayed a Parquet response for two seconds, then requested eviction while the query was still running. The query finished correctly in about 2.1 seconds. Celld let it finish before putting the cell to sleep.

After the cell had hibernated, I sent another request. Celld reconstructed it, restored a request counter I had saved in its SQLite database, and ran the DuckDB query again. The saved state survived, and the query worked after waking.

That part worked. The next thing to check was how much memory going to sleep actually gave back.

Putting the cell to sleep didn’t give all the memory back

I used ps to measure the celld process’s resident set size (RSS), the amount of its memory currently held in physical RAM. This measures the whole process, including the runtime and DuckDB. I recorded it before a query, after the query, and after hibernation, then repeated the test in a fresh celld process. I also checked celld’s internal state to confirm that the cell and its isolates had been released.

The rounded numbers I noticed were:

• about 140 MB for the clean celld process in this setup;
• about 230 MB after its first DuckDB query;
• about 95 MB added by that first query;
• about 210 MB after the cell and isolates were freed.

The cell was gone, but celld was still using about 75 MB more RAM than it had before the first query. That does not necessarily mean a memory leak. A process can hold on to memory for reuse rather than return it to the operating system immediately. What I could see in both runs was that putting the cell to sleep did not bring memory usage back to where it started.

Inspecting the Ducklings build also showed a separate constraint: DuckDB’s WebAssembly memory was capped at about 134 MB, with no disk fallback for queries that needed more working space. Raising that limit would require rebuilding Ducklings. That made the amount a query held in memory more important than the table’s total size, but I had not tested where different queries would hit the limit.

Putting the cell to sleep had released some memory, but the process was still well above where it started. On the next request, DuckDB would need to be opened again anyway. I wanted to measure how much that restart cost.

What does waking up a cell cost?

I used a query that just returned a number and read no tables. That meant the request could spend no time fetching Iceberg data, making it easier to see the cost of getting DuckDB ready.

The full request took roughly 320 ms in a fresh celld process. After putting the cell to sleep and waking it again, with celld itself still running, it took about 250 ms. Keeping the cell’s DuckDB database and connection open brought repeated requests down to about 3 ms. These were all local tests on the same machine.

I timed the startup stages separately to see where that difference came from. Waking a cell meant creating its runtime and opening DuckDB again. The SQL itself was only a small part of the work.

Keeping DuckDB open made repeated queries much faster. Putting the cell to sleep meant doing much of that setup again, even though the process still held on to memory. Could I initialize DuckDB once and use that as a starting point for cells waking up?

Could I make a cell template?

I was thinking along the lines of a custom Docker image. You install the dependencies once and use that image as the starting point each time. Could I do something similar for a cell, with DuckDB already initialized?

That would need to go a step further than preinstalled software. I wanted to preserve the work done when DuckDB starts, so a new cell would not have to repeat it. I looked through celld’s documentation and runtime source to see whether it could save and restore something like that.

Celld already avoids repeating one expensive step. A statically imported WASM module is compiled once per process and reused by later isolates. So the startup time I was measuring did not include compiling the 42.4 MB DuckDB module every time.

But I found no template containing an initialized DuckDB. Celld’s storage snapshots preserve SQLite state, not the JavaScript objects or WebAssembly memory of a running query engine. Restoring the cell’s saved data still left the runtime and DuckDB to be initialized.

There was another useful clue. A cell and an isolate are different things: the cell is the named object, and the isolate is the V8 runtime hosting it. Several cells from the same Worker can share an isolate. In one test, a new cell placed in an already warm isolate activated in about 1 ms. It still needed its own DuckDB database, but Ducklings was already initialized. Part of the starting point I wanted was already there.

That suggested a smaller experiment. When the last cell in an isolate went to sleep, could celld keep the empty isolate around for the next request? It would preserve the initialized runtime, though each returning cell would still have to open its own DuckDB database.

I made a local celld patch that kept at most one empty isolate per Worker. It reused an existing isolate rather than creating a snapshot or making copies of a template. I deliberately used a one-second idle timeout to make cells hibernate during the test; celld’s timed idle eviction is disabled by default.

The next request took about 145 ms with the isolate retained, compared with 254 ms when celld had to create a new one. Reusing the runtime skipped Ducklings initialization, though the reconstructed cell still had to open its own DuckDB database.

Keeping the isolate alive worked. It saved roughly 100 ms on the next request, but it also kept more memory in use. After the first hibernation, the process used about 243 MB with the isolate retained, compared with 218 MB when it was released. Repeated sleep-and-wake cycles pushed memory higher, though the short test was not enough to call that a leak.

So I had a useful tradeoff: faster wakes in exchange for more resident memory. Keeping isolates around indefinitely would need a memory budget, especially as the number of Workers grew. The next thing to try was keeping one alive briefly, then releasing it if no more queries arrived.

What if I kept a cell warm for ten seconds?

I changed the patch so an empty isolate could stay alive for ten seconds after its last cell went to sleep. That seemed like a useful window for someone editing SQL and running another query. If they stopped, celld could release the isolate.

This time I used the actual Iceberg count query. Each request opened a new DuckDB database, ran the query, and closed the database and connection afterward. A request arriving within the warm window completed in about 93 ms. After the isolate had been released, it took about 291 ms. The correct row count came back in both cases.

So the short window worked too. In this local test it saved about 200 ms on the full request. But an expiry time only answers how long to keep something when there is room for it. I also needed to know what happened when the node needed that memory sooner.

What happens when the node needs the memory back?

For a separate test, I deliberately lowered celld’s memory-pressure threshold to about 210 MB and extended the warm window to thirty seconds. This forced the memory policy to act while an empty isolate was still being retained. The first query succeeded, then celld put the idle cell to sleep and stopped admitting new work because memory usage was too high. My patch kept the empty isolate alive anyway. The next query waited until the client’s five-second timeout expired.

The isolate was eventually released, about 35 seconds after it became empty. Even then, the process held on to enough memory that celld stayed in its memory-pressure state. Twenty seconds later, there were no resident cells left to evict, but the node was still not admitting work.

The low threshold was deliberate. This did not show that a normal celld deployment would get stuck after one query. It showed that my retention patch and the node’s memory policy did not work together: the patch waited for its timer, and releasing the isolate still did not reduce memory enough for the node to resume.

Keeping the runtime warm had made queries faster. Making that useful needed more than a timer. Memory pressure had to be able to end the warm window early, and the node needed a way to recover or return a clear error when there was nothing left to evict.

I started this wondering whether Cloudfloe could run its queries in a celld cell. It could. What I hadn’t expected was to spend so much of the experiment looking at what happened between queries.

Keeping part of the runtime alive made the next query faster. Putting a timer on it helped, but the memory-pressure test showed why that wasn’t the whole solution. There seems to be something worth exploring here: keeping expensive runtimes ready while still letting the node recover memory when it needs to.

I’m leaving Cloudfloe’s containers in place for now. This was an experiment with a workload celld probably wasn’t designed around, and I had fun seeing how far it would go. Getting DuckDB to run was the beginning. Working out what to keep alive afterward turned out to be the interesting part.

Test setup and reproduction notes

These tests ran in September 2026. The timings describe this local setup, not production performance.

• Software: celld 0.4.0, Ducklings 1.5.5, and DuckDB 1.5.5. Initial tests used the official celld binary. Isolate-retention experiments used my local patches, not a built-in celld feature.
• Machine and data: Linux x86-64, Intel Core i7-13700H, 20 logical CPUs, about 15 GB RAM. Celld, MinIO, and the client ran on the same machine. The main fixture was a 37,537-row Iceberg v2 table created with PyIceberg; queries used its exact metadata JSON location.
• Deployment: Directly with celld deploy. The optional managed deployment route had a module cap of about 26 MB, below the 42.4 MB DuckDB module.
• Lifecycle settings: CELLD_IDLE_EVICT_S=1 forced frequent hibernation; timed idle eviction is disabled by default. My first patch used CELLD_RETAIN_EMPTY_CELL_ISOLATES=1. The timed version used a reserve of zero and CELLD_EMPTY_CELL_ISOLATE_GRACE_S=10. These retention settings belong to the experimental patches. Maintenance checked expiry periodically, so ten seconds was not an exact release deadline.
• Pressure test: A thirty-second grace and CELLD_MAX_RSS_MB=200 deliberately triggered pressure at about 210 MB. The policy required memory to fall below 80% of that threshold, about 168 MB, before accepting new work again. It did not reach that point during the observation period. This was a forced test, not the default configuration.
• Measurement: Client timings included routing, cell activation, initialization, and query execution. Separate stage timers and a random module ID distinguished fresh and reused runtimes. RSS measured the whole celld process. MB figures are decimal and rounded independently; the short runs do not establish long-term memory growth.
• Timed Iceberg queries: Every request opened and explicitly closed its DuckDB database and connection. Control and treatment used the same custom Ducklings build, with Arrow insertion support removed. That reduced raw bundle size by only 0.21%.
• Coverage: The v2 count and a separate equality-delete fixture passed. Positional deletes, v3 deletion vectors, time travel, and complex field-ID schema evolution were not verified. These tests do not establish a safe maximum table size.

CDC evaluation should start with failure behavior, not connector counts. Compare capture method, initial snapshots and backfills, schema evolution, delivery semantics, observability, deployment model, and who owns recovery. A production proof needs representative data and a broken-destination test, especially for type fidelity and duplicate handling.

Filament

Filament is pluggable data replication with checkpointing, batching, and integrity events.

It moves data from sources to sinks using full, incremental, or change data capture replication. Filament keeps progress durable as data moves, verifies batches on both sides of a write, and safely pauses or resumes runs from their checkpoints.

Sources, sinks, state storage, and event transport are replaceable interfaces. Run Filament as a service with its API and web UI, deploy it to your own infrastructure, or embed the engine in a Go application.

Install

curl -fsSL https://getgalaxy.io/filament/install | sh

On macOS via Homebrew

brew install galaxy-io/tap/filament

Prebuilt binaries for Linux and macOS are on the releases page.

Getting started

To run Filament locally, install Go, Docker, just, Node.js, and pnpm. On macOS, the repository's Brewfile installs the toolchain:

brew bundle

For Podman users, set CONTAINER_ENGINE=podman to use Podman with the commands below.

Start PostgreSQL and NATS, then run Filament:

just infra
just dev

Open http://localhost:5173 to use the web UI.

When you are finished, stop the local infrastructure with:

just infra down

Deploying Filament

Kubernetes with Helm is the recommended way to run Filament in production. See the Kubernetes deployment guide for installation instructions.

For a simpler hosted deployment without Kubernetes, deploy with 1 click:

Read the Railway, Render, and DigitalOcean guides for the deployed resources, runtime behavior, and production checklist.

Warning

The DigitalOcean deployment uses filament/standalone, an all-in-one image that includes the control-plane, server, and SQLite datastore.

How it works

A Filament pipeline connects a source to a sink and defines how selected resources should be replicated. During a run, Filament extracts records into bounded batches, writes and verifies each batch, and advances durable checkpoints only after successful work.

The runtime is event-driven: the server accepts pipeline and run requests, the control plane schedules and dispatches work, and workers execute individual runs. PostgreSQL stores durable state and NATS JetStream carries lifecycle events between components.

Developing Filament

Filament is written in Go, with a React and TypeScript web UI. Contributions to the engine, connectors, UI, deployment tooling, tests, and documentation are welcome.

Community

• Read the Filament documentation.
• Ask questions and meet other users in the Filament Slack community.
• Report bugs or propose features through GitHub Issues.

Why Spotify Is Not Using Bayesian A/B Testing

In recent years, amid growing interest in Bayesian interference for A/B testing, many commercial experimentation platforms have added a Bayesian mode, including GrowthBook, LaunchDarkly, PostHog, Amplitude Experiment, Optimizely, VWO, Statsig, and Eppo.

The main argument is that Bayesian A/B testing is modern, flexible, and easier to interpret than frequentist statistics. In addition, it’s often claimed that many of the frequentist complexities, like multiple testing and sequential testing, fall out naturally in the Bayesian way of thinking.

This blog post aims to interrogate these claims and to posit a counterargument: in Bayesian A/B testing, there is a theme of oversimplification that frequently leads to bad inference practice that is no better than peeking in frequentist statistics.

This post is derived from a paper we recently wrote that found that Bayesian and frequentist frameworks are much closer than the debate suggests.

We’ll break down the paper’s findings in five steps:

• First, we examine some generic claims about Bayesian A/B testing and show exactly when and how they apply.
• Second, we organize Bayesian configurations into tiers that provide different guarantees for the experiment program.
• Third, we show how Bayesian decision theoretic formulations can be translated into equivalent frequentist formulations.
• Fourth, we make the broader point that Bayesian and frequentist experimentation overlap.

We see many companies debating whether they need both Bayesian and frequentist modes while not yet nailing the basics of either. Many are drawn to tools with flexibility without accounting for their complexity. Our goal in this post is to articulate this complexity, to explain why Spotify currently sees no need to add Bayesian inference alongside existing frequentist tooling, and to help companies seeking to run experiments understand how to choose the best mode for their needs.

Clarifying what Bayesian inference for A/B testing is

The question of whether to use Bayesian A/B testing arises when a company wants to launch or enhance an experiment program, a series of experiments over time. Many companies have been led by the online discourse to believe that Bayesian inference is by default the best approach.

The problem is that Bayesian inference for A/B testing is not one thing. It is a family of configurations, each defined by a stopping rule, a prior, and a likelihood. Rather than starting with the tool (Bayesian A/B testing), a company should start by asking: what goals are important for my experiment program? For example:

• The program should limit the number of shipped features with no effect
• Impact should be estimated with a certain precision
• A certain cost function should be minimized over time

The statistical configuration that is best for the company then depends on the goals and constraints of the experimentation program.

Online discussion about Bayesian A/B testing is confusing because it implicitly mixes goals and configurations. Non-stats-savvy readers are left thinking that Bayes is an almost mystical form of inference that avoids the problems associated with frequentism. A claim that sounds like a fact about Bayesian A/B testing is often a fact about one goal paired with one configuration.

These are not variations on a theme. They are different configurations with different guarantees to meet different goals. In fact, the Bayesian inference framework covers everything from state-of-the-art sequential methods to procedures that are numerically identical to bad frequentist practice. We think understanding the Bayesian experimentation discourse lies in understanding these intricacies.

Further, we think it is important to realize that frequentistic error rates are not only for frequentists: Any experimentation program produces false positive and false negative results over time, regardless of whether its inference is based on Bayesian or frequentist philosophy, it is simply a classification of errors. The relevant question is whether the goal of the program is to control those rates, and whether the chosen configuration has parameters that control them.

Dissecting common claims about Bayesian A/B testing

In each of the four examples below, we take a claim, dissect it, and show under what conditions it holds.

1. Peeking

Common claim: “Bayesian A/B testing does not require peeking correction.”

This can mean two very different things. You simply may not care about controlling the false positive rate under peeking. Or you may be using a Bayesian configuration, such as Bayes factor stopping, that controls it automatically.

Those are not the same argument. The first implies the goal. The second implies the configuration. Treating both as a generic property of “Bayes” is where the confusion starts.

2. Multiple metrics

Common claim: “Bayesian testing handles multiple metrics automatically.”

This claim mainly hides how specific and demanding the relevant Bayesian configuration is. It’s like saying “peeking doesn’t inflate the false positive rate” for frequentist experiments. It’s true for some configurations of frequentist inference, and false for others: it’s not a statement about the framework itself.

There are two aspects of this statement that makes it muddle the frequentist vs Bayes discourse: First, the error rate that can be bounded without an explicit multiple-testing correction is the false discovery rate, not the false positive rate that many readers will assume. Second, the configuration that delivers false discovery rate control is very specific and far more demanding than the claim suggests.

3. The winner’s curse

Common claim: “Bayes fixes the winner’s curse.”

This is the least problematic claim. Any informative prior shrinks the effect estimate toward the prior mean, and that shrinkage works against the winner’s curse. With a reasonable informative prior, shipped effect estimates will be less inflated than estimates from a flat prior or a frequentist point estimate.

The problem is that most teams using Bayesian A/B testing run the platform default, which usually means a flat prior. A flat prior provides no shrinkage. Its posterior mean equals the maximum likelihood estimate, so the winner’s curse is just as severe as in the frequentist case. The claim “Bayes fixes the winner’s curse” is only true for people using an informative prior, and the degree of correction depends entirely on the quality of that prior.

4. Decision theory often connects to error-rate control

Common claim: “We need Bayes because we want to take a Decision theory-theoretic approach.”

Decision theory lets you specify costs instead of error rates. It’s strictly speaking not a Bayesian branch of math, but in A/B testing, the optimal policies turn out to be Bayes rules. For a simple and natural cost function that sums the costs of false positives, false negatives, and sampling, the optimal or near-optimal policy uses Bayes-factor thresholds. Because those thresholds also provide error-rate guarantees, the decision-theoretic and error-rate formulations become two parametrizations of similar configurations.

Bayes and frequentism are closer than you think

For the configurations most commonly deployed in practice, Bayesian and frequentist A/B testing are far more similar than the debate suggests.

Under a flat prior and the two-group normal model, Bayesian and frequentist procedures produce numerically identical outputs:

• The posterior mean equals the maximum likelihood estimate
• The one-sided posterior probability that B beats A equals one minus the p-value
• A 95% posterior-probability threshold is algebraically equivalent to the one-sided rejection region

An experimenter using flat-prior Bayesian inference with posterior-probability thresholds is running frequentist-equivalent inference under a different vocabulary.

Isn’t the interpretation nicer under Bayes?

Yes. Saying “there is a 95% probability that the effect lies between 0.2% and 1.1%” is more natural than explaining what “95% confidence” means. That said, we think the value of the probabilistic interpretation is oversold.

Under a flat prior, every probabilistic statement maps one-to-one to a frequentist quantity. The point estimates and interval bounds are the same. The posterior probability that B beats A is one minus the p-value. The probabilistic interpretation only becomes meaningfully different when informed by a good prior, and a good prior is not free: specifying, justifying, maintaining, and explaining it costs complexity. We gain interpretation but we pay in other kinds of complexity.

Why Spotify has not added Bayesian modes

Historically, all of Spotify’s production experiments are analyzed using frequentist statistics. We have long debated whether Spotify should allow experimenters to choose the mode of inference, or even only support Bayesian statistics in the platform.

To help us decide, we revisited the goals of the Spotify experimentation program:

• We want to minimize the number of experiments that lead to poor business decisions:
• Shipping changes that harm our product
• Shipping and maintaining changes that don’t improve the user experience
• Evidence from experiments should be trusted
• Experiment results should be hard to interpret incorrectly, especially across teams and departments
• Experiments should be easy to plan and configure, but hard to misconfigure

Because people at Spotify constantly collaborate in experimentation, both in setting up experiments and consuming results across the company, having two modes has a high cost. Supporting both modes of inference would require different planning, different monitoring, different interpretation, and possibly different-looking outputs. So far, the benefit of a second framework doesn’t outweigh those costs.

The advice we want to give is: Start with the experimentation program you want. Decide which guarantees you need and can maintain consistently. Then ask whether a specific Bayesian configuration meaningfully improves on what you already have. For Spotify today, it does not.

Pretraining progress is mostly coming from data

Breaking down 6 years of pretraining progress into data vs model improvements

How much of the rapid progress in AI that we’ve seen over the last few years has come from data versus model improvements? The answer has big implications for the economics of frontier labs and the pace of future progress.

We investigate this question at a relatively small scale, and for pretraining specifically, from 2019 to 2025. During each of those years, a new open model recipe was published which codified that year’s publicly known algorithmic tweaks (for example, improvements in architecture, optimizer, initializations, learning rate schedule, hyperparams, etc). And during each of those years, there was also a new public data corpus (produced by broader scrapes and new curation/extraction/filtering techniques).

We train combinations of these year-representative model recipes and data corpuses across different scales of training compute (up to 1e19 FLOPs).

Obviously, we can’t compare these different models by their cross-entropy loss against a fixed dataset, since we’re varying the datasets they’re trained on. So instead we evaluate these models on end capabilities as measured by the OLMES eval (which aggregates 10 different relatively easy benchmarks, mostly multiple choice QA). Unfortunately, evaluating end capability rather than pretraining loss adds some noise to our results, as you’ll see in the graphs below, though we try to get cleaner bounds by running multiple seeds.

We find that from 2019 to 2025, 3.24x more compute efficiency gains have come from data improvements rather than model improvements (12.0x for data and 3.7x for models), at the 1e19 FLOPs compute budget.

Here is a grid which shows how much better a model we train does on the end capability we're testing it on, relative to the 2019 data + architecture baseline, at 3.16e18 FLOPs.

We find that the gains from data and model improvements are mostly independent and don’t interact (i.e. realizing the gains from some model improvement doesn’t require a specific training datapile, or vice versa). 88% of the variance in the OLMES score can be explained by additive effects of the model and data improvements (using a linear model).

Discussion

For context, let’s briefly summarize what changed on both the data and the model side from 2019 to 2025.

On the model side, we went from GPT-2 to OLMo-2, including key innovations in optimizers, positional encodings, normalization, activation functions, initializations, and more.

On the data side, we started with OpenWebText in 2019, which contained just web pages linked from Reddit with enough upvotes and then deduplicated and filtered, and thus amounted to only ~9B tokens (this was mostly what GPT-2 was trained on). By 2025, open source data corpuses like UltraFineWeb not only are far larger (by using scrapes of the whole Internet), but also use much more sophisticated filtering (for example, by training a classifier to predict what data will empirically improve model performance).

A naive interpretation of our result is that most of the AI progress from 2019-2024 (the era of pretraining) was just better data engineering (extraction, curation, etc.), and that all the model work during that period was much less important.

But this is probably the wrong way to think about the value of model improvements. Their main contribution was not necessarily compute efficiency - that is, achieving the same performance with fewer FLOPs. Rather, it was making larger amounts of compute usable in the first place. As the number of parameters, context lengths, run duration, and clusters scale up, all kinds of things are prone to breaking (gradients explode or vanish, memory and bandwidth run out, training becomes infeasibly slow). Much of model research has consisted of removing or pushing back these constraints to scaling. Many of the most important innovations such as MoEs, sparse attention variants, stability innovations (norm placements, initializations, etc.) and system / kernel-level optimizations like FlashAttention fall into this category.

The data improvements we investigated here might matter less for larger models. Small models (like the ones we trained) see significant gains from data quality improvements, because they don’t have that much capacity, and so you have to be really careful about what you stuff into them. Whereas big models have so much excess capacity that maybe you just want to throw in as much stuff as you can, even if it’s mostly garbage, and the magic of stochastic gradient descent will separate out the signal from the noise. If you choose to filter aggressively, you’ll have to do dozens of epochs, which empirically gives worse performance than just having a lower average quality but larger dataset. In fact, aggressive data curation is even more harmful once you take into account that frontier models are up to 100x overtrained relative to Chinchilla optimal, in order to minimize the inference compute used for RL and for deployment.

An analogy might be the difference between a sailboat and a container ship - the container ship doesn’t necessarily go faster, but it can lug thousands of tons of cargo (analogous to hundreds of trillions of tokens of pretraining data), and won’t be toppled by choppy waters (analogous to training stably across hundreds of thousands of GPUs).

Now that we have more capacious and sturdy container ships, we don’t have to fret about exactly what we load on board - we can just fill them up with everything that’s even remotely and plausibly useful. Whereas for the tiny flimsy sailboats of 2019, you’d have to be incredibly careful about only carrying the most valuable cargo.

But to the extent that the nature of pretraining progress is simply loading more cargo into this ship, are we running out of cargo? This is a question about the data wall and about how well synthetic data has helped us leap over it. Synthetic data is obviously being widely used at the labs, and we have not at all investigated whether it can effectively expand a data corpus without hurting model performance. If the gains are limited, then the main driver of pretraining progress will stall, because we’re not generating more internet, and you can only curate a fixed set of data by so much. To be clear, we have no active reason to think this. But given how important data seems to be in driving pretraining progress, this seems like a crucial question to investigate.

Ryan Greenblatt noted that many of the historical improvements in pretraining data corpuses look like the kind of progress that automated researchers would be able to just test empirically - for example, run ablations trained on different data and see how the model performs. So it’s totally compatible with our results that the data progress which has propelled pretraining since 2019 might speed up a lot if and when we automate AI R&D.

We want to clarify that whether pretraining progress in isolation will speed up or slow down is not really the most important question for overall AI progress, because so many of the gains over the last two years have come from RL.

Future research

These are some directions of future research that we think would be really cool, and important questions to answer:

1. You could run this experiment at larger scales to see whether the data or model improvements are more dependent on scale (and thus far more impactful at the frontier).
2. What is the marginal value of novel high-quality data for both pre and post-training, as measured by end capabilities?
3. We want to know broadly how effectively synthetic data works. One concrete question to investigate is this: if you’ve got a small corpus of high quality data, how much better is it to magnify it via synthetic data generation relative to just training on it for multiple epochs?
4. You could figure out the implied value of data through lab spending on data brokers, environment producers, etc., relative to their spending on compute and researchers.

We wanted to investigate what role data has played in driving AI progress. There are lots of other ways one could probe this question, and some may be more clever and informative than ours. And even our experiment was done at an extremely small scale. We definitely think it’s plausible that there is something we missed - we’re eager to hear how others would research this question, and ideally to also see their results!

Appendix: Methodology

We pre-train these model recipes from scratch on these different data corpuses, at varying compute budgets, with multiple independent seeds. Our compute budgets are: 1e17, 3.16e17, 1e18, 3.16e18 and 1e19 FLOPs. The compute accounting convention is to use nominal compute C = 6ND (N = number of non-embedding parameters, D = tokens of data).

At each compute budget, we vary the number of parameters (and hence number of tokens trained on), to determine the compute-optimal mix for each training recipe x corpus combination. We use held-out loss on the corpus to determine this compute-optimal point. We can then obtain compute scaling curves of downstream performance of each combination, from which we can finally extract our compute multipliers.

We enforce a shared tokenizer and context length across every run: GPT-2 BPE (tiktoken, 50,257 vocab) and T=2048, batch = 262,144 tokens.

The end capabilities of our training runs are highly dependent on hyperparameters. Obviously, there is no way to sweep over all possible sets of hyperparams (hyperparam tuning is a fine art indeed)! We try to control for this as much as possible, and we consider peak learning rate as the main hyperparameter of significance.

Some algorithm vintages do provide specifications of what peak learning rate should be tuned to (as a function of other relevant variables such as model size, data budget, batch size, etc.). These serve as good priors for what we think the optimal learning rate is.

We first sweep learning rates at 5 anchor points - 3 different model sizes and 2 different D/N ratios. We determine the optimal learning rate of these anchor points, and fit an optimal learning rate parametric form.

For all the model recipes except OLMo-2, we fit a common exponent a and b, and a model-specific lr₀. For OLMo-2, we use the prescribed optimal learning rate according to the model recipe. The reason we do this for OLMo-2 is that Ai2 published small-model ladders as part of the recipe which specified optimal hyperparameters at the scale we are investigating. We also verify, at the compute-optimal point for 3.16e18 FLOPs, that our production learning rates are at or near optimal.

Main technical results

Explaining some anomalies in our graph

We observe generally increasing compute efficiency across time for both the model and data axes as expected. Some outliers that we observed:

1. NeoX performs worse than GPT-2 at 1e19 (although it does better across the 1e17 to 3.16e18 range). This might arise from noise in the OLMES evaluation. We also note that on held-out pretraining loss on the FineWeb-Edu corpus, NeoX performs better than GPT-2.
2. The Piles seems to do much worse than OpenWebText. This is not surprising since the Pile’s main improvement was data corpus diversity over filtering. It has a curated 22-source mixture including PubMed and arXiv papers, GitHub code, legal opinions, patents, and parliamentary proceedings. The amount of cross-domain transfer to OLMES (which is English web-prose MCQ) might be minimal for many of these tokens, thus resulting in lower compute efficiency. We note that by virtue of its larger size, we expect that the Pile should eventually be better than (the really small) OpenWebText at larger scales.
3. It is also worth noting that the compute multipliers for NeoX and the Pile are obtained by extrapolation, which introduces further potential error.

How compute multipliers were calculated, as well as their error bars

• Every point on the compute scaling curves is computed from multiple independently seeded training runs. The error bars there are the standard deviation of the OLMES eval over those seeds.
• Consider some given reference level of performance at some compute level for our reference model or data corpus.
• We then calculate the compute multiplier by finding the left-most point of the compute scaling curve of our candidate model or corpus that first attains that reference level of performance. The ratio of the compute required by the reference to the compute required by our candidate is the candidate’s compute multiplier.
• The error bars on the compute multipliers are obtained from a parametric bootstrap of the entire estimation pipeline, and are 1 standard deviation intervals.
• We do want to highlight that we expect the actual uncertainty in the compute multipliers of the model recipes to be higher than indicated by our error bars. This is because of additional uncertainty introduced by the limited extent of hyperparameter tuning we did, and end capabilities or held-out loss is probably quite sensitive to the exact choice of peak learning rate / batch size / etc.

It is also important to note that there are many reasons why our ablations do not necessarily capture the full scope of compute efficiency gains. Indeed, from 2019 to 2025, we observe year-over-year compute efficiency gains (CEG) of 1.24x on the model side and 1.51x on the data side. Measured jointly, we observe a 1.57x YoY CEG. This is indeed much lower than Anson Ho et al.’s mean estimate of 3x YoY, for the following reasons:

1. Many of the gains might be scale dependent or might be especially important at longer context, and we are operating at scales too small to realize many of the gains.
   1. For example, OLMo-2’s layer and QK norms, parallel attention + MLP block in NeoX
2. Inference efficiency optimizations (such as LLama-3’s GQA, which is a KV cache optimization) do not show up as compute multipliers in our study. We are also not investigating tokenizer improvements.
3. The compute multipliers we obtain are pretty sensitive to our choice of model recipe or data corpus for each year. We have chosen what we believe to be representative model recipes or data corpuses. But by no means do we exhaustively conclude that these are the best of each year.
4. We are looking at compute multipliers with respect to the OLMES benchmark (which combines 10 different relatively easy task types) rather than compute multipliers in getting to some perplexity metric. We would also have very different looking numbers if we were looking at other benchmarks (say, coding- or problem-solving-specific ones), which would probably reward very different methods of data engineering.

We also want to note that we have not investigated other data-side improvements, such as collecting more high-quality data from new sources, human expert generated data, synthetic data generation methods, etc. Most of the corpuses we have investigated are curations (subsets) of the same Common Crawl, rather than expanding the available set of data. This is clearly consumption of a finite stock - there is only so far we can push this lever.

Independence of gains from model recipe and data corpus

Here is the investigation that we did to determine how independent the gains from model recipe and data corpus are. We looked at the grid of OLMES scores at 3.16e18 FLOPs. A linear regression of OLMES score = mean + model effect + data effect gives an R squared of 0.88, which means 88% of the variance in the OLMES score can be explained by additive effects of the model and data improvements, with only ~12% of the variance accounted for by interaction or higher order terms, and eval noise. This hints that complex model-data interactions (where exploiting some model improvement is contingent on some specific data engineering, or vice versa) are relatively minor.

xlDuckDb

Use DuckDB within Excel with the xlDuckDb addin.

DuckDB is an amazing tool with deep integration with Python and R, but sometimes you just need data in Excel. xlDuckDb allows DuckDB SQL to be run within Excel. Query results are returned as regular Excel cells.

Installation

Download the xlduckdb.zip file from the latest release on the right.

Use the "Extract All" option on the right-click menu in Windows Explorer to extract contents of the zip file on your local PC.

Right-click on the xlDuckDb64.xll file and unblock it:

Double click on the xlDuckDb64.xll file to open the addin in Excel. The function DuckDbQuery will be registered automatically.

Requirements

xlDuckDb only runs on 64-bit Excel 365 released after Sept 2018. The dynamic array resizing functionality is required.

xlDuckDb v0.6.0 and above require the .Net 8.0 or later runtime. Usually a compatible version of .Net will already be installed, however the RunTime can also be downloaded from .Net RunTime

Usage

Any DuckDB SQL can be run and the results will be returned to Excel.

A note on copying queries into Excel

When copying text such as SQL commands into Excel cells, add a ' at the start so that Excel treats the input as a string.

Querying Excel Ranges

Excel ranges can be queried as if the range was a regular DuckDb table. The keyword xlRange is used in queries, internally xlDuckDb substitutes the Excel range data.

SELECT * FROM xlRange

The range is passed as the third argument to the DuckDbQuery() function.

Here we copy in the 2024 F1 Drivers Points and query the range:

The data is in the range A1:E25

The range is passed as the third argument:

=DuckDbQuery(A27,,A1:E25)

And the query is run against the Excel range:

Named Ranges

Queries can be run against named ranges.

Here we pass the named range DriverPoints:

=DuckDbQuery(A27,,DriverPoints)

Table Data

Queries can be run against Excel table data.

Here we pass the table name PointsTable[All]:

=DuckDbQuery(A27,,PointsTable[All])

Note: Multiple Excel ranges can be passed to the DuckDbQuery() function. Any number of ranges can be passed as the 3rd, 4th, 5th..nth argument. These ranges can then be referenced in the SQL query as xlRange[1], xlRange[2]..xlRange[n].

Querying JSON files

Reading data from JSON files can be difficult, particularly if the data is nested. DuckDB allows the JSON file to be treated just like a regular database table.

SELECT firstname, surname, bornCountry FROM 'laureate.json' LIMIT 5

We can now easily find the countries with the most Nobel prize winners:

SELECT bornCountry AS Country, COUNT(*) AS Number FROM 'laureate.json' GROUP BY bornCountry ORDER BY COUNT(*) DESC LIMIT 5

DuckDB supports the use of JSONPath to extract values from nested JSON fields. This allows us to extract the category and motivation of the first prize awarded to each person:

SELECT firstname, surname, prizes->>'$[0].category' AS Category, prizes->>'$[0].motivation'AS Motivation FROM 'laureate.json' LIMIT 5

Querying CSV files

DuckDB is able to handle most CSV files automatically, detecting the column delimiters, data types, and so on. DuckDB allows the CSV file to be treated as a regular database table:

SELECT Region, "Item Type", "Total Revenue" FROM '10000SalesRecords.csv' LIMIT 5

DuckDB has a user-friendly PIVOT statement that allows us to view the revenues in Asia and Europe broken down by item type:

PIVOT '10000SalesRecords.csv' ON Region IN ("Europe", "Asia") USING sum("Total Revenue") GROUP BY "Item Type" ORDER BY "Item Type"

Data from multiple sources can be combined in a single SQL query. We can combine the JSON and CSV data to show the total sales in the countries with the most Nobel laureates:

WITH CountrySales AS (SELECT CASE WHEN Country='United States of America' THEN 'USA' ELSE Country END AS Country, Sum("Total Revenue") AS Sales FROM '10000SalesRecords.csv' GROUP BY Country) SELECT cs.Country, SUM(cs.Sales) AS Sales, COUNT(*) AS "Nobel Laureates" FROM CountrySales cs INNER JOIN 'laureate.json' l ON cs.Country = l.bornCountry GROUP BY cs.Country ORDER BY "Nobel Laureates" DESC LIMIT 5

Querying Parquet files

Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. DuckDB has extensive support for efficient querying of Parquet files.

SELECT Survived, Pclass, Sex, Age FROM 'titanic.parquet' LIMIT 5

We can now contrast the ages across cabin classes for survivors versus non-survivors:

WITH Survivors AS (SELECT Pclass, Age, Sex FROM 'titanic.parquet' WHERE Survived=1) PIVOT Survivors ON Pclass USING AVG(Age) GROUP BY Sex

Querying from remote locations

DuckDB has functionality to directly query data located on https and in AWS S3.

SELECT * FROM 'https://duckdb.org/data/holdings.parquet';

Similarly we can attach to the DuckDB stations database in S3 and query the number of stations in each country.

Thanks

xlDuckDb would not have been possible without the ExcelDNA and DuckDB.NET projects.

The DuckDB query in Excel functionality was extracted from the commercial product xlSlim. If you need the same DuckDB functionality within a Digicert EV certificate signed addin then xlSlim is what you are looking for.

TPC-H Query 15 - Scalar Constants and Materialisation

Today's TPC-H query is the first one to use a WITH clause.

The same expression is used twice, and the query quietly tests for a simple, yet powerful, optimisation: materialisation.

As we shall see, Databricks, Trino and DataFusion all fail the test and scan 6M more rows than the databases that implement the optimisation.

Query 15

Here is Query 15, which is the first query featuring a Common Table Expression (aka: a CTE, it's that thing after WITH).

Notice that the same CTE is referenced twice in this query.

WITH revenue AS (
    SELECT 
      l_suppkey AS supplier_no,
      SUM(l_extendedprice * (1 - l_discount)) AS total_revenue
    FROM lineitem
    WHERE l_shipdate >= '1997-09-01'
      AND l_shipdate < '1997-12-01'
    GROUP BY l_suppkey)
SELECT 
  s_suppkey,
  s_name,
  s_address,
  s_phone,
  total_revenue
FROM supplier
INNER JOIN revenue
  ON s_suppkey = supplier_no
WHERE total_revenue = (SELECT MAX(total_revenue) FROM revenue)
ORDER BY s_suppkey

Estimation and Join Order

There is only one filter in the query, on l_shipdate.

Even with the filter, the output of lineitem is still much larger than supplier (which has no filter and emits 10K rows).

The join order of the outer query is trivial: lineitem ⨝ supplier.

Scalar Constants and Cross Joining

Notice this expression: (SELECT MAX(total_revenue) FROM revenue). This value can be calculated before running the join between supplier and lineitem in the outer query.

How do we then recombine the constant value with the query to evaluate the WHERE total_revenue filter?

There are two general approaches:

1. Run the MAX(total_revenue) subquery first and store the value as a parameter - then run the outer query
2. Run the MAX(total_revenue) as part of the outer query, and just do a cross join to evaluate the filter

If your join implementation is solid - then option 2 saves you a lot of trouble because it allows you to stay in a space that looks like relational algebra. You will be running this join:

FROM supplier
INNER JOIN revenue
CROSS JOIN (SELECT MAX(total_revenue) AS the_max FROM revenue) AS F
WHERE total_revenue = F.the_max

This is elegant and avoids duplication of code in the Execution Engine of the database. For example, here is DataFusion doing exactly that (Trino does the same thing):

Operator
SORT s_suppkey
SORT s_suppkey
INNER JOIN HASH ON MAX(revenue.total_revenue) = total_revenue <--- Here is the cross join!
│└INNER JOIN HASH ON s_suppkey = supplier_no
│ │└PROJECT supplier_no, total_revenue
│ │ AGGREGATE SUM(l_extendedprice * (1 - l_discount)) GROUP BY HASH l_suppkey
│ │ DISTRIBUTE HASH ON l_suppkey
│ │ AGGREGATE SUM(l_extendedprice * (1 - l_discount)) GROUP BY HASH l_suppkey
│ │ FILTER (l_shipdate >= DATE'1997-09-01') AND (l_shipdate < DATE'1997-12-01')
│ │ TABLE SCAN lineitem WHERE ((l_shipdate >= DATE'1997-09-01') AND (l_shipdate < DATE'1997-12-01')) 
| |   AND (l_suppkey <= 10000))
│ TABLE SCAN supplier
AGGREGATE MAX(total_revenue)
DISTRIBUTE GATHER
AGGREGATE MAX(total_revenue)
PROJECT total_revenue
AGGREGATE SUM(l_extendedprice * (1 - l_discount)) GROUP BY HASH l_suppkey
DISTRIBUTE HASH ON l_suppkey
AGGREGATE SUM(l_extendedprice * (1 - l_discount)) GROUP BY HASH l_suppkey
FILTER (l_shipdate >= DATE'1997-09-01') AND (l_shipdate < DATE'1997-12-01')
TABLE SCAN lineitem WHERE (l_shipdate >= DATE'1997-09-01') AND (l_shipdate < DATE'1997-12-01')

But, if you fail to realise that this scalar value is really a cross join in disguise, you end up with special, scalar handling code in your Execution Engine.

For example, here is Databricks scanning lineitem, storing it as a scalar, and then using it again on the second scan of lineitem:

Operator
SEQUENCE
├─DISTRIBUTE HASH ON s_suppkey
│ INNER JOIN HASH ON s_suppkey = revenue.supplier_no
│ │└DISTRIBUTE GATHER
│ │ TABLE SCAN supplier
│ FILTER 
│ AGGREGATE SUM(l_extendedprice * (1BD - l_discount)) GROUP BY HASH l_suppkey
│ DISTRIBUTE HASH ON l_suppkey
│ AGGREGATE SUM(l_extendedprice * (1BD - l_discount)) GROUP BY HASH l_suppkey
│ TABLE SCAN lineitem WHERE (l_shipdate >= DATE'1997-09-01') AND (l_shipdate < DATE'1997-12-01')
└─AGGREGATE MAX(revenue.total_revenue)
  DISTRIBUTE GATHER
  AGGREGATE MAX(revenue.total_revenue)
  AGGREGATE SUM(l_extendedprice * (1BD - l_discount)) GROUP BY HASH l_suppkey
  DISTRIBUTE HASH
  TABLE SCAN lineitem WHERE (l_shipdate >= DATE'1997-09-01') AND (l_shipdate < DATE'1997-12-01')

Materialisation

Remember that TPC-H is cleverly designed - and this query is no exception. It is fishing for a special optimisation.

First, realise that the 6M row lineitem table (with a filter bringing it to 226K rows) is mentioned twice in this query.

The outer query accesses it with a GROUP BY and this aggregate:

FROM lineitem
WHERE l_shipdate >= '1997-09-01'
  AND l_shipdate < '1997-12-01'
GROUP BY l_suppkey

This aggregate can easily be estimated; it can't possibly be larger than the size of supplier - which is 10K rows. This is much smaller than the rowcount of lineitem (by 22x).

The subquery makes use of the same construct, like this:

SELECT MAX(total_revenue)
FROM (
  SELECT SUM(...) AS total_revenue
  FROM lineitem
  WHERE l_shipdate >= '1997-09-01'
    AND l_shipdate < '1997-12-01'
) as subquery

Note: I specifically didn't represent the subquery by its CTE alias. What the optimiser is about to do often fine without that help from the user specifying a CTE.

Here is the realisation that a query optimiser might have:

• The same query is used in two places
• The output size of that query is small
• If I save the results of this query - I can eliminate one of the two 6M row scans of lineitem

This insight is sometimes called "materialisation".

DuckDB can do it, and so can Postgres and SQL Server. Here is the DuckDB query plan:

Operator
SEQUENCE
├─MATERIALISE AS m_revenue <--- Construct the aggregate
│ AGGREGATE sum_no_overflow #1 GROUP BY HASH #0
│ PROJECT l_suppkey, l_extendedprice * (1.00 - l_discount)
│ TABLE SCAN lineitem WHERE l_shipdate >= '1997-09-01' AND l_shipdate < '1997-12-01'
└─SORT s_suppkey
  INNER JOIN HASH ON s_suppkey = supplier_no
  │└INNER JOIN HASH ON total_revenue = SUBQUERY
  │ │└PROJECT CASE WHEN (#1 > 1) THEN ...
  │ │ AGGREGATE "first" #0, count_star() <--- Calculate the max from the cached
  │ │ AGGREGATE MAX(#0)
  │ │ PROJECT total_revenue
  │ │ SCAN MATERIALISED revenue
  │ SCAN MATERIALISED revenue <--- Scan from the cached
  TABLE SCAN supplier

Notice how the aggregate is materialised - then reused twice by the code.

In the case of TPC-H Q15 - this small optimisation saves a 6M row scan.

Summary

In today's analysis of Query 15, you saw how storing an intermediate result that is referenced more than once can greatly reduce the work needed to run the query.

In a way, this type of materialisation is almost like the automatic creation of a temp table.

Some databases, like DuckDB, SQL Server and Postgres, handle this gracefully and pass the test Q15 sets up before us.

We also saw that even though this optimisation is straightforward - Databricks, DataFusion and Trino all fail to make it.

Microsoft's CoreAI portals all sat on Fluent 2 and still felt disjointed, because shared components did not prevent teams from inventing different navigation, layouts, and flows. The CoreAI Design System, forked from Foundry's work, ships guidance through both a Figma library and an experimental MCP server so coding agents can consume it too.

I’m building out the design engineering practice at DuckDuckGo. It’s a small team right now, and we’re hiring our second person to help shape the practice and lead it with me.

That might sound early for a set of principles, but I think it’s exactly the right time. The decisions we make now will become the habits, expectations and quality bar for everyone who joins later.

I wanted to put down how we’ll approach some of the problems we’re trying to solve: handovers between design and engineering, a lack of shared knowledge across that boundary, AI-generated solutions that aren’t properly understood, and speed winning out over craft too often.

This is where I’ve landed.

Understand before we make

We start with the actual problem, desired outcome, and real constraints, not a preferred solution.

• Question assumptions and validate direction early.
• Understand the code and product context before changing them.
• Use AI to accelerate exploration and execution, never to replace judgement.
• If we cannot explain how something works and why it is right, we do not own it yet.

Own the experience, not the handoff

We share responsibility from problem framing through production and iteration.

• Work across product, design, and engineering boundaries.
• Prototype in the medium that answers the question best.
• Treat code, interaction, content, documentation, and support as parts of one experience.
• Stay involved until the shipped result meets its intent.

Make trust effortless

Privacy, clarity, accessibility, and reliability belong in the experience from the beginning.

• Hide unnecessary complexity from people, not important consequences.
• Make safe choices understandable and easy.
• Design for different abilities, skill levels, devices, and circumstances.
• Never trade user trust for engagement or short-term convenience.

Move fast by making less, better

When time is constrained, reduce scope before reducing quality.

• Ship the smallest complete and coherent experience.
• Separate what must be excellent now from what can genuinely wait.
• Prefer focused progress over a larger collection of unfinished ideas.
• Treat the quality floor as fixed; negotiate scope above it.

Finish the work

A working happy path is not the finish line.

• Care about every state, edge case, word, transition, and response time.
• Treat accessibility, performance, and polish as product quality, not optional refinement.
• Make time for the final pass when planning the work.
• Push back when competing priorities would leave the experience visibly unfinished.

Make quality repeatable

Craft should compound instead of depending on heroics.

• Turn repeated decisions into shared components, defaults, tools, and checks.
• Build design fluency in engineers and code fluency in designers.
• Share work early and explain the reasoning, not just the result.
• Leave the team better equipped to produce excellent work without us.

These are deliberately practical. I want us to use them when planning projects, reviewing work, hiring people and deciding what to cut when priorities compete.

The principle I expect us to test most often is moving fast by making less, better. We need to move quickly to compete with companies that have far more people and resources. But if every deadline lowers the quality bar, we’ll never build the practice or product we want.

The principles will evolve as the team grows. For now, they describe how I want us to approach the work and the standard I want us to hold each other to.

NN/g research finds real differences between carets, arrows, and plus icons on accordions, with the caret the safest choice and the most tapped. With no icon at all, users expect the item to open a new page. Split menus, where label and icon trigger different actions, cause the most mistakes and are best avoided.

Siri AI will remain in beta when it launches with Apple's OS 27 versions on September 14. The long-delayed assistant will be subject to use caps and fees, plus language and regional restrictions. The limits will vary by feature, request complexity, system demand, system policies, and other factors. Expanded access will be available for a fee in the future. Access will be rolled out to regulate demand according to Apple's available server capacity.

Listen Labs, a market research startup that uses voice AI to conduct customer interviews, recently signed a term sheet for a $125 million Series C at a $1.5 billion valuation, with Menlo Ventures set to lead the round, according to several people with knowledge of the matter.

But that round never closed, the people said. Listen Labs walked away from the signed term sheet, a rare occurrence in the venture world and one that is generally frowned upon, according to VCs.

The financing likely collapsed because of acquisition talks with Salesforce. The CRM giant has recently held talks to buy Listen Labs for around $2 billion, Business Insider reported. The discussions are not finalized, however, and may not result in a deal, the outlet notes.

Listen Labs is one of the leading startups in the rapidly growing field of automating customer research with AI. The three-year-old startup has about $30 million in annualized revenue, about three times more than Simile, a competing startup that predicts human behavior, according to two people familiar with the companies’ financials. In late July, Simile announced that it had closed a $200 million Series B at a $2 billion valuation led by Greenoaks — likely setting a new valuation benchmark for Listen Labs, one person said.

If talks with Salesforce collapse, several VCs told TechCrunch that they expect Listen Labs to return to market and target a valuation of $2 billion or higher.

While acquiring Listen Labs could strengthen Salesforce’s AI capabilities by using the startup’s AI to help predict customer needs, the CRM giant may ultimately decide that paying a 67-times revenue multiple is too steep a valuation, according to a person with experience negotiating exits to Salesforce.

Listen Labs, Salesforce, Menlo Ventures, and Simile did not immediately respond to requests for comment.

Listen Labs was co-founded in 2023 by Florian Jüngermann, a former German national champion in competitive computer programming, and Alfred Wahlforss, who previously founded a staffing startup called Bemlo. The two met while pursuing master’s degrees at Harvard.

Listen Labs’ AI develops survey questions and interviews customers over audio or video. The resulting conversations are then packaged into reports and PowerPoint presentations, similar to those traditionally produced by human market researchers.

Fortune 500 companies rely on this type of research to gauge customer needs and satisfaction with their brands and products, but traditional market research is expensive and can take weeks to complete.

Listen Labs’ technology helps reduce the time and cost of these projects, enabling companies to quickly understand how customers are reacting to product changes and iterate on them more efficiently.

The startup’s customers include Microsoft, Canva, Anthropic, and Sweetgreen. Listen Labs and Simile aren’t the only startups using AI to disrupt the customer research market.

Besides Simile, competitors in the space include Outset, Keplar, and Aaru. While some platforms automate interviews with real humans, others startups — like Aaru and Simile — take a synthetic approach, using AI to simulate human behavior and predict responses without interviewing anyone at all.

Listen Labs was previously valued at $500 million when it announced a $69 million Series B round in late January led by Ribbit Capital, with participation from returning backers Sequoia, Conviction, and Pear VC.

Q2D-Web is a large-scale benchmark and leaderboard for evaluating retrieval models on web search, encompassing 190 million documents and 69,721 queries in ten languages. It uses three separate sets of relevance judgments to minimize bias and improve reliability, assisting retrievers in distinguishing relevant data from distractors. Evaluation submissions are processed through a subsampling method to reduce resource costs while maintaining model ranking integrity.

Google Cloud and Accenture are working together on a joint unit dedicated to sending engineers into enterprises to help them better adopt Google’s AI tools and services.

The new unit, dubbed Accenture Gemini Enterprise Business Group, is Google’s latest foray into the increasingly competitive world of “forward-deployed engineers,” or FDEs. Rivals in the AI race, including OpenAI, Anthropic, Microsoft, and Amazon, have all recently launched separate business units in a bet that implementing AI models can become its own trillion-dollar business.

It’s the kind of bet AI companies and hyperscalers increasingly need to make. Hyperscalers are committing hundreds of billions of dollars a year to GPUs, data centers, and power capacity even as the revenue directly attributable to AI remains a fraction of that investment.

Google Cloud generated $24.8 billion in the second quarter, a big chunk of which was driven by enterprise AI. But the commitments behind that growth are enormous. Google Cloud’s parent company Alphabet reportedly accumulated $811 billion in purchase commitments and contractual obligations as of June 30.

This return on investment is not yet materializing in the way companies and investors need it to, so everything hinges on whether or not AI companies can create enough demand for their services. But that demand is not guaranteed, as enterprises themselves are struggling to see a true return on investment on their AI spending.

It’s conventionally held that enterprises have simply lacked the expertise to intelligently integrate AI tools and services into their workflows in a way that not only saves them money, but helps them make more of it in the long run. That’s where the FDEs come in as a steady, guiding hand that, ideally, possesses the perfect mental cocktail of business acumen and agentic AI prowess needed to change everything.

As part of its deal with Accenture, Google will train up to 1,000 of the consultancy firm’s FDEs to work with enterprises and build custom AI applications on the Gemini Enterprise platform. The organization will live under Accenture, according to a Google spokesperson.

According to August data from Ramp, Google accounts for roughly 6% of enterprise AI spending among Ramp’s U.S. customers, compared to Anthropic’s 43.5% and OpenAI’s 39.7%. (A Google spokesperson pointed out that many of Ramp’s customers exclude the types of major enterprises that are signing large, strategic AI deals with Google Cloud, which go beyond just model API usage — like Oracle, Meta, Anthropic, and ServiceNow.) Google’s new unit with Accenture, which The Wall Street Journal first reported, is the latest of its aggressive expansions of its FDE model this year as it attempts to resolve enterprise deployment bottlenecks and catch up to rivals.

Earlier this year, Google Cloud launched a $750 million partner ecosystem commitment that embedded Google’s own FDEs across multiple consultancies, including Capgemini, Cognizant, and Deloitte. The tech giant also struck a multi-year partnership with CVC Capital Partners to deploy FDEs directly into the investment firm’s portfolio companies.

Google isn’t the only giant at risk of being outpaced by newer firms. Companies that are dedicated specifically to embedding engineers into businesses to build bespoke AI workflows — like Ode with Anthropic, or OpenAI’s The Deployment Co. — threaten big consultancy firms like Accenture as well. For the professional services giant, the Google tie-up adds to its own wave of FDE programs this year, which include a similar Microsoft FDE practice in March, an FDE initiative with ServiceNow in May, and a joint program with SAP in June.

Full article content is not available for inline reading.

Read the original article →

AI music model maker Suno on Wednesday unveiled a new model family called Suno v6, which it says was developed using licensed data from music labels and distributors such as Warner Music Group, BMG, and Believe.

Suno has faced multiple lawsuits from record labels, which accused the startup of using copyrighted data to train its AI models, which let you generate music with prompts. Suno settled with Warner Music Group last year and struck a deal with BMG last month.

The company said Suno v6 is not trained using the data it used to train previous versions of its music-generating model.

The startup is releasing three versions of the model: The base Suno v6 model is available to paying users and is said to be reliable and steerable for controlled outputs; Suno v6-wild is an experimental model, also available to paying users, that’s meant to be used for ideation and getting unexpected results. Lastly, Suno v6-mini is a faster version available to all users. The company plans to retire its older models.

Suno says the new model lineup offers greater flexibility in how you use them. Suno v6 lets you edit a part of a song using a prompt or a word in the lyrics, and even lets you use text, images, or video as references to create tracks. You can separate an instrument from a sample and create a new beat with it.

The startup also said it plans to add new features such as letting users remix songs — provided the respective artists opt in to a new program it is setting up with music labels that lets Suno use their songs for AI-generated features.

Jack Brody, Suno’s chief product officer, said through derivative work, there is more opportunity for revenue generation for all stakeholders.

“I think the music ecosystem and our partners are always looking for ways to create more revenue opportunities for their rights holders and artists. So a big part of this release is creating additional revenue streams there,” Brody told TechCrunch in an interview.

Last month, Suno said it plans to add a watermark to songs generated using its platform. It also recently introduced new download limits based on the account’s tier.

“We can help manage some of the broader ecosystem challenges like streaming fraud and kind of mass exportation and uploading to distributors that are low intent. At the end of the day, we think it is up to the distributors and the platforms to govern what content goes on there,” he said.

Suno is still facing lawsuits from labels like Sony and Universal Music Group, artists like Jason Isbell, and users alleging it ignored security while chasing profits. The model announcement comes a day after the company admitted to training its models using YouTube videos.

Despite these controversies, Suno has raised over $819 million in funding to date, according to PitchBook data.

Software is about to eat the world much faster

Fifteen years ago, I wrote that software is eating the world - at the time, a controversial claim. The main prediction of the essay has come true. As a simple measure, technology’s share of the market capitalization among the ten largest businesses in the world has gone from 31.5% to 94.4% over that period. Amazingly, this happened even though only 1 in 300 people on earth wrote software, and every line of it had to be written by hand. Software has been eating the world at the speed of human hands. It is about to eat the world at the speed of compute.

Three years ago, when Scott Wu and his team began building Devin, the term “coding agent” barely existed. Today, if you talk to any software engineer at a leading company, they’ll tell you that agents write almost all of their code. In the past year, Devin has gone from writing 13% of Cognition’s production code to more than 90%.

When agents write 90% of the code, engineers can literally do ten times as much. The 10x engineer becomes the 100x engineer, as they shift from writing artisanal code to operating as the CTO of a fleet of agents. The 1000x engineer isn’t too far behind. As Scott puts it: “Within our lifetime, engineers will go from bricklayers to architects, focusing on the creativity of designing systems rather than the manual labor of putting them together.”

At Mercedes Benz, engineers turned what would have been an eight month long COBOL migration into 8 days of work with Devin. Rivian teams increased their test generation velocity by 10x. Devin triages and patches vulnerabilities across thousands of repos at some of the world’s largest financial institutions like Itau, where 70% of security vulnerabilities are automatically remediated by Devin.

Every time programmers get more leverage, doomers predict the end of software engineering. Compilers were supposed to shrink the profession. So was open source. So was the cloud. Instead, each leap made software cheaper to build, and demand for software, and engineers, exploded. It keeps happening because, as Milton Friedman observed, human wants and needs are infinite, so economic demand is infinite, and job growth can continue forever (see The Techno-Optimist Manifesto). Technology is how we unlock that demand. Technology gives us leverage, lowers the cost to produce, and makes previously uneconomical use cases economical. For software, Devin provides that leverage.

It is tempting to think of this as just another application of AI - one vertical among many. But if software continues to eat the world, the automation of software itself accelerates every industry.

This future is core to how we at a16z see the world. We have always believed that software expands markets, that technical founders with seemingly unreasonable ambition create the future, and that small teams armed with powerful tools can change entire industries. Cognition is a powerful expression of those beliefs: a deeply technical company taking on an enormous problem, moving at extraordinary speed, and building for a market whose ultimate size is difficult to comprehend.

Our conviction in Cognition comes from personally knowing Scott for over seven years. We first backed him at Lunchclub, where he and his team built a system for bringing people together and created relationships that have lasted through the years since. Scott is one of the most accomplished competitive programmers of his generation: a three-time gold medalist at the International Olympiad in Informatics and a world champion at 17. But after working with him for many years, we know that his technical brilliance is only part of what makes him exceptional. Scott has an unusual combination of speed, judgment, competitiveness, and leadership. He sees important technical shifts early. He holds himself and everyone around him to an extraordinarily high standard. He cares deeply about his customers, recruits exceptional people, and has built a culture that moves with the urgency of the opportunity. One of Cognition’s values is “every second counts.” They mean it.

An infinite market demands infinite ambition. Scott’s ambition and vision match the scale of the market and we are proud to back him for the second time. Scott, Steven, Walden, Russell and the entire Cognition team are working toward a world in which anyone can turn an idea into software.

I ended my 2011 essay with one line: “I know where I'm putting my money.” Fifteen years later, I still do.

Big one today — Tailwind is joining Shopify.

When I started working on Tailwind over nine years ago, my only goal was to create something that would make it easier to build beautiful interfaces for my own projects. Fast-forward to today and the framework is installed over 110 million times per week and is trusted by many of the world's biggest companies to style products like ChatGPT, X, Cloudflare, Reddit, and Shopify.

We're joining Shopify to give Tailwind a stable long-term home where it will be actively maintained for the millions of people who depend on it.

Why Shopify

We built a great little website template business around Tailwind over the years, but deep down I've always wanted the framework to be developed in service of a real product. A complex application solving important problems for real people, where we'd face the same challenges as our users, and could invent solutions that make the framework better for everyone.

Shopify provides an incredible surface area for us to do this work. Merchants need to be able to design and host beautiful custom storefronts, and manage sales and inventory in a powerful admin area. Their customers need delightful shopping and checkout experiences, and an intuitive way to keep track of their orders and discover new products through the Shop app. Shopify is also on the frontier of where user interfaces need to go next with their explorations into agentic commerce.

Shopify was also one of the very first companies operating at scale to see the potential in Tailwind CSS and start building with it, not only for themselves but betting on it for their customers too. Tailwind is a load-bearing very important part of the stack at Shopify, and they're invested in making sure it's actively maintained and continues to improve and adapt for how the ways we build are changing.

On a less technical note, I'm personally excited because entrepreneurship has completely changed my life. We are not doing enough as a society to produce and empower more entrepreneurs, and I believe deeply in Shopify's mission to help more people start, run, and grow their own business.

What's next

Nothing changes with Tailwind CSS or any of our other open-source projects. Everything will always be MIT-licensed, and our team will continue to lead and maintain these projects for the community with the support of Shopify.

On the commercial side, we'll no longer be trying to grow the business around Tailwind. All existing customers will of course maintain their access to products like Tailwind Plus and ui.sh, but we're closing sign ups for new customers to focus on Tailwind CSS at Shopify.

Thank you so much to everyone who has built something with Tailwind and supported us over these last nine years. I never could've imagined the project would become what it has today, and I truly believe there's no better place for us to continue to do this work than Shopify.

The price of AI is coming down. However, volume growth may not be large enough to counteract the effective decline in prices. The models driving volume increases are relatively cheap, standard models. These trends do not appear to be driven by the adoption of open-source models or Chinese models.

Matt Mullenweg, the founder and CEO of Automattic, best known as the parent company of WordPress.com, has been put on a leave of absence by his board against his will, according to a Slack message seen by TechCrunch.

Earlier Wednesday, Mullenweg posted in a Slack channel visible to all employees that the company’s chief financial officer, Mark Davies, had “conspired” with other board members Ann Dunwoody, Toni Schneider, and Sue Decker to vote to put Mullenweg on a paid leave of absence. The message read:

And the biggest news: I won’t be able to make the [meeting] tomorrow. @Mark Davies has conspired with @Ann Dunwoody, @Toni, and @Sue Decker behind my back and they voted to put me on a paid leave of absence. I voted against that. Wishing Mark and all of you the very best.

To clarify @Mark Davies was voted as interim CEO. I received the resolution 50 minutes before the meeting start, and requested repeatedly for time to have it reviewed by independent legal counsel, even a few hours, which was denied.

A Slack message to the open source WordPress.org community from the project’s executive director, Mary Hubbard, also confirmed Mullenweg’s change of status at the commercial company. However, Hubbard said that WordPress.org was not impacted. “Matt remains the leader of the WordPress project and I remain Executive Director of WordPress. Our teams, priorities, and work continue as planned,” she wrote.

Mullenweg’s Slack message did not make clear why Automattic’s board took this action. Sources told 404 Media, which was the first to report the news, that Schneider — the former CEO of Automattic from 2006 to 2014 — confirmed that the action was initiated by the board. Davies also told employees in the Slack channel that Mullenweg remained on the board of directors.

In an emailed statement to TechCrunch, Automattic confirmed Mullenweg was on leave. “Matt Mullenweg is currently on leave from Automattic,” the email read. “Mark Davies, Automattic’s CFO, will lead the company as interim CEO. The Board has full confidence in Mark’s leadership and in the team’s ability to execute against the company’s priorities.”

Automattic, which owns other brands including Tumblr, WooCommerce, and Pocket Casts, was founded by Mullenweg in 2003 after he co-created the popular open source content management system WordPress.

Over the past few years, Mullenweg and the company have been mired in lawsuits and controversy. The company is in a protracted legal battle with WP Engine, a WordPress web hosting competitor. The lawsuit followed allegations from Mullenweg that WP Engine was profiting from the open source WordPress project without contributing back to the community, and he demanded that it pay 8% of its monthly gross revenue as a royalty fee for using the WordPress brand.

WP Engine sued in October 2024 and accused Automattic and Mullenweg of defamation and abuse of power. Automattic filed counterclaims last year. In February, WP Engine claimed that Automattic intended to target 10 more competitors with royalty claims.

Tensions became so high under Mullenweg’s leadership that he told employees in 2024 to quit, with severance, if they disagreed with him, and 159 did so.

Last year, Mullenweg also threatened to deactivate the accounts of several members of the open source WordPress.org community, after two of them allegedly said they planned to spearhead a new forked version of the open source WordPress project. (Those involved denied their plans, but said Mullenweg went through with their account deactivations anyway, and also deactivated accounts belonging to other contributors, sources told TechCrunch.)

In April 2025, the company laid off 16% of its staff. The cuts included some longtime employees who had been with the company for more than a decade, sources told TechCrunch.

Some alumni have been asking current employees how they feel about the news, a source told TechCrunch. Some employees said they were “ecstatic” or “relieved,” but others had mixed emotions because the move creates more instability.

It isn’t clear why the company’s board made this decision now. One person speculated that the timing may have something to do with Mullenweg’s annual trip to the Burning Man festival, after which he tends to return “with ideas.” This year’s festival ended on September 7.

Mullenweg may have dropped some hints about the cause of the board’s action when he made a series of posts on X on Wednesday evening. In one post, for instance, he wrote: “That they have a pretty good chance of getting a judge to say I spoiled evidence in a malicious way trying to hide something.” That’s in reference to the ongoing legal dispute between Mulleweng and Automattic with hosting company and WordPress contributor WP Engine. In his X posts, Mulleweng vehemently denied the allegation.

In another post, he said he now expects “smear attacks, so get ready for some National Enquirer rumors or hit pieces.” He added that he believed there was one attempt at this when his “Nikon D6 camera was stolen at Burning Man (recovered a day later), but luckily of the 2000+ photos on the memory card there was nothing they can use against me.”

He then shared a few of the pictures of art pieces he took at the famed festival.

Mullenweg did not respond to our emailed request for comment.

Note: This story has been updated to include mention of Mullenweg’s posts on X.

There are massive implications of a race towards superintelligence, and not enough of Congress is focusing on them.

If you wanted to sum up the story of the last year in data, it'd be: AI has gotten really good at writing code, and people are using it for analytics.

“Vibe coded” dashboards, queries, reports, and apps are everywhere. Millions of “non-technical” people are writing billions of lines of code, extremely quickly. At this point, it may actually be the case that more "non-technical" people are writing data code than using legacy BI tools.

And for good reason! The alternative was filing a ticket and waiting two weeks for a chart that answered a slightly different question. Now, you can ask in plain language and get exactly what you want in moments. Nobody is going back.

But this is all very… messy. And the problem isn’t that it’s all code, rather it’s how it’s generated, and where it lives. It’s built on random context and porous permissions, then scattered across one-off chats, HTML files pasted into Slack, and someone’s laptop. It runs once. Nobody reviews it. No one can trace it, reproduce it, or vouch for it.

Are any of these answers actually right?

Let’s be honest: you have no idea, and no way to find out. There’s no record of what context the agent got, which table it grabbed, or which filter it silently dropped before the answer landed in a board deck. The numbers look great – formatted, charted, and confidently narrated. That’s the problem! You can’t evaluate quality you can’t observe, or propagate a metric definition into a thousand one-off chats. The only feedback loop is noticing that people disagree.

Traditional BI tools can’t help – they’re built around proprietary specs and rigid dashboards. Bolting on a chat sidebar just gives people one more thing to vibe-code their way around.

This is AI Data Sprawl: a growing pile of plausible-looking numbers, one-off apps, and millions of lines of ungoverned code. Nobody can trace it. The data team still owns the mess, though!

And so data platform owners face an unhappy choice: let people generate whatever code they want and give up on governance – or lock them into a BI tool they trust precisely because it can't do very much.

This is the big problem everyone is feeling – and no one has really cracked the code.

We're working on this at Hex. Generative Data Apps are a big step – making it easy to use agents with context, controls, and collaboration.

But this story is still incomplete! And filling that out is what we're focused on right now, with the largest and most ambitious project we've ever taken on.

We'll share more soon – we can't wait to show you.

JupyterGIS 0.16 adds real-time collaborative story maps, lazy tile rendering for remote openEO workflows, and native GeoZarr and GeoPackage support. Declarative symbology and Xarray tiling keep visualizations reproducible while allowing large geospatial datasets to stay remote instead of being materialized locally.

Adobe is working on a new web-based graphic design app called Project Oasis—and is recruiting designers to help shape it before launch.

In a September 7 post on Adobe Community, the company described the app as a modern workspace with brand-aware AI built into the workflow. The invitation targets agency, freelance, and in-house designers working on brand identity and marketing materials.

Selected participants will get early access, direct contact with the product team, and a private feedback channel. An NDA is required, so public details remain limited: Adobe has not shared a full feature list, pricing, or a launch date in the announcement.

Drop the design system into any codebase and your coding agent instantly knows how to build consistent, polished interfaces.

Andrew Tulloch, one of the highest-paid employees in the tech industry, had worked in Meta's TBD lab.

Apple's first foldable iPhone, the iPhone Duo, starts at $1,999. The device expands to a 7.6-inch screen when unfolded, and a 5.4-inch screen when used as a conventional handset. It comes in two colors: Star White and Night Sky. More details on the device, as well as Apple's other recent releases, are available in the article.

Purchasers of open source work should be able to check whether a vendor contributes.

Rumors suggest Apple's upcoming foldable iPhone will be noticeably heavier than a standard iPhone, prompting concerns from a physical therapist that the added weight and grip could increase thumb and hand strain during prolonged use. Similar complaints have already been reported by some owners of existing foldable phones, though the risk is mainly associated with repetitive, extended use. Experts recommend taking breaks every 30–45 minutes, stretching your hands, and putting the phone down regularly to reduce fatigue and injury.

boneyard snapshots your real UI and captures a flat list of skeleton "bones" — positioned, sized rectangles that mirror the page exactly.

cmmnts is a lightweight comment section and comments library for any website. Add threaded comments, live updates, and moderation with one script tag.

Tabby has unveiled a new brand identity by Ragged Edge alongside the launch of Tabby Cash, replacing typical fintech visuals with hand-drawn characters, custom bilingual typography, and authentic photography to emphasize financial confidence and approachability. The redesign is built around the idea of helping users take control of their money, using playful visual elements and motion to make everyday financial tasks feel more engaging without sacrificing clarity. After debuting across the app, website, and digital channels, the refreshed identity is now expanding into physical advertising across the UAE.

Lora Zombie, a self-taught painter, built an international following through DeviantArt before exhibiting in Toronto, Los Angeles, London, and New York.

Illustrator Igor Bastidas has spent three years creating weekly illustrations for The Guardian's You Be the Judge column, developing a distinctive style that uses simple, empathetic storytelling to capture the human tension behind everyday disagreements.