Full article content is not available for inline reading.

Read the original article →

The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We're not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance.

In response to a comment in my last post that the cost of formerly specialized performance work has dropped by many orders of magnitude and performance work that used to require a person or team that had a rare set of skills can be done by anyone who can type a few sentences, which means that you can do all sorts of optimizations that used to be too expensive to be worthwhile for all but the largest scale or most lucrative projects, Marc Brooker responded with

Completely agree with your closing point. Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome. (Which comes with all kinds of fun risks and opportunities of its own). Kind of reminds me of FFTW. And a ton of weird old demoscene techniques which were all about being super fast and small on a very particular problem (and often very particular hardware). For example, I remember a demo that re-used its code as textures to get great cache locality.

And Michael Malis has noted

There’s been a meme circulating about how AI doesn’t help because “code was never the hard part.” I think that’s true in some domains, but in others, writing the code absolutely was the hard part. JIT compilers are a great example of that. For many pieces of software, a JIT compiler would help a lot with speeding up the code. The rarity of JIT compilers makes me believe that implementing a JIT compiler historically was too difficult for it to be worthwhile. LLMs have lowered the barrier to entry and made it much easier to write a JIT compiler. This is the thesis behind pgrust. Databases historically were the hardest piece of software to build and were limited because of that. Now, with AI, we can be more ambitious about the type of software we build.

Optimizing for a class of workload

Let's try this out with FRE, the regex engine we built in the last post. Recall that it was created by having an agent loop for a month on improving regex engine performance with access to the rebar regex benchmark suite. This resulted in FRE being heavily overfit to rebar until we warned our agent that we had a holdout benchmark, which caused the agent to generalize the optimizations enough that performance was ok-ish on our holdout. There's no particular reason to use a "software factory" regex engine that doesn't beat a well-tested regex engine on holdout benchmarks, but one notable thing about FRE was that the native AOT compiled version did quite well at longer searches. We noted that, it stands to reason that one could run the native code compiler in another thread while ripgrep was running its normal matcher and then cut over to the native code when it finished compiling and generally get better performance. Of course this will generally result in worse performance for short queries as we lose a thread to compilation, but I care a lot more about how long ripgrep takes when it runs for many seconds or minutes than when it runs for a few seconds, so I'm ok with that tradeoff.

In the same way we could build a regex engine in a few minutes of human time, we can also just try this experiment in a few minutes of human time. I typed a few sentences and an agent went and did the work to allow this to happen (which would be a decent chunk of code surgery for a human) and it ran the benchmark on actual ripgrep queries that come from my codex history. For longer queries, we see a 2x-4x performance improvement here for a few very simple queries. But most queries are more complex, and when we run on representative holdout queries, for queries where AOT should be enabled, we get about a 7% speedup. Not an earth shattering result, but also not a bad outcome for spending a few minutes typing to codex (and it's still doing more optimization and will presumably speed things up further).

Build an index?

This is arguably a silly thing to do, since if we're repeatedly searching for text on a computer, the obvious thing to do to speed that up isn't to write a native code compiler for regex matching, it's to create an index. But the point here is just that this kind of technical work, which used to take a fair amount of time and expertise, can just be done trivially now. And if we wanted to build a text index, it just so happens that I worked on BitFunnel, the Bing search index that was specialized for constant/fast text ingestion that won Best Paper Award at SIGIR, so I can think of a few experiments to try if we're going to build a fast local index of our entire machine.

If I were working at an AI lab and had access to things like SOTA models running on Cerebras chips or other accelerators that greatly increase tok/s and therefore load/demand for search, I might actually survey the existing indexers to see if they're fast enough or if I'd want to build something custom myself. While the open source version of BitFunnel "only" contains a bytecode interpreter and one JIT, the Bing version contains multiple JIT compilers. A project that did that level of optimization used to be a major undertaking, but "I could do that in a weekend" is now actually true for some of these kinds of projects. With my lowly $200/mo account, I think a somewhat faster ripgrep plus any off-the-shelf index is fine, so maybe this fast-ingesting whole-machine index project can be left as an "exercise for the reader (who works at an AI lab)".

Optimizations are cheap

The drastic reduction in the cost of optimizations has been true going back to November 2025 and maybe even somewhat before then with public models. For an example from the GPT-5.1 or 5.2 days, with no knowledge of game AIs, I tried building an Azul AI. This ended up being the strongest AI in the world for the game by a pretty large margin. From reading the thesis that describes the 2nd strongest AI, I think my AI is probably a bit better on the "AI" side of things, but the main place it wins is on optimization despite spending what looks like maybe two orders of magnitude less time and also mostly working on my laptop vs. having a cluster of machines to use. For example, that other AI is single-threaded and my AI is multi-threaded. Since I have a native code version as well as a heinous shared wasm memory + javascript version, and two different search architectures for two different versions, this would've been a fairly large undertaking if done by hand.

There's a bunch of standard stuff it makes sense to do to debug and verify a multithreading algorithm for something like this, like implementing replay from debug logs that can reproduce bugs despite the algorithm being nondetermistic. Doing that alone would've probably been days to a week of work had I done it by hand, but it's exactly the kind of thing an agent can trivially do in a loop. A lot of the tedium it used to take to get a tricky optimization like this working is gone.

This also applies to a lot of other tricky optimizations. From having written CPU microcode, done CPU verification, worked on optimizing a search engine index, etc., I have a lot of experience looking at optimizations and thinking "hmm, this would increase performance by 2%, but it's going to take N person-days to verify that this tricky optimization works" and making a call to go ahead or not based on whether or not it's worth the time to get the optimization working. Now that this N has dropped by a tremendous factor, the number of these kinds of optimizations it makes sense to do goes way up.

Going back to the game AI case, at least for the AI I tried, it seems like you gain about 100 Elo for every doubling in speed. Just adding multithreading alone is enough to wipe the floor with an otherwise comparable AI on a large machine. If you stack in 10-20 more optimizations that seem too annoying for most people to do by hand, the difference in strength is tremendous and it's not really reasonable to try to keep up with a hand-written AI.

To pick another example, as part of preparing for performance interviews, Jamie Brandon tried Anthropic's now public performance takehome. After trying it, he had Claude pick up where he left off and it got a much better result. When he looked at what Claude did that he didn't, he said a lot of the optimizations were things that occurred to him but he hadn't gotten to yet, and "[o]thers were just crazy shit that I would never try unless I was working on this for weeks".

Workload-specific optimization

Coming back to this part of Marc Brooker's comment:

Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome.

This seems pretty inevitable. In another response to my post, Michael Malis of pgrust said something similar:

[discussion of pgrust optimizations] ... I think it's easy enough to create these optimizations that we could look at a customers workload and add them as needed

Without having any kind of framework or setup, right before I started writing this post, I had an agent do workload-specific optimization for my ripgrep queries, which took about 2 minutes for me to launch. The optimizations run on a set of queries, and then there's a later holdout set of queries to run against. After one pass of optimization, the workload optimized version is 2% faster than standard ripgrep on the holdout and it's still getting faster. 2% isn't a big deal for my local ripgrep usage, but considering that this took minutes of time, I'd take a 2% win here.

Appendix: There's no reason for software to be slow anymore

I've been on the record for a long time as strongly disagreeing with the general sentiment that the developers of X are bad and should feel bad for writing slow code because there are a lot of different kinds of programming expertise and not only is it not the case that most programmers don't have performance expertise, it probably doesn't even make sense for them to develop, so of course most projects will have very poor performance compared to what a performance expert can do. I can see why a performance expert would look at the growing gap between how fast a program can be and how fast programs actually are and think that it's ridiculous.

I still don't think someone is bad and should feel bad if their software has poor performance, but I do think that someone who doesn't know anything about performance and is a reasonable user of LLMs should generally be able to create software that has decent performance. If you just tell an LLM to optimize, it will often do all sorts of incorrect things that are really bad that you have to catch, but that's generally true of using the LLM effectively in the first place, so getting decent performance is no longer a specialized skill.

Appendix: How is codex running ripgrep?

Here's some information about the distribution of riprep queries on my machine. The p50 is 55 unicode code points, which is already longer than things I grep for by hand, and the p90 is 119!

But some are funny numerical constructions. The entire pipeline for that was

cargo clippy … | rg 'crates/fre-aot-regex/src/module.rs:' | rg NUMBER_REGEX | head -250

which might be an odd thing for a human to do, but agents seem to do this kind of thing all the time.

On another topic, if we look at how long ripgrep queries took, there are quite a few slow queries, e.g., p99 is almost 1 minute! And p999 is almost 10 minutes! And the maximum query over this time period is approaching 2 hours!

On a draft of the last post, Peter Geoghegan noted

It's also possible for a regex implementation to be faster by supporting fewer features. Some implementations don't support back references, etc.

which is also true here. The workload-specific optimizations done here were fairly superficial because I just gave codex some short instructions and let it do whatever it wanted, but with a more detailed plan, more focused optimizations supporting the common use cases for my queries could be expected to yield larger gains.

AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support

Today, we are announcing the general availability of AWS Glue 6.0, delivering 30% lower pricing than previous AWS Glue versions and introducing full support for Apache Iceberg v3 features. AWS Glue 6.0 is built on a fully modernized runtime, Apache Spark 4.1, Python 3.13, and Scala 2.13, delivering faster performance.

With this release, AWS Glue provides the most complete Iceberg v3 implementation on any fully serverless managed Spark service, along with new capabilities that simplify ETL authoring, improve PySpark performance, and enable real-time streaming with single-digit millisecond latency.

What is new in AWS Glue 6.0

AWS Glue 6.0 delivers the complete Apache Iceberg v3 specification, built on Iceberg 1.11.0. The headline feature is the VARIANT data type with shredding support, which achieves faster query read performance compared to traditional string data type columns for semi-structured data.

With VARIANT shredding, you can store and query JSON, logs, and event data without flattening schemas, eliminating duplicate data copies, custom parsing code, and pipeline breakage when schemas change. This capability transforms how teams handle semi-structured data at scale.

Additional Iceberg v3 capabilities include:

• Geometry and Geography data types: Enable native spatial processing for GIS analytics, location intelligence, and geospatial data pipelines directly on managed Spark.
• Nanosecond-precision timestamps: Support IoT sensor data, scientific computing, and high-frequency financial workloads that require precision beyond standard milliseconds.
• Unknown type handling: Process data with unexpected or evolving schemas without pipeline failures, providing resilience against upstream schema changes.

AWS Glue 6.0 also includes most significant upgrade in Spark 4.1, the modern runtime engine:

• Spark declarative pipelines: Spark Declarative Pipelines introduces a simplified approach to ETL authoring. Data engineers declare transformations, specifying what data should look like, while the engine automatically determines execution order and optimization. This reduces the complexity of pipeline development and eliminates manual orchestration overhead.
• Arrow-native Python UDFs and UDTFs: AWS Glue 6.0 introduces Arrow-native execution for Python User-Defined Functions (UDFs) and User-Defined Table Functions (UDTFs). This eliminates serialization overhead between Python and the JVM, improving PySpark performance for complex transformations.
• Real-time streaming mode: For stateless streaming use cases, AWS Glue 6.0 introduces a real-time streaming mode that achieves single-digit millisecond latency. Built on Spark 4.1’s Real-Time Mode with Glue-optimized execution, this capability supports real-time event processing, low-latency data transformation pipelines, and time-sensitive data routing.

Getting started with AWS Glue 6.0

No API changes are required to use AWS Glue 6.0. You can select the new version using the existing --glue-version parameter in the create-job or update-job APIs through AWS Command Line Interface (AWS CLI), AWS SDK, AWS Glue Studio, Amazon SageMaker Unified Studio, and your preferred IDE.

To get started with AWS Glue 6.0 jobs in the AWS Glue Studio console, open the AWS Glue job and on the Job Details tab, choose the version Glue 6.0 – Supports Spark 4.1, Scala 2, Python 3. You can create new AWS Glue jobs on AWS Glue 6.0 to get the benefit from the improvements, or migrate your existing AWS Glue jobs.

To start using AWS Glue 6.0 on an AWS Glue Studio notebook or an interactive session through a Jupyter notebook, set 6.0 in the %glue_version magic. You can also upgrade existing jobs to Glue 6.0 using the Spark upgrade agent on AWS Glue Studio or use the auto-upgrade feature in their existing Glue jobs to automatically upgrade them to Glue 6.0.

To learn more, visit the AWS Glue 6.0 version detail and Migrating AWS Glue for Spark jobs to AWS Glue version 6.0 in the AWS documentation.

Now available

AWS Glue 6.0 is generally available today in all AWS Regions where AWS Glue operates. For Regional availability and a future roadmap, visit the AWS Capabilities by Region. If you want to call APIs, search documentation, find regional availability, and check troubleshooting about this new feature, try using the AWS MCP Server and plugins with your preferred AI tool.

You pay an hourly rate, billed by the second, for crawlers (discovering data) and extract, transform, and load (ETL) jobs (processing and loading data). For the AWS Glue Data Catalog, you pay a simplified monthly fee for storing and accessing the metadata. The first million objects stored are free, and the first million accesses are free. To learn more, visit AWS Glue Pricing page.

Give it a try in the AWS Glue Studio console, and send feedback to AWS re:Post for AWS Glue or through your usual AWS support contacts.

The folks at Microsoft Azure recently wrote up a post incident review for a networking issue in their West U.S region. From the included timeline, it looks like the impact was on the order of five hours. It’s a pretty short write-up, but let’s take a look at the contributors.

On 23 July 2026, a break-fix repair was initiated on an optical device to address a network reliability risk.

The first contributor mentioned in the write-up was work that was done to repair a device in their networking stack. Here I can’t help but think of the first bullet in my conjecture on why reliable systems fail. They made a change to the system in order to fix an ongoing problem, and due to a set of circumstances, things got worse rather than better.

A defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices egressing a specific datacenter.

The second contributor mentioned was a (presumably) latent defect in their system. Note the irony of the failure mode here: I suspect this blast radius analysis system usually contributes to reliability, but in this case it hurt reliability by increasing the blast radius.

The safety validation step, which is designed to confirm that at least one of the two redundant datacenter paths remains available, ran but incorrectly concluded the operation was safe.

The third contributor mentioned was a safety check (good!) that passed even though the action was unsafe (bad!).

The checks validated each device individually rather than evaluating the aggregate effect of isolating all devices at once, a scenario that was not accounted for because the system was never designed to process a full datacenter’s worth of devices in a single request.

The reason it failed was due to an interaction with the second contributor: the blast radius being all of the optical devices egressing the datacenter. The designers never envisioned that the check would have to handle the sort of scenario that occurred as a result of the blast radius analysis system defect.

As a result, routes were withdrawn from multiple devices simultaneously, disrupting connectivity between the datacenter and the WAN – therefore impacting traffic entering or leaving the West US region.

It sounds like this change effectively disconnected the West US datacenter from the internet.

Once the route withdrawals took effect at 14:44 UTC, physical links and routing adjacencies continued to appear healthy, which initially masked the correlation between the break-fix activity and the connectivity disruption

Here we have our fourth contributor: the operators were receiving misleading signals from the system. The links and routes looked healthy, even though connectivity was broken.

The impact presented as a WAN routing anomaly, as third-party networks could not reach Azure in the region, rather than as a datacenter connectivity failure.

Our fifth contributor is another flavor of misleading signals. The symptoms presented as a routing issue between Azure and third-parties.

Although all physical work in the region was stopped, our engineers could not correlate to this recent change because the preparation activities in advance of the break-fix did not succeed, so the physical layer and traffic appeared healthy.

This is the sixth contributor mentioned in the writeup. The writing is a little oblique here, but I think what they are saying is that the repair event did not show up in their event log because the repair event didn’t actually complete. It sounds like the preparation activities were the ones that triggered the incident. But, because the repair event didn’t actually happen, the operators looking for events that correlate in time with the onset of the incident didn’t see the triggering event because it didn’t show up in the log of events. That’s my best guess, anyways.

Our automated recovery and rollback system detected the device failures, and attempted multiple retries to restore the affected devices. However, because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful.

This is the seventh and final contributor mentioned. Azure has an automated recovery and rollback system (good!), but the failure mode in this case prevented automated rollback from succeeding (bad!).

As always, I’d love to know more about how the operators identified what the failure mode actually was, and how they traced it back to the optical device repair work.

AI-accelerated attacks are redefining the threat landscape, but many of them still rely on one of the oldest tactics in the book: exploiting known vulnerabilities. The difference today is speed. Vulnerabilities that once took skilled hackers months or weeks to exploit can now be weaponized in hours or minutes. This acceleration is forcing organizations to rethink how they identify and remediate risk.

To help contend with the high volume and velocity of attacks, the Cybersecurity and Infrastructure Security Agency (CISA) released Binding Operational Directive (BOD) 26-04: Prioritizing Security Updates Based on Risk. This mandate changes how federal agencies must prioritize vulnerability remediation. In this post, we’ll explore:

• The requirements of BOD 26-04
• The challenges of achieving compliance
• How the Datadog Runtime Prioritization Engine can help customers meet their compliance obligations

What is BOD 26-04?

BOD 26-04 codifies risk-based prioritization and patching timelines for federal agencies based on four key variables:

1. Asset exposure: Is the vulnerable asset publicly exposed?
2. Known exploited vulnerability (KEV) status: Is the vulnerability, as identified by a Common Vulnerabilities and Exposures identifier (CVE ID), in CISA’s Known Exploited Vulnerabilities Catalog?
3. Exploit automation: Is an adversary able to automate all the steps necessary to exploit the vulnerability?
4. Technical impact: Does an adversary gain partial control or total control of the vulnerable asset after exploitation of the vulnerability?

BOD 26-04 establishes remediation timelines that are dependent on the answers to these questions. The most critical vulnerabilities require remediation within 3 days and include requirements for forensic triage and investigation. Conversely, vulnerabilities that do not meet any of the directive’s risk criteria are deprioritized and can be remediated during the next system upgrade. The following table shows the remediation timelines:

BOD 26-04 focuses patching efforts on the areas of highest risk rather than treating all vulnerabilities and systems equally. It also revokes BOD 19-02, eliminating the requirement to use the Common Vulnerability Scoring System (CVSS) as the primary mechanism for vulnerability prioritization.

While BOD 26-04 applies specifically to federal agencies, its underlying message is relevant to every security team: Business-aligned prioritization is critical as attack volumes increase. Organizations need to focus on the vulnerabilities that are most likely to be exploited and most likely to affect critical business operations.

Challenges of BOD 26-04

BOD 26-04 assumes organizations can answer questions such as the following:

• Is the vulnerable asset actually exposed?
• Is the vulnerable code running in production?
• Is the vulnerability likely to be exploited?
• Would exploitation impact a critical business function?
• Who is responsible for remediating the vulnerability?

These questions seem straightforward, but answering them at scale across modern cloud environments can be difficult. Many organizations have vulnerability data but lack the contextual depth needed to determine which findings represent meaningful risk. Instead, they rely on manually maintained asset metadata, point-in-time scans, and disconnected tooling.

Even when organizations successfully identify high-priority vulnerabilities, remediation often stalls because security and engineering teams operate in different tools, with different workflows and competing priorities. To understand what issues are critical and how to take action, teams need runtime context in a shared platform.

How the Datadog Runtime Prioritization Engine can help

The Datadog Runtime Prioritization Engine combines runtime behavior with exploitability, exposure, and business context from Datadog observability and security telemetry data to identify the small percentage of findings that pose real, exploitable risk. As part of Datadog Cloud Security, the Runtime Prioritization Engine makes prioritization transparent and explainable by evaluating findings across five dimensions:

• Reachability: Is the vulnerable component actually running in production?
• Exposure: Can attackers realistically reach the affected resource?
• Exploitability: Is there evidence that the vulnerability is likely to be exploited, such as public exploit code, high Exploit Prediction Scoring System (EPSS) scores, or inclusion in CISA’s Known Exploited Vulnerabilities Catalog?
• Business criticality: Would a successful compromise impact a critical business service, sensitive data, or a high-value asset?
• Actionability: Is ownership known, and is a fix available so that remediation can happen quickly?

For example, the Runtime Prioritization Engine automatically infers business-critical assets, known as crown jewels, from observability signals such as service dependencies, APM traces, traffic patterns, service level objectives (SLOs), and incident history. This information enables teams to understand not just whether a vulnerability exists, but whether exploitation would impact a critical business function.

The Runtime Prioritization Engine also infers ownership by using operational metadata such as service ownership, deployment information, on-call configurations, source control integrations, and service catalog data. This metadata is combined with runtime package usage, exploit intelligence, and network exposure analysis to give security teams a continuously updated picture of which findings deserve immediate attention, who is responsible for addressing them, and how urgently they should be addressed.

By identifying ownership automatically and integrating directly with engineering workflows, the Runtime Prioritization Engine helps ensure that prioritized findings don’t stop at triage. Teams can route findings and coordinate remediation by using their existing collaboration tools, giving responders the context they need to understand the risk and take action.

The Datadog Security MCP toolset extends the Runtime Prioritization Engine’s capabilities by enabling AI agents to securely access Datadog security context and remediation processes. Teams can use their preferred AI agent to analyze, triage, investigate, and correlate signals and findings while using Datadog as the system of record.

Accelerate prioritization and remediation for BOD 26-04 with Datadog

BOD 26-04 changes how federal agencies must prioritize and fix cyber vulnerabilities, but the directive’s sentiment is valuable to all security teams: Identify and prioritize the highest risks, and remediate them quickly. The Datadog Runtime Prioritization Engine helps teams achieve these goals with a combination of runtime prioritization, ownership intelligence, workflow integration, and AI-assisted operations that aligns security efforts with business risk. Instead of triaging thousands of findings based primarily on CVSS scores, teams can focus on remediating the vulnerabilities that are running, reachable, exploitable, impactful, and actionable.

For more information, read the Runtime Prioritization Engine documentation. To stay updated about the latest features, join the Runtime Prioritization Engine Preview program.

If you’re new to Datadog, you can sign up for a 14-day free trial to start identifying and prioritizing your security risks.

This post is for informational purposes only. Nothing here constitutes legal advice, a compliance assessment, or a warranty of any kind. While Datadog offers powerful tools to assist customers in achieving their own compliance with government and industry standards, customers are responsible for their own compliance obligations.

A good SQL index comes from the queries your application runs, not from the table schema. Composite indexes need the right column order: equality columns first, then the column you sort or range on. EXPLAIN ANALYZE is how you verify it: a sequential scan over 1 million comments takes 17ms, and the right composite index answers in 0.04ms.

What does a good SQL index look like?

The answer will vary based on your queries and access paths. The only way to confidently know is to examine the query plans with EXPLAIN ANALYZE and figure out from there which index might help.

So let's do exactly that. I seeded a Postgres 18 instance in Docker with an issue tracker: 100 users, 10,000 issues, and 1 million comments. By the end, one query drops from 436ms to half a millisecond.

What Is a SQL Index?

An index stores your chosen columns in sorted order, with every entry pointing back to its full row. The default kind in every major database is the B-tree: a shallow tree, a few levels deep even at millions of rows. A sequential scan reads all 1 million comments; an index scan descends those few levels and fetches only the matches.

Start With the Query, Not the Table

You don't pick indexes by staring at the schema; they come from the queries your application actually runs.

My comments table serves three access patterns:

• All comments by a user
• All comments for an issue
• Comments for an issue from one user, newest first, last month only

Reading the First Plan

The first pattern, with no index beyond the primary key:

EXPLAIN ANALYZE
SELECT COUNT(*)
FROM comments
WHERE user_id = 1;

---
Finalize Aggregate
  ->  Gather
        ->  Partial Aggregate
              ->  Parallel Seq Scan on comments  (actual time=0.010..11.727 rows=3356.67 loops=3)
                    Filter: (user_id = 1)
                    Rows Removed by Filter: 329977
Execution Time: 17.066 ms

EXPLAIN ANALYZE runs the query for real and prints the plan Postgres used: a Parallel Seq Scan reads all 1 million rows to count 10,070, in 17ms.

Create the index and rerun the query:

CREATE INDEX ix_comments_user_id
ON comments (user_id);

Aggregate
  ->  Index Only Scan using ix_comments_user_id on comments  (actual time=0.024..0.348 rows=10070.00 loops=1)
        Index Cond: (user_id = 1)
        Heap Fetches: 0
Execution Time: 0.612 ms

17ms down to 0.6ms. It's an Index Only Scan because the index alone can answer a COUNT(*): Postgres never touches the table.

Column Order Is Everything

The third access pattern is the interesting one:

SELECT *
FROM comments
WHERE issue_id = 10
  AND user_id = 29
  AND created_at >= NOW() - INTERVAL '1 month'
ORDER BY created_at DESC;

With no index, it's another sequential scan: 16.6ms. With single-column indexes on issue_id and user_id, Postgres intersects them with a BitmapAnd and still sorts the survivors: 0.6ms, in three steps.

A composite index answers the whole query in one motion:

CREATE INDEX ix_comments_issue_user_date
ON comments (issue_id, user_id, created_at DESC);

Index Scan using ix_comments_issue_user_date on comments  (actual time=0.019..0.026 rows=2.00 loops=1)
  Index Cond: ((issue_id = 10) AND (user_id = 29) AND (created_at >= (now() - '1 mon'::interval)))
Execution Time: 0.039 ms

All three conditions moved into the Index Cond, and the Sort is gone: the index already returns rows ordered by created_at DESC. Runtime: 0.04ms, over 400x faster.

A composite index sorts by its first column, then the second within equal values, then the third. Postgres jumps straight to the issue_id = 10, user_id = 29 section and reads it in order.

Column order also decides what else the index can serve: issue_id alone works, issue_id plus user_id works, but user_id alone doesn't (its values are scattered across the whole tree). This is the leftmost prefix rule, and it's why the index on user_id stays.

The rule of thumb: equality columns first, then the column you sort or range on.

The Query Our New Index Can't Serve

Every issue tracker runs this dashboard query: the 25 newest open issues, each with its latest comment, fetched by a LATERAL subquery:

SELECT i.id, c.body, c.created_at
FROM issues i
CROSS JOIN LATERAL (
  SELECT body, created_at
  FROM comments
  WHERE issue_id = i.id
  ORDER BY created_at DESC
  LIMIT 1
) c
WHERE i.status = 'open'
ORDER BY i.created_at DESC
LIMIT 25;

Nested Loop  (actual time=0.790..352.076 rows=6537.00 loops=1)
  ->  Seq Scan on issues i  (rows=6537.00 loops=1)
  ->  Limit  (rows=1.00 loops=6537)
        ->  Sort  (actual time=0.053..0.053 rows=1.00 loops=6537)
              ->  Bitmap Index Scan on ix_comments_issue_user_date  (loops=6537)
Execution Time: 435.794 ms

The composite index gets used, but its entries are sorted by user_id before created_at, so a Sort runs 6,537 times, once per open issue: 436ms.

Column order strikes again. For this access path, created_at must come right after issue_id:

CREATE INDEX ix_comments_issue_date
ON comments (issue_id, created_at DESC);

Each probe becomes a one-row index scan: 25ms. But the LIMIT still can't stop the loop, because issues arrive unsorted. One more index streams them newest-first:

CREATE INDEX ix_issues_status_date
ON issues (status, created_at DESC);

Limit  (actual time=0.086..0.465 rows=25.00 loops=1)
  ->  Nested Loop  (actual time=0.085..0.463 rows=25.00 loops=1)
        ->  Index Scan using ix_issues_status_date on issues i  (rows=25.00 loops=1)
        ->  Limit  (rows=1.00 loops=25)
              ->  Index Scan using ix_comments_issue_date on comments  (rows=1.00 loops=25)
Execution Time: 0.489 ms

Every node reads only what it returns: 25 issues, 25 probes, one comment each. 0.5ms, nearly 900x faster.

What Do Indexes Cost?

Every insert, update, and delete now maintains every index, so each one you add slows writes a little. They take disk space, too:

SELECT indexrelname AS index_name,
       pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE relname = 'comments';

Each composite index weighs 30 MB for 1 million comments, against about 7 MB per single-column one. And (issue_id, created_at DESC) makes the plain issue_id index redundant, so drop it. Index the queries you actually run, not the ones you might run someday.

Summary

• Design indexes from your queries, not your tables.
• Composite indexes need the right column order: equality columns first, then the sort column.
• LIMIT only helps when an index feeds it rows already in order.
• EXPLAIN ANALYZE is the proof. Read the plan, not just the timing.
• Every index costs writes and space.

Frequently Asked Questions

What is a composite index?

A composite index stores several columns together, sorted by the first column, then by the second within equal values, then by the third. Postgres can jump straight to the section matching the leading columns and read it in order.

What order should columns go in a composite index?

Equality columns first, then the column you sort or range on. On the demo query, an index on (issue_id, user_id, created_at DESC) moved all three conditions into the index condition and removed the sort, going from 16.6ms to 0.04ms.

What is the leftmost prefix rule?

A composite index only serves queries that use its leading columns. With (issue_id, user_id, created_at DESC), filtering on issue_id works and issue_id plus user_id works, but user_id alone doesn't, because its values are scattered across the whole tree.

Why is my query still doing a sort when it uses an index?

The index returns rows in the wrong order. A dashboard query hit the index on (issue_id, user_id, created_at DESC), which sorts by user_id before created_at, so Postgres ran a sort once per open issue, 6,537 times, and the query took 436ms.

What is an Index Only Scan in Postgres?

An Index Only Scan means the index alone can answer the query, so Postgres never touches the table. Counting comments for one user ran as an Index Only Scan with zero heap fetches, in 0.6ms instead of a 17ms sequential scan.

Do indexes slow down writes?

Yes. Every insert, update, and delete maintains every index on the table, so each one you add slows writes a little. Indexes take disk space too: each composite index here weighed 30 MB for 1 million comments, against about 7 MB per single-column index.

DuckDB v2.0: Your Database Deserves a Better Parser

TL;DR: DuckDB v2.0 replaces its PostgreSQL-derived SQL parser with a PEG-based parser that is easier to evolve and can be extended at runtime.

At DuckDB, one of our goals is to make working with a database system as easy as possible. Users interact with the system through the widely understood Structured Query Language (SQL). Previous blog posts have covered DuckDB’s friendly SQL, including GROUP BY ALL and column selection using SELECT * EXCLUDE (...). Before DuckDB can execute a query using these features, however, it first has to determine whether its syntax is valid. That is the job of the parser, and in DuckDB v2.0 we are completely replacing it without you noticing.

What is the Role of a Parser?

At a high level, DuckDB processes a SQL query through the following stages:

In this blog, we focus on the tokenizer, parser, and transformer:

• Tokenizer: This is the first step and is responsible for splitting up the raw input string into tokens. These can be of various categories, for example: KEYWORD, NUMBER, or IDENTIFIER. It is also where comments, in SQL denoted with either -- or /* */, are recognized and skipped.
• Parser: The parser determines whether these tokens follow DuckDB's grammar and produces a ParseResult tree.
• Transformer: Converts the generic parse results into DuckDB’s internal abstract syntax tree (AST), forming structures such as SQLStatement, TableRef, and ParsedExpression. The resulting AST is passed on to the binder.

The parser determines whether a query is syntactically valid, while the binder determines whether the tables, columns, and functions it refers to actually exist.

Consider the following query:

SELECT *
WHERE true
FROM range(1);

Parser Error:
syntax error at or near "FROM"

LINE 3: FROM range(1);
        ^^^^

Every individual token in this query is valid, but the clauses occur in an order that DuckDB’s grammar does not accept. Friendly SQL allows both SELECT-first and FROM-first syntax, but it does not allow the clauses to appear in an arbitrary order.

By comparison, the following query is syntactically valid, so it passes the parser and transformer. However, it fails later in the binder because the table missing_table does not exist.

FROM missing_table;

Catalog Error:
Table with name missing_table does not exist!

LINE 1: FROM missing_table;
             ^^^^^^^^^^^^^

The DuckDB SQL Dialect

Although a SQL standard exists, every database system supports different parts of the standard and adds its own syntax and behavior. The resulting variants are commonly referred to as SQL dialects. Examples include the dialects supported by PostgreSQL, Oracle, GoogleSQL for BigQuery, MySQL, MariaDB, SQLite, Spark SQL, and, of course, DuckDB.

DuckDB’s SQL closely follows PostgreSQL conventions, but it has evolved considerably over the years. We have added features of our own, such as GROUP BY ALL, as well as features inspired by other database systems. At the same time, DuckDB does not implement every aspect of PostgreSQL’s behavior. DuckDB therefore speaks its own SQL dialect, which we will refer to as DuckSQL in this post, even though it remains strongly influenced by PostgreSQL.

This distinction is important when talking about the parser. The SQL dialect that DuckDB accepts and the implementation used to parse that SQL are two separate things. For DuckDB v2.0, we are replacing the parser implementation and rewriting its grammar. What we are not replacing is DuckSQL itself.

Outgrowing the PostgreSQL-Derived Parser

When DuckDB started out, it made a lot of sense to use the PostgreSQL-derived parser and grammar. This parser was already part of the first commit to DuckDB in 2018. It gave DuckDB a mature, battle-tested SQL grammar based on syntax that many users were already familiar with. We adapted the parser to our needs and added a Transformer that converted the resulting PostgreSQL-style parse tree into DuckDB’s internal AST.

However, over the years this parser also came with some downsides. Extending DuckSQL meant modifying the underlying YACC/Bison grammar. Because Bison generates an LALR(1) parser, seemingly small additions to the grammar can interact with existing rules and introduce shift/reduce or reduce/reduce conflicts. As DuckSQL grew, making changes to the grammar therefore became increasingly difficult.

This was one of the motivations behind our earlier blog post on runtime-extensible SQL parsers. In that post and the accompanying CIDR paper, we explored whether Parsing Expression Grammars (PEGs) could provide a better foundation for an extensible database parser. At the time, the PEG parser was still an experimental prototype capable of parsing only a subset of SQL.

A Primer on PEG Parsers

Before looking at how we turned the prototype into a production parser, let us briefly revisit how a PEG describes a language.

A PEG consists of named rules that describe how an input should be matched. Consider the following rules from DuckDB’s new grammar:

SelectFrom <- SelectFromClause / FromSelectClause
SelectFromClause <- SelectClause FromClause?
FromSelectClause <- FromClause SelectClause?

The <- operator defines a rule, / specifies a choice between alternatives, and ? makes an element optional. Together, these rules state that DuckSQL accepts both a traditional SELECT-first query:

SELECT *
FROM range(1);

And DuckDB’s Friendly SQL FROM-first equivalent:

FROM range(1)
SELECT *;

A PEG evaluates alternatives in order. When matching SelectFrom, the parser first attempts SelectFromClause. If that does not match, it attempts FromSelectClause. The first successful alternative is selected. As a result, PEG grammars do not have the same shift/reduce and reduce/reduce conflicts as LALR grammars. Instead, alternatives are ordered explicitly, and that order forms part of the grammar’s behavior.

We are not the only ones changing to a PEG-based parser. Python switched from its LL(1) parser to a PEG-based parser in Python 3.9, also motivated by the additional flexibility PEG provides to evolve the language.

In DuckDB, these rules operate on the tokens produced by the tokenizer. The matcher applies the grammar rules to those tokens and constructs a generic ParseResult tree, which is subsequently transformed into DuckDB’s internal AST.

Going from Prototype to Production

The research prototype demonstrated that a PEG-based SQL parser was feasible. Replacing DuckDB’s existing parser, however, required considerably more than parsing a subset of SQL. The new parser had to accept all of DuckSQL and produce the same AST expected by DuckDB’s binder.

The PEG grammar was first introduced in DuckDB v1.2, where it handled autocomplete in the CLI. Later, in DuckDB v1.5, we introduced the complete PEG parser as an experimental, opt-in feature. We also used it for an April Fools' joke that made DuckDB speak Dutch. Since then, the grammar, matcher, and transformer have been steadily improved to make the PEG parser the default for DuckDB v2.0.

Among other things, the parser had to support:

• Every statement and expression type: Supporting the complete DuckSQL dialect includes both common syntax as well as the less frequently used statements and expressions.
• Operator precedence and associativity: For example, SELECT true OR true AND false; must be interpreted as (true OR (true AND false)), because AND binds more tightly than OR.
• Correct keyword classification: Some keywords, such as SELECT, are RESERVED and cannot be used as unquoted table or column names. Other keywords may be used as identifiers depending on their context.
• Compatibility with DuckDB’s internal AST: The PEG transformer must produce the same DuckDB AST structures as the transformer for the PostgreSQL-derived parse nodes wherever the language behavior is intended to remain unchanged.
• Correct error reporting: For an invalid query, the parser should report where parsing failed and, where possible, provide context and a useful indication of what went wrong. Ideally, it should do so without pointing to a manual.
• Performance on unusual inputs: Besides keeping normal parsing fast, we also had to make sure that malformed queries do not suddenly take a long time to parse.

Avoiding Repeated Work with Packrat Parsing

One issue we encountered was repeated work during backtracking. A naïve PEG matcher can evaluate the same grammar rule at the same token position many times while trying different alternatives. For certain malformed inputs, the amount of repeated work can grow exponentially.

We encountered this with queries containing a large number of unmatched opening parentheses:

SELECT ((((((((((((((((((;

With the experimental PEG parser shipped in v1.5, adding one more opening parenthesis approximately doubled the parsing time:

18 opening parentheses:  5.303 seconds
19 opening parentheses: 10.640 seconds

We addressed this using packrat parsing, a memoization technique commonly used with PEG parsers. For each memoized matcher, we store the result of applying it at a particular token position. If the parser later attempts the same matcher at the same position, it reuses the cached result instead of evaluating it again.

With packrat parsing enabled, the same malformed query was rejected almost instantly:

19 opening parentheses: 0.001 seconds

As a result, a memoized matcher is evaluated at most once at a particular token position, removing the repeated work that caused the exponential behavior in this example. This requires additional memory while parsing, but that is a worthwhile trade-off for avoiding cases such as this.

Turning the prototype into a production parser involved much more than translating the grammar. The new parser had to cover the complete DuckSQL dialect, preserve DuckDB’s existing AST, remain compatible with existing queries, and handle both valid and malformed input efficiently.

The resulting architecture replaces the PostgreSQL-derived parser front end, while the binder and the remainder of DuckDB’s query-processing pipeline continue to operate on the same internal AST.

Evolving DuckSQL

With the PEG parser now in place for DuckDB v2.0, we have also continued to extend DuckSQL with new syntax.

One example is the new expression-statement syntax. Until now, executing a query consisting only of expressions always required writing a SELECT:

SELECT date: current_date(), time: current_localtime();

With an expression statement, the SELECT can be omitted:

date: current_date(), time: current_localtime();

As a bonus, this also works with prefix aliases.

Another example is the new CONNECT statement, introduced for Quack. It allows you to connect to a remote database and route subsequent queries to it until you run DISCONNECT:

CONNECT 'postgres://localhost/mydb';
SELECT count(*) FROM orders; -- Runs on the PostgreSQL server
DISCONNECT;

There will also be new syntax for working with external resources. This will allow you to manage resources that live outside DuckDB through an extension. You will be able to create, register, inspect, connect to, or destroy a resource all from within DuckDB:

CREATE EXTERNAL RESOURCE '<resource-type>' AS <name> (...);
REGISTER EXTERNAL RESOURCE '<resource-type>' AS <name> FROM <handle>;

SHOW EXTERNAL RESOURCES;

CONNECT TO EXTERNAL RESOURCE <name>;

DESTROY EXTERNAL RESOURCE <name>;

We have also extended COPY TO with PARTITION BY and ORDER BY syntax:

COPY orders TO 'orders'
(
    FORMAT parquet,
    PARTITION BY (year, month),
    ORDER BY (order_date)
);

These additions would also have been possible with the old PostgreSQL-derived parser, but adding them would have been considerably more cumbersome. The PEG grammar makes it easier for us to continue evolving DuckSQL.

So far, these rules are all part of DuckSQL itself. The next step is allowing extensions to add rules of their own.

Extending the Parser

Extensions are a central part of DuckDB. They can already add scalar and table functions, optimizer rules, query-plan rewrites, and even custom physical operators.

Extensions that add new syntax already exist, such as psql and duckpgq, but under the hood they work as fallback parsers. DuckDB first tries to parse the query itself and only calls the extension if that fails. This works well for self-contained syntax, but an extension that wants to add syntax inside SQL also has to parse the surrounding SQL itself. These fallback parsers also make it impossible to combine the syntax of multiple extensions.

With the PEG parser, extensions can instead extend individual parts of DuckDB’s parser. They can extend the tokenizer, add grammar rules, and register custom matchers while continuing to reuse the rest of DuckSQL.

Warning The API shown below is still a preview and may change before DuckDB v2.0. You can follow the ongoing development on GitHub.

To make this concrete, we use Google’s pipe query syntax. This is an extension to SQL that adds piped data flow syntax. Pipe syntax expresses a query as a sequence of operators, where each operator consumes the result of the previous one.

FROM produce
  |> WHERE
        item != 'bananas'
        AND category IN ('fruit', 'nut')
  |> AGGREGATE COUNT(*) AS num_items, SUM(sales) AS total_sales
     GROUP BY item
  |> ORDER BY item DESC;

A simplified PEG grammar for this needs a handful of rules:

PipeSelectAtom <- PipeSource PipeStage+
PipeSource <- FromClause / SelectStatementType / SelectParens
PipeStage <- '|>' PipeOperator
PipeOperator <- PipeAggregate / PipeAggregateGroupOnly / PipeWhere / PipeSelect / PipeExtend / PipeDistinct / PipeOrderBy / PipeLimit
PipeWhere <- WhereClause
PipeSelect <- 'SELECT' TargetList
PipeExtend <- 'EXTEND' TargetList
PipeDistinct <- 'DISTINCT'
PipeOrderBy <- OrderByClause
PipeLimit <- LimitClause OffsetClause?
PipeAggregate <- 'AGGREGATE' TargetList GroupByClause?
PipeAggregateGroupOnly <- 'AGGREGATE' GroupByClause

Here, + means that PipeStage must occur one or more times, so a pipe query must contain at least one pipe operator.

This grammar can reuse existing rules, such as GroupByClause, to reduce the amount of grammar the extension needs to define. An extension can still define its own rule where DuckDB’s existing syntax does not fit.

Registering the Grammar

Defining just the PEG rules does not yet make them part of DuckDB’s grammar. The extension must also specify (1) the existing grammar rule it wants to extend and (2) the transformer rules that convert the new syntax into DuckDB’s AST.

In the current prototype, certain grammar rules expose extension points. Pipe SQL registers PipeSelectAtom as an additional alternative for SelectAtom, together with the new keywords AGGREGATE and EXTEND.

static void LoadInternal(ExtensionLoader &loader) {
    ParserExtension extension;
    extension.grammar_extension.grammar = PIPE_SQL_GRAMMAR;
    extension.grammar_extension.select_atom_rule = "PipeSelectAtom";

    extension.grammar_extension.RegisterSelectAtomTransformer(
        "PipeSelectAtom",
        TransformPipeSelectAtom
    );

    loader.RegisterKeyword(
        "aggregate",
        ExtensionKeywordCategory::RESERVED
    );
    loader.RegisterKeyword(
        "extend",
        ExtensionKeywordCategory::RESERVED
    );

    loader.RegisterParserExtension(std::move(extension));
}

By registering this alternative, the resulting grammar is effectively:

SelectAtom <-
    PipeSelectAtom /
    SelectParens /
    SelectStatementType

The extension alternative is now tried first. If no pipe syntax is present, it fails without consuming any tokens and the query is parsed with the built-in alternatives.

Transforming the Result

Adding a grammar rule only gets us as far as a ParseResult. The extension still needs to transform that result into the DuckDB AST that is expected by the binder. Since PipeSelectAtom extends SelectAtom, its transformer returns a SelectStatement:

static unique_ptr<SelectStatement>
TransformPipeSelectAtom(PEGTransformer &transformer, ParseResult &parse_result) {
    auto &pipe = parse_result.Cast<ListParseResult>();

    // PipeSelectAtom <- PipeSource PipeStage+
    auto statement = TransformPipeSource(transformer, pipe.GetChild(0));

    auto &stages = pipe.Child<RepeatParseResult>(1);
    for (auto &stage : stages.GetChildren()) {
        ApplyPipeStage(transformer, stage.get(), *statement);
    }

    return statement;
}

The shape of the ParseResult follows the grammar rule we defined earlier. PipeSelectAtom contains a PipeSource and one or more PipeStages. We first transform the PipeSource into a DuckDB SelectStatement. Each PipeStage is then applied to that statement in order. The resulting SelectStatement is then returned and can continue through the rest of the parser's pipeline and eventually on to the binder.

This is where reusing DuckDB's existing grammar becomes especially useful. The extension only needs to transform the new syntax it introduced. When it reuses an existing DuckDB grammar rule, such as GroupByClause, it can also reuse the corresponding transform function instead of having to implement GROUP BY itself.

This is an important difference from the fallback parsers that are available today. An extension no longer needs to implement expressions, table references, GROUP BY clauses, and the rest of SQL itself. Instead, it can add only the syntax it needs and reuse DuckDB’s grammar and transformations for everything else.

Executing Pipe SQL

With the extension registered, we can now execute queries using the new pipe syntax. For example, we can combine the pipe operators added by the extension with existing DuckSQL features such as range() and prefix aliases:

FROM range(6) t(i)
  |> WHERE i % 2 = 0
  |> SELECT i, doubled: i * 2
  |> ORDER BY i DESC;

┌───────┬─────────┐
│   i   │ doubled │
│ int64 │  int64  │
├───────┼─────────┤
│     4 │       8 │
│     2 │       4 │
│     0 │       0 │
└───────┴─────────┘

The extension only defines the pipe-specific syntax. Expressions, table references, WHERE, SELECT, ORDER BY, and other reused rules are still parsed and transformed by DuckDB itself. This means that new syntax can be combined with DuckSQL without the extension having to implement the rest of SQL again.

To Conclude

With DuckDB v2.0, we are replacing the PostgreSQL-derived parser with a new PEG parser. Existing DuckSQL queries should continue working as before. Under the hood, however, the new parser gives us something that is easier to evolve and designed for runtime extensibility.

The runtime grammar extension API shown in this post is still a preview and may change before v2.0 is released. However, the underlying idea is already working. Extensions can add their own syntax directly to DuckDB's grammar while reusing its existing rules and transformations. This means they no longer need to parse the rest of SQL themselves.

We are excited to see what new syntax the community will create. In the meantime, we will continue evolving DuckSQL and improving the parser.

If you do find an existing query that behaves differently with the PEG parser, please let us know by filing an issue.

One AI Output Is an Example, Not an Evaluation

Summary: One output cannot establish how well an AI system performs. Evaluate with multiple representative inputs, repeated runs, and confidence intervals.

Suppose you ask an AI customer-service system the same question several times: “Can I return an opened product after 30 days?” One answer may explain the policy accurately. Another may omit an important exception. A third may confidently promise a refund that the customer is not entitled to receive. Which of these answers represents the system? The answer is: all of them, taken together — and none of them, taken in isolation.

Yet teams often evaluate AI systems by running them once, inspecting the result, and drawing conclusions about what AI can do or how it should be used. This is not because they are lazy or careless. It is because decades of deterministic software have taught us that a feature that works once will work the same every time. AI systems offer no such guarantee. One good output demonstrates that a system can perform a task. It does not show how often or how reliably the system will do so.

AI Outputs Are Nondeterministic

A nondeterministic system can produce different outputs when given the same input.

Language models assign probabilities to possible next tokens and select among them as they generate a response. As a result, submitting the same input repeatedly may produce answers that differ in wording and quality.

This variation can be substantial. Research comparing repeated generations from the same language models has found meaningful differences in performance across runs and has shown that evaluation results can depend on how outputs are generated.

A single output is an example, not an evaluation.

AI Evaluation Should Resemble a Quantitative UX Study

Imagine that we want to evaluate the usability of an ecommerce checkout flow quantitatively, perhaps for a UX-benchmarking study. We would not ask one participant to complete one checkout task and conclude that the site has perfect usability simply because that participant succeeded.

Instead, we would define a set of representative checkout tasks that involve different types of items to be purchased. We would then observe many participants as they attempted them and would report averages for metrics such as task success and time on task. We would also report confidence intervals to indicate the uncertainty around those estimates ––– how much those estimates are likely to vary across the broader population of users.

An AI evaluation follows similar logic. Instead of asking people to complete tasks with an interface, we ask an AI system to produce outputs for a set of test inputs.

The analogy is not exact. An AI run is not a human participant. However, both types of evaluation rely on multiple observations to estimate how the system performs.

Quantitative-UX researchers summarize results across tasks and participants, rather than relying on one successful attempt. AI evaluations should do the same: use representative inputs, collect enough observations, and report average performance, together with variability and uncertainty.

We do not need to invent a new science of AI measurement; we need to remember familiar principles of experimental design and statistics.

What Would an AI Evaluation Look Like?

Suppose that we want to evaluate an AI system that answers customer-service questions using a company’s policies.

For illustration, we could select 10 representative questions covering different topics, levels of complexity, and types of customer situations. For example, the questions might address returns, cancellations, damaged orders, subscriptions, and warranties. We could then submit each question to the system 5 times, producing 50 answers. (These numbers are not universal recommendations. A real evaluation may require more questions or more runs, depending on the variability of the system and the importance of the decision.)

Before conducting the evaluation, we would define what constitutes an acceptable answer. For example, an answer may count as acceptable only if it:

• Answers the customer’s question correctly
• Includes all important conditions and exceptions
• Is consistent with the company’s policies
• Tells the customer what to do next when appropriate

An answer that fails any of these would be considered unacceptable. We could then calculate each question’s percentage of acceptable answers across its repeated runs and then summarize performance across all 10 questions.

At minimum, we would report:

• The percentage of outputs that were acceptable
• A confidence interval around that percentage
• How much performance varies across different questions
• How consistently the system answers the same question

A confidence interval represents a range of plausible values for the system’s average performance. It reminds us that the score obtained in one evaluation is an estimate, not an exact and permanent property of the system.

Again, the purpose of the example is not to prescribe 10 questions and 5 runs as universal numbers. It is to illustrate the basic design: test multiple representative inputs, run each input repeatedly, and summarize the resulting distribution of scores.

Note that any evaluation is a snapshot of a particular system at a particular time. You need to carefully document all the details: record the model and version, prompts or instructions, settings, tools, context, and evaluation date. If any of these components change, the evaluation may need to be repeated.

Two Questions an Evaluation Must Answer

The example above helps us answer two separate questions:

1. How well does the system perform across the range of questions that users may ask?
2. How consistently does it answer the same question?

These questions capture two forms of data variability: test-input variability and run-to-run variability.

Test-Input Variability

A customer-service system may perform well on short, straightforward questions but poorly on questions involving ambiguous policies or multiple conditions. It may explain the standard return policy correctly but fail when a customer’s situation falls under an exception.

An evaluation containing mostly easy questions will therefore overestimate real-world performance.

Test-input variability represents variation in the outputs that results from the particular inputs included in the evaluation.

Including more questions reduces our dependence on the particular examples selected. However, quantity alone is not enough. A large test set containing only simple questions will produce a precise answer to the wrong questions. The test inputs need to be representative for the inputs that will be used in real life, by actual users.

Run-to-Run Variability

Because it’s nondeterministic, the system may also give different answers when the exact same question is submitted repeatedly. This is run-to-run variability.

Run-to-run variability occurs when the system produces different-quality answers for the same test input.

Repeating an input allows us to determine whether the system handles it consistently. Without repeated runs, we cannot distinguish a task that the system performs reliably from one that it completes successfully only occasionally.

The Same Overall Score Can Conceal Different Problems

Suppose that two customer-service systems are each tested on 10 questions, with 5 runs per question. They both produce acceptable answers on 80% of the 50 runs. The first system is predictable: it consistently handles some types of questions and consistently fails on others. The second system is unpredictable: any customer question may receive an incorrect answer.

But the overall score alone does not reveal this distinction. To understand an AI system, we need to know both how well it performs and how its failures are distributed.

Rigorous AI Evaluation Is Not New

Some established AI evaluations already include repeated attempts and account for variability and uncertainty. Coding benchmarks, for example, use pass@k –– a metric that estimates the probability that at least one of k generated attempts succeeds and requires repeated runs.

Rigorous methods do exist. In practice, however, they are concentrated in academic-research papers. Product teams do not necessarily need to adopt the exact metrics used by academic benchmarks. However, they should apply the same underlying principles: collect multiple observations, account for important sources of variation, and communicate the uncertainty around the results. In other words, they should not treat one output as sufficient evidence.

The Purpose of the Evaluation Does Not Change the Method

Quality assurance deserves special mention because traditional software testing assumes determinism: a test that passes once is expected to pass every time. For a nondeterministic system, a single passing test is a sample, not a proof. A new feature that succeeds in a demo may still fail for 1 customer in 5. QA for AI systems should therefore track pass rates across repeated runs, not the outcome of one test.

When an evaluation will drive a decision — launching a new product feature, choosing a vendor, recommending a new process, or claiming improvement — it should be based on repeated runs with multiple representative inputs. It should report average performance, a confidence interval, and information about the consistency of the results.

Conclusion

When evaluating an AI system, do not ask only whether it produced a good output. Ask how often it produces good outputs across the range of inputs that users will submit and across repeated runs of the same input.

A sound AI evaluation should include: (1) multiple, representative inputs; (2) repeated runs on each input; and (3) averages and confidence intervals to communicate performance and uncertainty. What we do need is to stop treating one impressive output as evidence. A single good output is like a participant who completes a task: encouraging, but not an evaluation.

Anthropic has made Claude Mythos 5 available for code scanning in Claude Security and is integrating it into partners’ defensive products, with users receiving outputs rather than direct access to the model. It is also committing $35mn in credits to open-source security work.

Anthropic is widening access to its most capable cybersecurity model, without letting most people near the model. Claude Mythos 5 now runs code scans inside Claude Security and is being built into the products defenders already use.

The distinction is the whole design. A user of a partner tool receives a specific artifact, a suggested patch or an alert, and has no way to prompt the model to write an exploit instead.

What is live today is the scanning. Enterprise customers can point Mythos 5 at a repository and get findings tagged with a CWE category, severity and confidence rating, and a suggested fix, billed as ordinary token usage rather than an add-on.

Humans stay in the loop by design. Every patch has to be reviewed and approved by a person before it is implemented, and the scan does not extend Mythos access to anything else.

The second announcement is money, and it points at the real bottleneck. Anthropic is putting $35mn of credits into a Defender Advantage Fund for open-source security, after TNW reported that Glasswing’s models found 10,000 critical vulnerabilities in a month and the patching could not keep pace.

Grants will go to three things. Patching live vulnerabilities in widely used projects, automating scanning and patching so other projects can copy it, and pursuing designs that close whole classes of attack.

For European maintainers the timing is not incidental. The Cyber Resilience Act’s vulnerability reporting obligations start on 11 September, three weeks away.

Those rules land on open-source stewards specifically. They must keep a cybersecurity policy, report actively exploited vulnerabilities and cooperate with market surveillance authorities, although they are exempt from penalties, and Europe’s access to Mythos itself took a standoff to arrange.

Credits are not maintainers, which is the limit of this. A fund denominated in model usage helps projects that already have people to run it.

The competitive picture is converging on the same shape. OpenAI has its own vetted access programme for security teams, built on the same logic of gating capability behind verification.

The caution behind all of it is recent. Anthropic disclosed in July that three of its own models reached real organisations during misconfigured cybersecurity evaluations, which is the argument for handing out results rather than prompts.

Grok Bot can operate as more than a set of independent assistants. Bots become a persistent multi-agent system when they are assigned explicit ownership, reusable Skills, event-driven Routines, typed handoffs, verification rules, and approval boundaries. This document presents a practical architecture for building this system from one repeatable workflow. It maps the complete path from a single Bot to an always-on team that can execute, verify, and deliver recurring work with minimal human routing.

The Evolution of the Agent Harness

Models keep absorbing the harness into their weights — soon, it will be a harness for human attention rather than for the model.

Sometime around Christmas 2025, AI engineers noticed a change in agents. They started to work! It’s hard to pin down exactly why. Maybe we finally had holiday downtime to try the newest agents with the newest models. Maybe the models had crossed some capability threshold. Maybe the wrappers around the models had matured.

What I’ll argue in this post is that it was the confluence of the last two. The model and the harness improving together and then their curves of improvement crossing at the right moment. And that dynamic helps to explain what comes next: models keep absorbing the harness into their weights, engineers keep deleting what got absorbed, and what remains is a harness for human attention rather than for the model.

Lukasz Kaiser, one of the people who invented the Transformer, said on “Unsupervised Learning” in June:

“The change last winter, last Christmas — it’s a little hard to pin down. I mean, the harness changed and a little post-training changed and then new pre-trained models came… but it felt like a big jump which is not that easy to pin down what did it.”

The answer to “What happened?” isn’t solely in the model weights. It’s in the system that grew up around the weights.

The answer is in the agent harness.

Think back to November 2022, when ChatGPT was the most advanced AI tool. The only capability at its disposal was next-token prediction and some Reinforcement Learning from Human Feedback (RLHF) that allowed it to act like a helpful assistant. No tools, no search, and no reasoning.

The original ChatGPT was confined to its training data and the prompt you sent it. No more, no less. It was a brain in a vat.

The agent harness is a way for the LLM to break free from that confinement and interact with real digital information space.

What a Harness Actually Is

An agent harness is everything besides the model weights that makes the agent work. The environment, tools, context and guardrails that surround the model. Without the harness the model is a brain in a vat. It can take an epistemic action, but needs the harness to actuate that decision in real digital space.

The harness is like giving the mind of the model a body. With the harness, the model can perceive (context), act (tools), persist information (memory and compaction), and enforce its boundaries (permissions and guardrails).

Harness 1.0: The Past, “The Bolt-On Era”

Two curves run through the path of model / harness evolution. What the harness asks of the model, and what the model can deliver in practice.

The gap between these two curves is equal to the effectiveness of an agent, and the closing of that gap is what I’ll argue led to the tangible improvement in agents that Lukasz Kaiser referenced.

Here’s how the gap closes, in stages:

1. ReAct, “The Harness on Paper” (October 2022): ReAct is a prompting technique to get models to reason through prompting. It’s the agentic loop on paper, external to the model weights. It defines the idea of an “agent loop” where a model reasons -> acts -> observes -> repeats. Again, the ReAct loop exists only as a prompting method. Prompting is the only reasoning method that exists at this time and no one calls it a “harness.” Toolformer (Meta, Feb. 2023), that same winter, hints that tool use could be trained in rather than prompted. It’s a bit like Alan Turing’s idea of the computer before it was instantiated in a physical substrate. A powerful idea that is only later made manifest. (ReAct predates ChatGPT by a month — October 2022 vs. November 2022). Both the curves are near zero at this point. The gap is small because we are just getting started.

2. AutoGPT/BabyAGI, “Premature Autonomy” (Spring 2023): With AutoGPT/BabyAGI, the harness curve sprints ahead of the model capability curve. Both hand the model full autonomy, asking the model to act as an “autonomous employee,” but the models at this point are still little more than brittle next-token predictors. A loop doesn’t add capability to a model. A loop amplifies the capability a model has, and below some threshold the loop amplifies errors rather than reliability. Consider the power of compounding in the negative: 95% reliability per-step over a 20-step task results in a ~36% average success rate. The harness hands the model an assignment it has no realistic chance of completing. This is where the gap is at its widest and the next 18 months are a reaction and attempt to close that gap.

3. Cursor/Copilot, “Retreat to Human in the Loop” (2023 - 2024): The first AI-powered IDEs recognize the failure-mode of giving the model too much autonomy. They close the gap by pulling the harness curve down below the model curve. Don’t give the model the loop directly; give the human the loop and empower the human to orchestrate the loop while the model speeds the human up. The first version of Devin tries to hand the autonomy back to the model. A test from the team at Answer.AI shows that is still premature, with a ~15% success rate. It’s evidence that the move from the IDEs to retreat from full autonomy is not cowardly, but the correct move. However, while the prevailing tactic is to pull the harness down below the model, models continue to improve. Near the end of 2024, with the introduction of o1 — the first reasoning model — for the first time the gap inverts and we begin to see the first signs of a model capability overhang.

4. Claude Code, “The Curves Cross” (February 2025): The inversion at the end of 2024 sets up an opportunity that someone has to seize: if the model is now ahead of the harness, then a harness intentionally riding the brakes of the model is leaving capability on the table. Claude Code is the first coding agent built to seize that opportunity. It abandons the IDE for the terminal, gives the model bash and file read/write access, and replaces the need for human approval on every change with permission rules. The model is handed the loop again, and this time it understands the assignment. Boris Cherny and team build Claude Code with the next model’s capabilities in mind, not the current one. It is such a hit not because it’s the first product to give the model autonomy, but because it’s the first product to do so at the right time. That time is the crossover point where the model has gotten reliable enough to succeed with autonomy. Claude Code grows to roughly $1B ARR within six months, all because Anthropic seized the opportunity available when the curves begin to meet.

What happens next is that the curves don’t just meet, they begin to braid together.

Harness 2.0: The Present, “The Co-Training Era”

Today the harness matters, and in a way we can measure. Harness-Bench ran the same model over the same 106 tasks in different harnesses, and scores ranged from 52.4 to 76.2: a 23.8-point spread with zero change to the model. Half the agent is the harness.

OpenAI achieved a similar result on ARC-AGI-3 with harness changes. Adding only retained reasoning and compaction, GPT-5.6 Sol’s ARC-AGI-3 score tripled from 13.3% to 38.3%.

What’s happening under the hood is that Reinforcement Learning (RL) has moved inside the harness. From OpenAI’s codex-1 release announcement in May 2025: “codex-1 was trained using reinforcement learning on real-world coding tasks in a variety of environments.”

The two curves join and start to braid as one unified system.

This is the dream of Toolformer manifesting in reality. Rather than a tool prompted from the outside, now tool calling is trained from within the environment of the model.

Then, as the models are trained in the environment of the harness, they start to absorb the harness capabilities into the model weights, learning how to auto-compact with knowledge of their own context window, for example.

GPT-5.1-Codex-Max launch:

“The first model natively trained to operate across multiple context windows through compaction.”

Once the models absorb the harness capabilities, the harness can shed the scaffold. It’s production by reduction. Thariq Shihipar from Anthropic said that the team recently deleted 80% of Claude Code’s system prompt.

The measure of the pace of agent harness evolution is how much of the harness you get to delete, while retaining the same capability level. This is the future we need to build towards as AI engineers.

This, then, is the loop of model / harness evolution: train -> absorb -> shed -> repeat. The model climbs to the next thing it can’t do yet.

The jump that Kaiser pointed out is hard to pin down because it’s not a discrete event. A pre-training leap is noticeable because you can articulate it in a model card. A model / harness co-evolution jump is less so, because there’s no documentation of the evolution process. That’s the answer to the jump last Winter: it happened in the space between the model and harness working together.

We need to ask: if every harness capability will eventually get absorbed into the model, what does that leave us with?

Harness 3.0: The Future, “The Attention Era”

Keep deleting everything that the model can absorb. Imagine what your agent looks like at the conclusion of that process. What are you left with in your hand when you’ve deleted everything?

What do the model weights absorb next? Multi-agent orchestration, tool selection, memory...to name a few possibilities. Researchers are building self-improving harnesses that can themselves be trained in a similar way to models.

What’s left at the end of this deletion and absorption process are the human-centric agent capabilities. Things like permissions, identity, trust and legibility. A model that absorbs permissions into itself has dissolved permissions. Absorption doesn’t end the harness. Absorption inverts the harness.

The harness becomes the agent’s interface to the human that operates it.

The harness was born as the human interface to the model. We grew from chatbox to IDE to the terminal. If the model absorbs the computer-facing capabilities, the next stage of evolution becomes one layer of abstraction up. The harness becomes the model’s interface to our human attention.

It becomes the attention-interface.

Ryan Lopopolo said on the “Extreme Harness Engineering for Token Billionaires” episode of Latent Space:

“The only fundamentally scarce thing is the synchronous human attention of my team.”

Tokens became abundant and reliable, yet we remain bottlenecked on scarce human attention.

We see sparks of this already, with Anthropic’s long-running agent progress files and agentic approval queues.

The gap between the model and harness curve doesn’t disappear when the model absorbs the harness. It migrates across the human boundary and creates a new pair of curves with a new gap. The new gap is the space between what the agent asks of the human, and what the human is able to answer.

The Attention-Interface

I predict that within a year, every company building agentic AI will ship a human attention policy surface in the way that every agentic AI company shipped AGENTS.md.

AGENTS.md tells the agent how to work with your codebase. The attention-interface will tell the agent how to work with you. It will govern when it’s allowed to interrupt you, when it should keep working, which decisions it can make alone and which decisions need your approval. And like everything else in the agentic system, it will become a learnable component of the system that can learn with more data. Every correction becomes useful data.

The model began as a brain in a vat. The harness gave the brain a body, then the body started to dissolve into the brain. What’s left for us to build is the thing no future model will ever absorb. The interface to the one true scarce resource: human attention.

Inherent, a London AI lab founded by Google DeepMind alumni, says its AI agent just outperformed much larger models from Anthropic and OpenAI using a fraction of the size.

Of all the startups launched by Google DeepMind alumni, Inherent has gotten relatively little attention. But while better-funded rivals have yet to show the world anything concrete, the London-based team is starting to share what it’s been building.

Just weeks after emerging from stealth with a $50 million seed round, the British startup says its newly released AI agent, Faraday, has outperformed larger, better-known models at a specific task: independently reproducing the findings of published scientific papers without being told the answer in advance.

That may sound like a mere party trick given Inherent’s much loftier goal — building AI that can discover new scientific knowledge and not just verify old results. But paper replication is a standard training exercise for human scientists, too, co-founder and chief scientist Edward Hughes said. “Many PhD students actually start by doing this.”

Beating other AI systems at the task wasn’t the point, Hughes told TechCrunch; how they got there was. “What was most interesting to us about this was not so much the result of beating those frontier agents — which of course we liked — but was actually the way we went about building this.”

Here’s the part that should catch an investor’s eye: Measured against Anthropic’s Claude Opus 4.8 and OpenAI’s GPT-5.5 — both much larger, frontier-scale systems — Faraday runs on a comparatively tiny model called Qwen 3.6 that has just 27 billion parameters. (Roughly speaking, “parameters” is a proxy for a model’s size and, typically, its training costs, as well.) Inherent’s bar for success was also higher than simply accuracy. Beyond replicating results, it wanted Faraday to demonstrate “research taste” — an instinct for what experiments are worth running and how to design them well.

Teaching something as intangible as taste is hard, which is where reinforcement learning comes in. It’s a training method that rewards an AI system for good outcomes rather than spelling out rules for it to follow. Rather than training its agents primarily on the study of how science itself is conducted, Inherent leans on this reward-based approach, betting it will generalize better to its longer-term goal of agents capable of contributing across many scientific fields.

“We’re always guided by that north star of building an AI scientist agent and imbuing our agents with taste,” Hughes said. That focus has also shaped what Inherent chooses not to build. Rather than developing its own coding tool, it had Faraday use OpenAI’s GPT-5.5 Codex instead, much the way human scientists lean on existing software rather than building everything themselves, according to the company.

Inherent is also trying to avoid building agents that simply tell users what they want to hear. Instead, Hughes said, the goal is modeled on his favorite kind of teammate — the kind who comes back and says: “I got curious about this, and I went off and I did these experiments. What do you think of these results?”

That collaborative instinct extends to how Inherent operates as a company. Its dozen employees all work in person out of an office in King’s Cross — the once-rundown London neighborhood that Google DeepMind’s presence helped turn into one of the world’s top AI hubs. “We believe that London is the place to be,” Hughes said.

Hughes is bullish on London’s density of AI talent, but he has also added his voice to calls to end “garden leave” — the practice, common in the U.K., of barring departing employees from joining or starting a rival company for months after they resign. It’s a restriction American researchers generally don’t face, giving U.S. startups a head start on hiring talent who’ve left a prior role. “This is a personal view rather than a company view, but I was affected by the garden leave problem,” he told TechCrunch.

Hughes eventually got around that constraint and started Inherent alongside two other DeepMind alumni and a fourth co-founder. The startup isn’t slowing down either. It plans to grow its headcount to “about 20 to 25” by the end of the year. Given its ambitions in world models as well, and with Demis Hassabis’ new role leaving some DeepMind staff unsettled, Inherent’s hiring push could make it an appealing landing spot for DeepMind employees weighing a move.

Michael Polansky is remarkably unassuming for someone operating in a corner of the world known for outsize egos.

Seated at a leafy patio outside a popular bakery in Mill Valley, an affluent town about 15 miles north of San Francisco, Polansky — bespectacled, his fresh face framed by dark hair shot through with gray — has the look and the friendly demeanor of a young professor.

He is, in fact, both the co-founder of a buzzy AI and biology startup called Outer Biosciences and the creative, business, and romantic partner to Stefani Germanotta — better known as Lady Gaga. It’s an unlikely double life. On the one hand, the couple invariably moves in the world that comes with celebrity; on the other, he’s running an outfit that has spent years figuring out how to keep living human tissue alive outside the body — for over a month, so far — without anyone outside the company knowing.

He didn’t see any of it coming. Polansky grew up in Minnesota and went on to Harvard, where he studied applied mathematics and computer science, graduating in 2006. Afterward, he logged three years at the hedge fund Bridgewater Associates — “a very unique place,” Polansky says over coffee, and one where he had “a really good experience,” even if it wasn’t a place he was going to “wake up and be excited about… every day.”

His pilgrimage from Bridgewater to Silicon Valley ran through Minnesota. By coincidence, Sean Parker’s assistant at the time had been Polansky’s neighbor growing up. At a 2009 wedding in their home state, she mentioned that Parker was looking for someone to work with him. Polansky already knew who Parker was and was looking to move west. The two had dinner in New York and “hit it off immediately.” Polansky quit Bridgewater the next day and moved to San Francisco.

He first landed a role as a principal at Founders Fund when the firm was run by its four original partners — Peter Thiel, Sean Parker, Luke Nosek, and Ken Howery — with Polansky and another young principal at the time, Brian Singerman, sharing an office. “It was a really, really great experience,” Polansky says.

When Parker left Founders Fund after becoming liquid in his Facebook stock and wanted to build out his own family office, he brought Polansky with him. Polansky ran that office — handling Parker’s business, investment, and philanthropic interests, including helping stand up the Parker Institute for Cancer Immunotherapy (where Polansky remains executive director) until COVID, when his life “moved in a different direction.”

That change had a lot to do with Germanotta. Polansky met her in late 2019 at one of Parker’s birthday parties. Charmingly, the meeting came at the urging of Germanotta’s mother, Cynthia Germanotta, president of the Born This Way Foundation, whom Polansky had come to know through his philanthropic work.

“She had said, you know, for months and months, ‘I want to set you up with my daughter,’” Polansky recalls. “I was like, I think you’re making fun of me.” She wasn’t. When their own mothers later met, he says, laughing, “it all made perfect sense.” (His mother and Germanotta’s mother are now close friends.)

Their relationship is a full partnership, professionally and personally. Notably, for Lady Gaga’s most recent world tour, which began in July of last year and ended in April, the couple managed a massive operation across three 747s, something Polansky compares to running “a 200-person startup that travels around the world every day.”

It also includes Haus Labs, the cosmetics brand Germanotta initially built “on her kitchen floor,” Polansky says, instead of simply licensing her name to an existing company. That business, based in El Segundo, California, an operation with roughly 70 employees, is reportedly thriving.

Germanotta also sits on the board of Outer Biosciences, and the two companies collaborate at the margins. For example, Outer Biosciences’ chief scientist, Kyung-Jin Jang, sits on Haus Labs’ scientific advisory board, and the companies have run some joint projects.

Says Polansky, beaming as he talks about Germanotta from our sun-dappled table: “People really haven’t gotten to know a certain side of her publicly… She’s such a brilliant businessperson.”

From cancer to tissue in a dish

While Polansky is a businessperson, he’s not a scientist. He got into life sciences “accidentally,” he says, through more than a decade spent alongside Parker in cancer immunotherapy, a field that was “very fringe” when they entered it.

Outer Biosciences, co-founded in 2022 by Polansky (CEO), Jang, Chris Hinojosa (CTO), and Stanley King (chief business officer), grew out of frustration that the pace of innovation in biology and chemistry has never matched software, in large part because there’s no ethical way to run experiments directly on people. Meanwhile, the proxies scientists rely on instead — animal models, simplified cell cultures, lab-grown organoids — are poor stand-ins for how a real human organ behaves.

So Outer Biosciences took a different approach. Instead of engineering a synthetic organ, the company sources human skin that would otherwise be discarded after surgery — mostly plastic surgery — through what it describes as vetted nonprofit and commercial biobanks and brokers operating under “institutional review board oversight and documented donor consent,” principally the National Disease Research Interchange and the Cooperative Human Tissue Network. (Both outfits receive federal funding from the NIH and the National Cancer Institute without being federally operated.)

Polansky is careful to note that there is no single government tissue network that qualifies buyers. He says Outer Biosciences pays fees to these suppliers on a cost-recovery basis rather than purchasing tissue outright. He also says the company spent roughly two years on building that pipeline and handling the protocols required to receive that tissue “within hours” of surgery, while it’s still living.

Asked about the donors’ privacy, he says that every sample arrives already de-identified — stripped upstream by the supplying organizations of names, contact information, and other direct identifiers. A proprietary support system developed by his team then feeds the tissue nutrients and removes metabolic waste, extending its viable life well past the industry norm.

That norm, by the way, is a matter of days. That’s enough time to test for acute toxicity, but not for slower biological processes like collagen remodeling, pigmentation change, or barrier repair that take weeks to unfold. (Dermatologists routinely tell patients to expect changes over a matter of weeks for this same reason.)

Outer Biosciences’ system meanwhile keeps tissue alive for up to a month, says Polansky, who says it retains its “day-zero architecture and preserves its day-zero epidermal, stromal, and immune-associated molecular programs.” In plainer English, that means that 30-day-old tissue cared for by the company looks a lot like day-one tissue, but it isn’t fully indistinguishable from it.

Sunburn is one of the clearer examples of what that extra time buys Outer Biosciences. Polansky says that its researchers can induce UVB damage in living tissue, then track the stress, inflammatory, and recovery-related responses that follow over the following weeks as an information-gathering exercise. He says the team isn’t “healing” the skin but rather watching an injury happen and then watching the biology that follows it over time.

Perhaps anticipating pushback from the scientific community, Polansky is careful about how he frames the company’s achievements when this reporter asks questions about rival technologies. The startup’s value, he says, isn’t any one piece of what it’s doing but how the pieces fit together: human tissue that can be kept alive for weeks, a diverse donor pool that Outer Biosciences’ team can subject to controlled experimental conditions, and repeated molecular measurements taken along the way.

And it’s all fed into one, closed self-enforcing system. An AI model predicts which untested chemicals are likely to have a beneficial effect on a specific skin function. Those chemicals get run through the living-tissue system. Then the results, whether the prediction was right or wrong, get fed back into the model, improving the next round of guesses. That loop is the actual product Outer Biosciences is selling right now: not a single skincare ingredient, but a faster way of finding one.

That speed is a more recent development and a giant improvement from where things started, says Polansky. Early on, the company relied on a “brute force” approach, mining scientific literature and partnering with the National Cancer Institute on natural compounds from extreme environments. That phase produced a couple of leads over about 18 months, but with AI layered in, the company is now generating a new candidate roughly every six weeks, with six leads currently active in its pipeline and several dozen additional “hits” logged.

What makes that pace truly astonishing, Polansky says as the lunch crowd around us thins out, is the size of the current universe of skin-active ingredients.

It’s almost impossible to know the exact number, but it’s small. “Active ingredient” means something different informally than it does formally. While the FDA maintains rules covering 13 categories of over-the-counter skin drugs (think sunscreen, antifungals), across all of them, only about 120 to 130 active ingredients are approved. Add cosmetic ingredients backed by actual research, says Polansky, and that number is closer to 200.

This, of course, presents opportunities.

Outer Biosciences is discovering cosmetic ingredients, not drugs, so there’s no FDA approval to seek out. Instead, the route runs through two steps: first, getting the ingredient a standardized industry name; then safety testing under guidelines set by the OECD, a Paris-based international body whose member countries agree to accept each other’s properly run studies.

A partner outfit then commercializes the whole thing. Indeed, rather than build its own consumer brand, Outer Biosciences right now plans to license or sell its finished ingredients to beauty or pharma outfits that will then formulate these into actual products (a serum, a cream) and bring them to market under their own brands. Already, four of Outer Biosciences’ six current leads look likely to reach commercialization, Polansky says.

In the meantime, the company is generating money from collaborative research partnerships, including a pharmaceutical partner that’s studying why certain cancer drugs cause severe skin rashes, and consumer beauty brands that are testing whether Outer Biosciences’ data holds up against their own product-development and marketing needs.

If Polansky is raising more money for the company currently, he won’t say. To date, the company has raised roughly $23 million, with early backers including Wing Ventures, Initialized, and Polansky’s own investing firm, Hawktail, among others. The company employs 19 people, with all but Polansky based just outside Cambridge, Massachusetts.

Asked why he’s chosen now to talk about the company after years of near-total silence — he says he barely discussed it even with close friends — Polansky points to the data the team is beginning to amass and the confidence that has given them. “Trying to do this in private is hard,” he says, as the waitstaff start flipping chairs onto tabletops, signaling it’s closing time at the bakery. “We kind of want to start working in public now,” he adds, shrugging.

Outer Biosciences is not the only company chasing this idea, even if its use of real tissue, rather than synthetic, is distinctive.

Vivodyne — a Philadelphia-based competitor building lab-grown human organ tissue to generate causal biological data — just this month announced it has raised close to $80 million to date, including a $38 million seed round and a $40 million Series A, both led by Khosla Ventures.

Other rivals are pursuing various flavors of organ-on-a-chip and microphysiological systems for preclinical testing.

Polansky doesn’t seem especially preoccupied with any of them — less, it seems, out of arrogance than because he seems to have his hands full. Besides, there’s plenty of room for everyone in this moment. Unlike AI companies training on scraping the internet, there is no “biology internet” to scrape.

And Outer Biosciences has two other reasons to focus on its own knitting. First, the data it generates doesn’t exist anywhere else, which, conceivably, makes the company’s position more defensible, if a lot slower, to build than “traditional” software-based AI startups. Second, it’s cheaper to run, with modest compute demands compared with training a large language model. In fact, all of the company’s AI work currently runs on-premise, not in the cloud, because “we don’t want the data in the cloud,” Polansky says.

Whether over time Outer Biosciences becomes a stand-alone commercial-ingredients business, licenses its discoveries, or eventually reorganizes around a single breakout compound, Polansky says he hasn’t settled on — and the team doesn’t need to. The more important goal, he says, is a predictive model that’s accurate enough that the company can spot promising directions in skin biology without having to run every experiment physically first, opening up a rate of discovery in dermatology that doesn’t currently exist.

For now, instead, the work of turning a promising compound into a real product — the formulation, the manufacturing scale-up, the supply chain, the safety testing — is still done manually by the same scientists who discover the compounds in the first place. Building out a product-development team, with people who’ve done this kind of work before, is next on the roadmap.

“I think it’s going to be fun,” he says, “to have people know that this is what we’ve been doing.”

Some of Nvidia's biggest customers have been told that the price of the AI servers containing its flagship Vera Rubin and Grace Blackwell chips is going up by more than 15% due to the soaring costs of memory chips. The price hikes will go into effect early next year. The increases will depend on the generation of Nvidia chips and the memory configurations. Nvidia has also raised prices for its gaming-oriented PC graphics cards.

Today we’re excited to publish an updated roadmap for the Model Context Protocol (MCP), covering the next specification release and beyond.

It sets the direction for protocol work over the coming months and was developed by the Core Maintainers together with our community of maintainers and Working Groups.

Looking back

The previously published roadmap came out in March with four priority areas: transport evolution and scalability, agent communication, governance maturation, and enterprise readiness. We’ve made significant progress in all of these over the past five months.

The bulk of the changes landed in the 2026-07-28 specification release - you might’ve already seen them in our SDKs and documentation. The improvements ranged from minor modifications to major protocol overhauls.

One of the biggest changes we shipped is that protocol-level sessions and the initialization handshake are gone, so a server can scale horizontally without holding state (SEP-2575, SEP-2567). Additionally, clients can now call server/discover to learn a server’s supported versions and capabilities before doing anything else. List results are also cacheable (SEP-2549).

On the agent communication side, Tasks were reworked based on early adopter feedback - we moved them into an official extension (SEP-2663). The brand-new Multi Round-Trip Requests pattern (SEP-2322) replaced server-initiated requests so that elicitation and similar flows work on stateless servers.

The Server Card Working Group continues to work through the .well-known metadata conventions for MCP servers, so a server can be discovered and reasoned over without connecting to it.

Governance has evolved as well. We formally adopted a Contributor Ladder, Working Groups now triage SEPs in their own area, and the specification has a proper feature lifecycle and deprecation policy that the 2026-07-28 deprecations were the first to follow.

Enterprise readiness was heavily focused on security in the past release cycle, and as expected most of this work arrived as authorization improvements: issuer validation, issuer-bound client credentials, and Client ID Metadata Documents (CIMD) as the preferred registration path for clients, with Enterprise-Managed Authorization available as an extension (which is also now stable).

This is significant progress in a very short span. The updated roadmap picks up from here.

Priority areas

The new roadmap is organized into five priority areas. Several of them pick up work that the previous version of the roadmap listed as being on the horizon, including server-initiated events, result type improvements, and agent identity, which have since matured enough to become priorities in their own right. Each area has a set of Core Maintainers responsible for it and one or more Working Groups.

Agentic messaging primitives

Modern agentic workloads no longer fit the standard request-and-response pattern. Loops can run for longer, servers can push streamed results, and there is a clear need to steer work mid-flight. MCP has been growing to meet these requirements, introducing Tasks, subscriptions/listen, and progress notifications. We want to make sure that we not only offer the right primitives for the job, but also that they work well together. The work here spans server-initiated events (webhooks and channels, so clients aren’t left polling for results), a composition review across the Agents, Transports, and Triggers & Events Working Groups, and maturing the Tasks extension (SEP-2663) so it can move into the specification.

HTTP-native transport unification and hardening

With the 2026-07-28 release, a remote MCP server is now no different from any other HTTP workload, making it easy to host and operate one on any infrastructure that developers and organizations already use for their APIs and services. This approach has proven to scale, and we want to stretch it to cover other deployment modes as well, including local servers speaking Streamable HTTP over stdio. Unifying on one transport lets us simplify MCP server and client development even further.

Agent identity and enterprise-ready security

MCP authorization today is built around a person approving access in a browser. That works well for interactive clients, but more and more of the callers are agents running as cloud workloads with their own identity, acting on behalf of a user who isn’t present, or delegating narrower authority to sub-agents. We want MCP servers to have a standardized way to recognize and trust those agent identities, built on existing standards rather than pasted API keys and long-lived tokens.

The work here covers finalizing Demonstrating Proof of Possession (DPoP) and driving its adoption, and defining an opinionated path for agent identity and delegation through Workload Identity Federation, the ID-JAG grant behind Enterprise-Managed Authorization, and standard token exchange. We will also continue to grow our engagement with the OAuth standards bodies, including the IETF OAuth and WIMSE working groups, to help the underlying standards evolve with the building blocks that agent identity needs.

Improved primitives

Tool calling is the part of MCP most developers touch first, and it has held up well over the lifetime of the protocol. Where it falls a bit short, however, is in the result handling. A tools/call response can carry the same output in more than one form, and a server developer today has no way to know which form a given client will put in front of the model. We aim to make this easier by standardizing on one clear contract.

The other challenge we need to address for primitives is their ever-growing scale. Connecting to a server with a hundred tools means the model pays for that entire surface before the user has asked a single question, and tool selection tends to get worse as the list grows. We’re starting a progressive discovery effort so a server can offer a small entry point and reveal more of its catalog as the conversation narrows.

Improved SDK developer experience

Our SDKs are how developers experience MCP. We are investing in their ergonomics and their conformance with the specification, and in making them intuitive and well-documented across every platform and language we support. This is even more important now that many developers build MCP clients and servers by pointing an agent at our libraries, where clear APIs and accurate docs decide whether the code will work with minimal friction.

Proposal prioritization

Specification Enhancement Proposals (SEPs) that fall within these priority areas get expedited review and have the best chance of acceptance. Proposals outside them aren’t rejected automatically, but maintainer review time is scarce and goes to these areas first.

If you’re considering a SEP, identify the priority area it belongs to, raise it with the relevant Working Group, and work with its members to shape your proposal. Each area on the roadmap names the Core Maintainers responsible for it, and anyone interested in contributing can reach them on Discord. We’re excited to work with the community to review and build on the proposals that support this roadmap.

Get involved

Every priority area above has a Working Group behind it or forming around it, and all of them have room for more contributors. There are several ways to participate:

• Join a Working Group or Interest Group: see the Working and Interest Groups page and the community channels.
• Propose or comment on a SEP: read the SEP guidelines, then open one or weigh in.
• Start an experimental extension: SEP-2133 lets any WG or IG experiment in an experimental-ext- repository before a formal SEP.
• Contribute directly: the contributing guide covers the specification, SDKs, and tooling.

We look forward to growing and evolving MCP together!

Fable & The End of the Free Lunch

There’s some talk today about how agentic coders are balking at Anthropic’s pricing and adopting alternatives. I was reminded of a thought I had in the weeks following Fable’s release: the free lunch was over.

When Moore’s Law was in effect, it didn’t make sense to ruthlessly optimize your code. In 18 months, a CPU would arrive that would double your performance. Herb Sutter famously referred to this as, “the free lunch,” in a seminal essay.

When Moore’s Law slowed in the mid-2000s (specifically, single-threaded performance stagnated), we suddenly had to think about parallelization, architecture, memory locality, etc.

We had to think about what work went where.

Prior to Fable, it felt silly to waste too much time improving your coding harness or context strategies. A new model would arrive at the same price (or cheaper!) and paper over most of your problems.

But then Fable landed. It was (and still is!) incredible. But the cost was so high and Opus was good enough (as was 5.6, K3, and even GLM) for most of the code we needed.

So we started to think about what work went where.

GLM 5.2 is worth focusing on. It came out the same week as Fable and is roughly 1/9th the cost (and ~1/5th the cost of Opus 5). Is GLM 1/9th the quality of Fable? Perhaps, for certain classes of tasks. But for most rote coding it’s more than sufficient. Especially when provided with great context. I frequently chat with Fable to interrogate and shape a design, before handing off a brief to GLM.

I get pushback that falling inference prices will eventually bring us back to sending everything through the largest models. But I’m not so sure: those same gains will benefit the K3s and Qwens, and as we continue to develop better harnesses it will be easier to provide weaker (but still great) models with sufficient context to perform well.

Plus, Fable’s other shock likely locks in this change. Fable’s access controls, dynamic degradation, and required data retention spooked enough companies (and countries!) into thinking about where they send their traces and where they get their tokens.

Full article content is not available for inline reading.

Read the original article →

“How do you find problems worth working on?” a senior engineer I mentor asked me recently. He’s trying to make the jump to staff engineer and realized that the role isn’t just about doing the work he’s assigned. He also needs to get involved in figuring out what his team and org should be building.

Someone else had suggested blocking out time in his calendar to think about the bigger picture. He’d tried that, but hadn’t found it productive, so he asked if I had any alternatives.

I told him I rarely find good problems by staring at a blank page and trying to “think strategically.” Instead, I act like a sponge. I listen to the stream of day-to-day noise, absorb the problems people are having and let them sit in the back of my mind. Over time, some fade away while connections begin to appear between others that initially seemed unrelated. Eventually, I start to see what’s really slowing people down and what my team or I can do about it.

I’ve worked with many engineers who’ve never really tried this. They wait for managers or leads to identify opportunities, then demonstrate their value by solving the hardest assigned problems. That can absolutely lead to promotion. But the projects that have made the biggest impression in my career were the ones where I found and solved an important problem my leaders did not yet realize existed.

One caveat: my experience comes mainly from working on infrastructure and developer tools at large companies, on teams where engineers have a lot of bottom-up autonomy to influence their roadmaps. In a more top-down environment, there may simply be less room to work this way.

Absorb problems, not requests

People love talking about the problems they are facing: in meetings, chat threads, presentations and email. They explain why their work is hard, complain about what slows them down and describe what they wish they could do.

When something overlaps with my area, I start pulling on the thread. I might ask, “If X existed, would it solve your problem?” or point them at an existing feature in a product I own and ask how much of their use case it covers.

Users often ask for a particular solution instead of explaining their root issue. Rather than taking the request at face value, I keep digging until I understand what they are trying to accomplish and why existing products do not work for them.

As a natural introvert, this sort of ambient listening works particularly well for me. I don’t need to fill my calendar with speculative meetings just to find ideas; there is already an enormous amount of useful information flowing around me during a normal week.

When a problem seems worth exploring, though, I become more active; I need to see how it affects the team’s day-to-day work. I’ll sit with them as they walk me through their workflows and the bugs they’re investigating. When I can, I’ll try working through some of those bugs myself. Seeing the problem firsthand makes it easier to separate what the team actually needs from the solution they asked for.

I also seek out people who see more of the organization than I do: those who own critical systems, work across several teams or have particularly deep insight into the work downstream of my team. I’ll arrange a 1:1 or coffee chat and ask about interesting problems they’ve come across. They may have already seen the same issue in several places and started connecting the dots, giving me a head start on patterns I might otherwise have taken much longer to notice.

Let problems accumulate

Several times, I’ve been burned by moving too fast. I became excited by a request from a vocal team, built the feature and watched them barely use it. Their priorities had changed, or the request had come from a one-off investigation that no longer mattered. How eager a team was in that moment wasn’t the same as how important the feature was relative to everything else my product needed to support. By hyperfocusing on their request, I lost sight of the bigger picture.

That taught me to let potential problems pile up. Listening the way I do leaves me with far more of them than I could possibly solve, and not all deserve action. Most don’t need to turn into projects the first time I hear about them; waiting can be a superpower.

Waiting means the same problem might pop up independently in different teams, making it a higher priority to solve. Or problems that look different on the surface might turn out to have the same shape, so I can address several use cases in one shot. Or, as I’ve learned painfully, the requesting team didn’t even care that much in the first place.

Instead, I make a mental note and revisit the problem if it comes up again. Other engineers I know write this sort of thing down more systematically. The mechanism is a personal choice: everyone has to figure out what works for them. What matters is keeping unresolved problems around long enough for more evidence to accumulate.

Find the common shape

Waiting helps me collect evidence, but that alone doesn’t tell me what to build. I still need to work out whether the problems I’ve retained are genuinely related and what, if anything, could address them together.

Perfetto, the performance debugging tool I work on, is a good example. It displays recordings of system activity on a timeline made up of rows called “tracks.” Over a couple of years, teams kept asking for small, specific additions to the UI. One wanted a command to keep their preferred tracks pinned to the top of the screen; the next team wanted the same, but for a completely different set of tracks. Others wanted Perfetto to open already zoomed in on a particular part of a recording, or to show a custom aggregation tuned to what they cared about. A few had stopped waiting for us and built elaborate workarounds with bookmarklets.

By the time enough of these had piled up, my head was the usual tangle: the requests themselves, the constraints on each and a handful of half-formed solutions. I’ve learned not to force a solution by just sitting at a desk and thinking. Instead, my best untangling happens on long, aimless walks around London, where connections come more easily when I’m not trying to force them.

What I eventually realized was that none of these teams really wanted the specific feature they’d asked for. Each wanted to personalize Perfetto for their own workflow without imposing their choices on everyone else. The underlying need wasn’t any one feature but rather the ability to extend the UI. When a connection like that finally clicks, it’s one of the best feelings in the job: several awkward requests collapse into a single idea, and possibilities open up that none of them hinted at on their own.

That feeling, though, is exactly when I have to be careful, because a common shape is only a hypothesis and elegance is not evidence. When it happened with extending the UI it turned out to be real, but I’ve been fooled before.

In another recent case I was convinced that building a transparent caching system for querying Perfetto traces would solve issues with sharing large traces and repeated queries. It was only as I wrote the RFC and built a prototype that I realized the elegance was a lie: the two problems wanted genuinely different solutions. I reluctantly split the design in two, both halves of which have since shipped.

Pressure-test before building

You’d think this would be the moment I start building, but it usually isn’t. How far I go depends on how sure I am that the idea works and that people actually want it.

If something is useful and low-risk enough, I act straight away: I send the change and let my manager know. When I’m unsure whether an idea will work or how much effort it will take, I build a throwaway prototype instead; it exposes the failure points and gives me something concrete for others to react to. And when an idea is big but I’m convinced by it, I commit to the full effort: weeks or months of work and the hard yards of building support across other engineers and teams.

Through all of it, I’m not only trying to convince other people; I’m also trying to convince myself. Sometimes the honest answer is to stop: if people don’t see the value I do, or we hit a major technical wall, I’d rather drop the idea now than build something no one uses or that becomes a maintenance nightmare. And sometimes it holds up but the timing is wrong, so I park it, ready to spring into action the day it becomes an org priority.

When an idea does hold up, I don’t necessarily need to be the person who builds it. I might implement it, someone else on my team might, or it might change what the org focuses on. Finding and shaping the right problem can have an impact even when I don’t own the implementation.

The Perfetto extensions idea was worth that full effort. We were already building plugins to modularize the UI, but they weren’t enough: teams had to open source all their plugin code, which wasn’t an option for many internal use cases. So before building anything new, I took the problem and my proposal to my manager, teammates and the client teams. I ended up writing two RFCs, having several 1:1s and giving a couple of talks, refining it as the feedback came in.

In the end, I designed and implemented macros as “lightweight extensions”: a way to automate actions in the UI without writing a plugin. Extension servers took the idea further by letting teams share their macros.

Instead of implementing every requested feature ourselves, we gave teams ways to adapt Perfetto to their own needs. Dozens of teams inside Google now use macros and extension servers, and several other companies use extension servers internally too.

Solving useful problems helps me find the next one

The more often I go through this process, the easier it becomes. When I show genuine interest in someone’s problem, ask useful questions or help solve it, they remember. They start coming to me earlier and bring me into conversations with other people facing related issues.

That gives me a wider view of what is happening across the organization, making it easier to spot patterns and build things people actually need. Solving one of those problems brings me into more conversations, and the loop continues.

Those successes build the kind of trust that comes from long-term stewardship. Early on, I had to turn many of these ideas into something real myself to prove that my judgment was sound. Over time, my manager and org gave more weight to my assessment of what mattered. That allowed me to influence the roadmap without needing to own every project.

This differs from the idea that becoming a staff engineer means replacing technical work with meetings and coordination. For me, conversations are inputs into what I build, not the end result.

Conclusion

That is what I wanted my mentee to understand: finding problems worth solving isn’t separate from the rest of the job. It comes from staying engaged with people’s work long enough to see what no single request can show you.

Nvidia will invest $1 billion in AI startup Poolside at a pre-money valuation of $12 billion and will pay $6 billion to license Poolside's technology and hire the bulk of its engineers.

Fast and Hard Code

One of the memes on Twitter is that “programming is solved now.” I’m not sure to what degree it is, but one thing is pretty clear: the act of familiarizing yourself with a language no longer matters and some of the friction that mattered for humans does not matter for agents.

As a result, LLMs make language choice much less consequential than it used to be. If you don’t like the choice, you can seemingly rewrite it in another language and you can make it pick a language that you, as a programmer, are entirely unfamiliar with.

Which in turn means that people can, and do, choose based on the marketing of languages much more. As a long-term Rust programmer I found it quite fascinating to see people now ship Rust code who previously might not have chosen it. I attribute at least one part of this to two recent vibe shifts: there is a lot more talk about wanting fast software, and about LLMs being exceptional at optimizing code without regressing behavior.

Folks like Mitchell Hashimoto, Charlie Marsh, Jarred Sumner, Daniel Lemire and quite a few others always carried a certain level of obsession with fast and performant software and they also all happen to be receptive to agents writing code. Maybe as a result, or unrelated others are now joining in. That’s because with things like autoresearch you don’t even necessarily need to know all the tricks: you just need to put an agent on it — though knowledge greatly helps!

If you look around, there are plenty of projects that want to be fast and small, and they increasingly pick “hard languages”. And it’s not just Rust that is benefiting. Even Zig — despite the fact that the creators and parts of the core community are pretty negative on the whole AI thing — is too. For instance Cloudflare’s new Artifacts service uses a pure-Zig Git-protocol engine, compiled to a roughly 100 KB WebAssembly module and Vercel released fx, a Zig coding agent advertised to be small and fast. From what I can tell, all these projects are largely LLM-assisted.

But it’s not just people picking less common languages but also that they are increasingly working with “much harder” technologies. All of a sudden I have seen people do some really impressive stuff with DWARF files, eBPF, custom network drivers, custom crypto and really old computing hardware. Many of these things were previously off-limits for lots of developers. In some cases (eg: crypto) you were even pushed away because those things were intentionally gatekept by the people in the know.

So maybe the world will have more slop, but it might also have more developers in it, that want things to be fast and small.

Why start a quant trading firm?

In 2021, I was sleeping very little. Naturally, I wanted to find a way to sleep more while making money.

I dreamed about running bots that would print money for me while I lay comfortably in bed.

As a naïve 21 year old, I thought: well, it can’t be that hard.

All of my advisors told me it was a stupid idea. I had no idea who I was up against (Jump, Tower, Jane Street), with 0 domain experience.

They were right.

Prior to Manifold:

• I got into crypto in 2016/17 as a white hat hacker
• I ran validators for a couple of years
• I made some lucky trades and investments during DeFi summer and NFT mania

None of the above had anything to do with quant trading. In fact, I didn’t even know what an “alpha” was!

But as always, I didn’t listen to my advisors. Five years later, I’ve gotten better at that (right shoku?), but 21-year-old me was quite stubborn and loved learning things the hard way.

The early days

A lot of people think quant trading is just:

1. Make model
2. Find pattern
3. Execute
4. Make profits

I was one of them. And it was far from the truth.

Some of the early iterations of our trading strategies were quite embarrassing.

We went wide. Arbitrage, spread capture, basis trading, stat arb, yield farming, and even deep learning!

In the beginning, none of them worked. Trading is a zero-sum game. If enough people are playing the same game better than you, there will be nothing left on the bone.

We had to find our edge. But what would it be?

Table selection

Most top firms have an edge in one of the following:

1. Speed/latency (trading infrastructure)
2. Proprietary data/flow
3. Research/alpha generation

On centralized exchanges, the above was already completely dominated by the likes of Jump and Tower.

They already had state-of-the-art low latency infrastructure set up across all major exchanges (Binance, OKX, Bybit, Coinbase, etc). They had fee tiers that we couldn’t touch without their scale of volume. They had clients providing them with juicy flow and data.

Our trading and research infrastructure was basic at best, and we didn’t have an army of engineers and researchers to battle them on their turf.

We had to start by finding a different game, and build an advantage there.

I realized our edge had to be in DeFi, or decentralized exchanges.

Back then, most blockchain foundations were spending a crazy amount of capital incentivizing people to provide passive liquidity to decentralized exchanges. Instead of an orderbook, most decentralized exchanges used an AMM (automated market maker) model, executing trades on a pricing curve instead.

Simply put, decentralized exchanges required a unique trading system to handle both the market data (price, liquidity band/slippage) and order placement (trade execution / confirmation).

The top firms weren’t too aggressive yet in this game likely due to a) deviations from their robust infrastructure b) the market was smaller compared to classic centralized exchanges c) regulatory uncertainty around decentralized exchanges and d) they were already making a lot of money elsewhere.

The talent requirement was also a better fit for our team. We had a unique blend of crypto native engineers fluent with smart contracts (needed for executing trades on chain) working with quantitative researchers from more traditional backgrounds (from Citadel, Tower, etc).

For example, one of our best hires turned out to be a data engineer at an insurance company. He was an individual MEV searcher on the side who had been running his own atomic arbitrage strategies at small scale on the Polygon blockchain. Before he joined, I talked to him through his anonymous 0xaddress@protonmail, and I saw that he had potential scrolling through his bot address’ transaction history on a block explorer.

Before shifting the entire team’s focus into decentralized exchanges, I tested the thesis by writing a simple on-chain trading system entirely in typescript to run an arbitrage strategy between centralized and decentralized exchanges (Binance/FTX vs. top 10 blockchains in terms of liquidity). We called this CEX-DEX arb.

The reason this exists is because blockchains “block” transactions over some block time (some 200 ms, 1-2 seconds, or even several seconds). This is inherently slower than each tick on a centralized exchange, so there will always be some sort of a lead-lag. Most of the liquidity on decentralized exchanges were passive/stale liquidity and not actively managed by bots, so there would be price discrepancies between venues as price discovery mostly happened off-chain.

Even with such a crude, latency-insensitive prototype, the strategy was hitting real arbs, and was profitable after fees!

After months of research going nowhere, I had finally found a glimmer of potential.

Finding an edge

One of the exciting things about the strategy was that it was somewhat working even in its prototype state.

With arbitrage strategies, it’s rare for prototypes to work. By definition, arbitrage is risk free profit on the ground. Usually, there are other people that are readily able to pick it up (faster than you can), and the opportunity is gone or less clear.

We already knew several different ways to make the strategy better:

1. Overhaul the typescript system into a different language (faster speed)
2. Add more decentralized exchanges per chain, and add more chains (more opportunities to arbitrage)
3. More precise math (sizing, slippage)
4. Fee optimization (increase our margins per trade)
   • Our centralized exchanges were far from top fee tier, which could get lower with higher volumes
   • There were creative ways to reduce the gas fees (blockchain fee) per transaction on chain
5. Better on-chain execution techniques (increase fill rate)
6. Inventory optimization (increase rate of return, capital efficiency, and uptime)

It was all about implementation from here.

The nice thing about high frequency trading is that the market gives you immediate feedback. When you implement something and run the strategy, you get a binary result almost instantly whether it increased profitability. This gave us a very quick feedback loop on engineering to result in dollars.

Quick side story:

I remember December 24, 2021 when Sid (my co-founder) and I were sitting in our apartment office coding through the night. The two of us had no weekends or holidays. Ironically, starting Manifold forced me to work even more and sleep even less.

Through the constant keyboard clacking, neither of us even noticed it was Christmas until around 1am when I looked at the time and said “Sid, it’s Christmas!”

…

30 seconds later (I thought he completely ignored me but had forgotten about it and moved on already) he responded “oh. Merry Christmas.”

And then we went back to coding.

Looking back, this was probably my favorite office where it all started.

Improving and monetizing an edge

This piece is already longer than I thought. I need to practice writing with fewer words.

To save time, I’ll do a deeper dive on just a few of the improvements we made to CEX-DEX arbitrage that made it more profitable.

If you’re not interested in trading nitty gritty, you can skip this part. But maybe some of these techniques will inspire you to find and improve your edge in a different market, or even a completely different field. I will try to explain in words instead of mathematical formulas and code.

As we made the general improvements listed above, CEX-DEX arbitrage was starting to print over $10K per day on just a few million dollars deployed. We had focused our early efforts on getting a new system (written in Go + Solidity) good enough to scale across several chains and 10-20 DEXs on each. We also started integrating newer decentralized exchanges that adopted UniV3’s tick pricing model, which were more efficient DEXs with less price slippage due to concentrated liquidity. With more scale, our CEX trading fees came down, as we jumped up the volume thresholds. We also added more symbols to the trading universe as we added more capital.

Evolving game

As we took down the lower hanging fruits, we noticed that competition was starting to grow rapidly. On some chains, we could no longer trigger arbitrages at a 10 bp spread, because other bots were willing to trigger trades at lower. The game was evolving from a simple “identify arb then execute” bot, which had flushed out the initial hand-click arbitrageurs, to requiring slightly more sophistication.

A quick example:

ETH on Binance is $2,000
ETH on Quickswap on Polygon is $2,001

The spread is 2,001/2,000 = 0.05% = 5 bps

A bot (assuming net of fees and slippage) configured to trigger at any arbitrage opportunity at a 5 bp spread would sell ETH on Polygon and buy ETH on Binance, causing the price to revert back to approximate parity.

This means that our bot would never see an arbitrage opportunity because the other bot would close the spread before ETH on Polygon became $2,002, giving us the 10 bps spread we wanted.

Of course, everyone has a different cost to execute each arbitrage trade based on a) their fees on centralized exchanges b) whether they were taking or making on the CEX leg (making tends to be cheaper or gives you rebates in some cases) and c) their gas fees (blockchain fee to process the transaction, more on this later).

We were starting to run into other firms like Wintermute, and had to run priority-gas-auctions (PGAs) against a set of bot addresses to bid higher on specific transactions. This is a technique from the MEV (miner extractable value) world, where you constantly resubmit a specific transaction with a higher priority gas fee in order to move up the priority within a given block.

To participate in these, you had to:

1. Know almost exactly how much you could profit from hitting a certain transaction, giving you the exact fee you would be willing to pay
2. Have a reliable infrastructure that would allow you to land transactions within a block of seeing an arbitrage opportunity
3. Know which addresses you were bidding against, and quickly know when you won (or lost) the auction

There are similar / adjacent techniques on different blockchains depending on their architecture on how transactions are ordered and submitted. Blockchains constantly changed the rules and mechanisms in order to try to bring more value accrual to users/protocols/the chain itself, and arbitrageurs had to constantly adapt to them.

Gas optimizations

The total gas fees to submit an on-chain transaction is approximately a function of the gas size needed for the transaction multiplied by the gas price, which is based on the current network’s congestion.

Gas price / priority fees come into play during PGA style auctions. There isn’t much further optimization to do here as long as you are landing transactions within a block.

The gas size, however, can be reduced significantly through gas optimizations, which linearly decreases the fees required to transact on chain.

Amongst many, a fun technique we used was efficient bit packing. The ethereum virtual machine (EVM) stores data in 32 byte (or 256 bit) slots. By using smaller data types next to each other, we could get Solidity to group them into a single slot. This reduced read/write (SSTORE and SLOAD) execution costs significantly.

Now, we could go for opportunities that were “invisible” to others who had higher gas fees, because it would only be profitable by making these optimizations. For example, an arbitrage trade that would net us $1 after fees may have netted someone else $0 with no optimizations. We would close the spread before other bots would even see the opportunity.

It would also allow us to bid higher on transactions to “win” the arb while staying profitable. Over time, winning more arbs not only meant more profit but also more volume, which compounded into more advantages (better fee tiers on CEXs and hitting larger arbs in one transaction).

Capital efficiency

With more and more optimizations and improvements, we focused all our efforts into scaling the strategy. We went wide by integrating more chains, DEXs, and symbols. With a larger trading universe came higher capacity. It made sense to deploy most of our book into CEX-DEX arb, which was generating over 100% in annualized returns with no down days.

But while our book was growing rapidly from the profits, we knew that there was much more capacity for the strategy. How could we make more with what we had?

The problem at the time was that CEX-DEX arbitrage required capital scattered across venues. It also required you to have inventory ready in several different symbols.

As a quick example, if you are arbitraging between ETH/USDT on Binance vs. Arbitrum, you need ETH and USDT readily available on both venues. Let’s say that you start off with $500K of ETH and USDT each on both venues, and that Binance is trading at a premium. You would end up selling the ETH on Binance while buying ETH on Arbitrum. If the premium persists, you could quickly end up with no more ETH on Binance from selling it all, while having no more USDT on Arbitrum from buying ETH.

Centralized exchanges offer margin services which allows you to take some leverage to keep trading throughout inventory skews. But on these chains, the bots stopped arbitraging if it was tapped out of inventory. To keep it running, we would have to rebalance by moving the now excess-USDT on Binance to Arbitrum and excess-ETH on Arbitrum to Binance.

But even if we were to readily rebalance 24/7, we would still be missing out on a lot of profits. Arbitrage PnL is concentrated during periods of high market volatility, because the spreads get wider. During these moments, $500K of inventory can be used up in mere seconds. During high volatility, exchange withdrawal times go from the already slow ~5 minutes to even longer. This meant our bots couldn’t arbitrage the juiciest moments!

Our first idea was to create an automated system called Hydra, which would be constantly moving capital around, utilizing bridges between blockchains and withdrawal APIs on CEXs. We wanted the system to identify excess assets in certain venues and use them to replenish assets getting low on others. This didn’t work as intended at all. Cross-chain bridges were extremely unreliable. Assets would go missing and disappear, and we’d have to manually track which bridge failed and when. Bridging would also often take too long, and we’d miss out on the volatility anyway.

Then came version 2 of Hydra. Instead of relying on bridging, we created our own on-chain spot margin system by integrating Aave (or something similar if unavailable), a lend/borrow natively deployed on most major chains.

The idea was quite simple. In our previous example, if we bought ETH on Arbitrum, we would end up with excess ETH and run low on USDT. The system would identify the excess ETH, lend some of it out on Aave, and borrow USDT against it.

This allowed our bots to continue arbitraging through high volatility by self-leveraging its assets, and automatically unwinding and leveraging the other way if a discount on-chain turned into a premium.

Although quite simple in implementation, Hydra v2 actually had one of the greatest impacts to our PnL at the time.

Reflection

Looking back, it would have been impossible to figure out all of these techniques at once, although no single improvement was rocket science. If the initial prototype had failed miserably, maybe we wouldn’t have tried to innovate much longer. But once we had a live strategy, we were constantly iterating by monitoring our fill rates, profitability, and figuring out what we could improve / optimize based on the market’s quick feedback loop. This feedback loop helped accelerate growth.

I learned that it is very difficult to crack a game when you start off with tons of mysterious barriers. It is easier to stay ahead if you are amongst the top players (frontier) and have the foundations figured out. Especially in markets, table/game selection is just as important as execution.

Life cycle of an alpha

CEX-DEX arb was a staple profit generator for quite a long time. During its glory days, it printed over six figures a day when paired with high market volatility. We generalized the infrastructure to a point where we could integrate a new chain within an hour. This consistently made us the first arbitrageur on some new chains during times of maximum inefficiency, where we could charge massive spreads.

But as with almost any alpha, it decays over time. With arbitrage especially, competition will reduce the spreads to levels where the margins get much lower. Imagine two firms with exactly equal gas optimizations and fee tiers on CEXs in our PGA example from earlier. Because they have the same cost per transaction, they will end up bidding up the PGA to their minimum profitability. By the end of 2025, CEX-DEX arb was no longer the crazy high-yielding strategy it once was.

Fortunately, by having a stable high-yielding strategy running in the background, we were able to re-invest our time and capital into other, even greater profit generating trading strategies over time.

I could write more about all the other crazy things we came up with in the years after, but that’s a story for another time. CEX-DEX arb still has a special place in my heart because it gave us a legitimate starting point and gave us an anchor to build our position from.

DigitalOcean Inference Router, Now Cache-Aware: Why the Cheapest Model Isn't Always the Best Deal

Coinbase CEO Brian Armstrong recently posed the question every company scaling AI is asking: how do you keep spend flat while token usage grows exponentially? This isn’t hypothetical. It’s confronting companies across every sector:

• Uber exhausted its annual AI coding budget within the first four months of the year and subsequently introduced a $1,500 monthly limit per employee.
• Walmart placed token limits on its internal Code Puppy agent after employees repeatedly asked it to solve similar problems.
• A Priceline employee reported that a routine Cursor renewal came back 4-5x more expensive.

Usage caps may help control the bill, but they also limit productive work. A better answer is to improve the economics of every request through better defaults, routing, and making caching work for your specific workload scenarios.

Today, we’re making DigitalOcean Inference Router cache-aware. In April, we launched preference-aware routing, so our router could match each request to the model that best fit a developer’s task and priorities. Now, it can also account for the value of context that’s already cached. This advances Inference Router from selecting the right model for each request toward optimizing the entire agentic session across quality, cost, and cache locality. With this release, our Inference Router now offers a comprehensive set of controls for you to build an intelligence layer that fits how your team actually works.

A warm cache can be more valuable than a cheaper model

Caching is a critical consideration when building agents, because they repeatedly send the same large body of context: system instructions, tool definitions, repository context, and an accumulating conversation history.

Here’s how top providers are putting caching to work:

• Z.ai uses a 90.9% cache-hit rate as its average assumption for coding workloads when estimating usage for its coding plans.
• Anthropic shares that Claude Code uses prompt caching to make back-to-back calls cheaper and faster.
• OpenAI reports that cached prompts can reduce latency by up to 80%.

For agents, caching is not a marginal optimization. It shapes the cost and latency of almost every subsequent model call. At DigitalOcean, we are seeing this first-hand as we scale more models on behalf of customers. With the recent release of Kimi K3, we’ve observed an aggregate cache-hit rate of 90%+ across our own workloads as developers use the model for coding and long-horizon tasks; individual workloads will differ.

Cache-aware routing changes the economics of model routing. Consider, as an illustration, an agent with 90,000 input tokens already cached on Claude Sonnet 5 out of a 100,000 token context. At the standard pricing of $2.5 per million input tokens (with cache writing enabled) and $0.2 per million cached tokens, a 90% cache hit makes the next request cost approximately $0.043 in input tokens. (Pricing information is current as of the publication date.)

While GLM‑5.2 appears cheaper at its $0.7 per million uncached input rate, switching models discards the warm cache and forces re-processing of the full 100,000 input token context. On the assumption in this illustration, that request would cost $0.07: approximately 1.6 times more than staying on the nominally more expensive model in this scenario. Sticking with the warm model requires prefilling only the 10,000 uncached tokens; switching requires all 100,000—10x more prompt processing before generation can even begin. That doesn’t translate into a 10x latency increase, since prefill performance varies by model and serving system. But it does explain why a cache-breaking switch can meaningfully increase time to first token, even when the destination model is otherwise faster.

How Inference Router supports cache-aware routing

Before the launch of cache-aware routing, Inference Router evaluated each request independently. It could correctly determine that another model was more affordable or better suited to the context presented, but didn’t recognize that the request belonged to an ongoing agent session with a warm prompt cache.

But the act of switching models can invalidate the existing cache and force the destination model to process the entire prompt again. For agents that repeatedly send large system instructions, tool definitions, repository context, and conversation history, using the “cheaper” model can make the next request more expensive and slower. Another complication is that it can also change model behavior partway through an agent’s loop.

When customers told us they needed more control over that tradeoff, we built cache-aware routing. It introduces two complementary mechanisms: explicit model affinity for applications that already manage sessions, and a routing-budget policy that determines when breaking affinity is worth the additional cost.

Explicitly associate requests with a custom HTTP header: X-Model-Affinity

Applications that already maintain session or task identifiers can pass an explicit affinity key with each request:

import os
import uuid
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["MODEL_ACCESS_KEY"],
    base_url="https://inference.do-ai.run/v1/",
)

session_id = str(uuid.uuid4())
messages = []
user_turns = [
    "Help me debug this failing test.",
    "Here's the stack trace, what's causing it?",
    "That fixed it, now can you also add a regression test?",
]

for user_turn in user_turns:
    messages.append({"role": "user", "content": user_turn})

    response = client.chat.completions.create(
        model="router:<your-router-name>",
        messages=messages,
        extra_headers={
            "X-Model-Affinity": session_id,
        },
    )
    assistant_reply = response.choices[0].message.content
    messages.append({"role": "assistant", "content": assistant_reply})

The first request is routed according to the developer’s configured task, model pool, and routing preferences. Requests with the same X-Model-Affinity value are then treated as part of the same unit of work, allowing Inference Router to preserve the session’s model binding and reuse its cached context. Affinity identifiers should represent meaningful units of work: a coding session, research task, support conversation, or individual agent run. When the application starts a genuinely new task, it can provide a new identifier, allowing the Inference Router to make a fresh preference-aware decision.

For common agentic requests, Inference Router can also infer affinity if an explicit identifier is not available. It derives a stable session key from the request context that remains unchanged across turns, including system and developer instructions, tool definitions, and the first user message. If that stable prefix changes, Inference Router treats the cache as cold and establishes a new binding. This reassigns the session to a model, which then starts accumulating its own warm cache from scratch.

Control cache-breaking switches with a routing budget

Model affinity headers are ideal for applications that already track meaningful units of work—such as research tasks and support conversations—and want deterministic control over which requests share the same model binding. With this release, we’ve also introduced the routing budget: a complementary control that keeps Inference Router evaluating alternative models without requiring any changes to your application code.

When the routing policy proposes switching models, Inference Router calculates the incremental cost of leaving the session’s warm cache. It does this by comparing the cached input cost of staying on the current model with the uncached cost of rebuilding the context on the candidate model, then evaluates that cost against the session’s cumulative switching spend.

Developers can define this trade-off with a maximum switching budget, set relative to what the session would have cost had it stayed on the existing model. For example, X-Routing-Max-Switch-Spend-Pct: 20 limits cumulative switching costs to 20% above that baseline. Model selection and economics remain separate: the router identifies its preferred model, while the routing budget determines whether switching to it is worth the additional input cost.

curl -i "https://inference.do-ai.run/v1/chat/completions" \
  -H "Authorization: Bearer $MODEL_ACCESS_KEY" \
  -H "Content-Type: application/json" \
  `# X-Model-Affinity is managed by the router if not set explicitly` \
  -H "X-Routing-Max-Switch-Spend-Pct: 20" \
  -d '{
    "model": "router:software-engineering",
    "messages": [
      {"role": "user", "content": "Summarize this in 2 sentences."}
    ]
  }'

Together, developers can use these controls to choose the appropriate level of involvement:

• Use X-Model-Affinity when the application already has an authoritative session or task identifier.
• Let Inference Router automatically detect affinity based on stable agent context across related turns.
• Configure X-Routing-Max-Switch-Spend-Pct to control how much additional input cost the router can incur by switching models.

New Analyze Page

We have updated the Analyze page to give you detailed visibility into how your router makes cache-aware decisions.

In the top-line router view, you can quickly get answers to questions like:

• What is overall caching efficiency?
• How many requests switched models?
• How many were held to keep the cache warm and what is the overall latency as a result?

This lets you get a high level snapshot of your router at a glance.

From here, you can drill down further into the behavior of specific models and tasks to get a more detailed understanding of the traffic mix. This makes it easier to identify specific model hotspots, validate routing strategy, and tune router preferences over time.

We’ve also added trend tracking for cache efficiency giving you visibility at both request and token level over time. This makes it easier to spot cache regressions, understand performance trends, and validate the impact of prompt and cache tuning changes.

Together, these views help teams move from high-level monitoring to targeted optimization right from the Inference Router UI.

Routing that starts with developer preferences

When we launched DigitalOcean Inference Engine and Inference Router, we gave developers a way to define tasks, create model pools, and express whether they wanted to optimize for quality, cost, or latency. Inference Router then semantically matches each request to a task and applies those preferences to select a model. Developers can start with DigitalOcean presets—opinionated, routinely updated model selections informed by our evaluations—or define their own tasks, model pools, and priorities. Either path works out of the box: no router training or application-side routing logic required. Early customer LawVo reported reducing inference costs by more than 40% while maintaining the accuracy, speed, and reliability its users expected*.

This approach is grounded in years of research into preference-aware routing. In Arch-Router: Aligning LLM Routing with Human Preferences, our team introduced a compact 1.5-billion-parameter model that maps requests to developer-defined domains and actions and can incorporate new models without retraining. We published the model with open weights—the broader approach remains available through Plano, our Apache-licensed open-source AI proxy and data plane. That research originated from a simple observation: benchmarks are useful, but they are not preferences.

Model benchmarks are maps, not routing tables

Benchmarks let us compare models under controlled, repeatable conditions. They help narrow a large model catalog, identify broad strengths, and bootstrap routing before an application has enough real-world traffic to run its own evaluations. That makes them a valuable starting point for DigitalOcean presets.

But model performance is conditional on the surrounding application: the system prompt, tool definitions, context, output constraints, conversation history, and definition of success. Change the agent harness, and the relative ranking of models changes with it. A model that performs best on an isolated coding benchmark may not be the ideal choice for use within a coding agent operating across a large repository with dozens of tools and a long conversation history.

Relatedly, one developer may prefer a particular model’s visual style for image generation, while another may prioritize instruction following, tool-call reliability, latency, or cost. Neither preference can be inferred from a general-purpose leaderboard. Preselecting a model on benchmark scores alone is not intelligent routing. Routing is only intelligent once it knows what the developer is optimizing for. Over time, it becomes a personalization problem. But even a preference-aware router can make the wrong economic decision if it evaluates every request in isolation.

Better defaults, better routing, and better caching

Across sectors, the knee-jerk response to rapidly growing inference bills has often been to ration access. Yet Coinbase has publicly reported that 91% of Coinbase employees were not reaching their existing usage caps. Lowering those caps would have generated more alerts and friction without addressing what actually drove most of the spend. Coinbase instead moved toward cheaper defaults, task-aware routing, and better caching, which it reports improved LibreChat’s cache hit rate from 5% to 60%.

These three controls reinforce one another:

• Better defaults prevent every request from beginning on the most expensive model.
• Preference-aware routing selects models based on the task and the developer’s values.
• Cache-aware routing preserves the accumulated economic value of an agentic session instead of discarding it between turns.

A cheap default may not meet the quality bar for a complex task. A benchmark-driven router may not reflect an application’s real evaluations. A cache-aware system should not preserve a warm model when it is no longer appropriate for the work. No single technique is sufficient on its own. The objective is not to maximize tokens or blindly minimize their price, but rather to maximize useful intelligence per dollar spent while preserving the quality, latency, and reliability each application requires.

Routing is only intelligent when it understands what you are optimizing for and what switching away from an in-progress task actually costs. The DigitalOcean Inference Router gives you the control and visibility to build an intelligence layer that fits how your team actually works. Use it now to create a preset or custom router and add model affinity to your next agentic workflow. All figures in this post are illustrative and based on the pricing, models, and configurations available as of the publication date; third-party figures are as reported by those parties. Results and savings vary with configuration, implementation, and usage, and are not guaranteed. All marks are the property of their respective owners, and no affiliation or endorsement is implied.

*Disclaimer: This reflects LawVo’s own reported experience in its own environment and is not necessarily representative of results other customers will achieve.

PostgreSQL backup tool

Databasus is a free, open source and self-hosted tool to backup PostgreSQL. Make backups with different storages (S3, Google Drive, FTP, etc.) and notifications about progress (Slack, Discord, Telegram, etc.). With a focus on Point-in-Time Recovery at low RPO/RTO

🌐 Databasus website

✨ Features

📦 Backup types

• Physical: file-level copy of the entire database cluster over PostgreSQL native incremental backups mechanism
  • Full: a complete, self-contained copy of the cluster
  • Incremental: stores only what changed since the previous full backup, so backups stay small and fast
  • WAL streaming: continuously captures the database write stream, enabling Point-in-time recovery (PITR). Designed for disaster recovery and near-zero data loss
• Logical: native dump of the database in its engine-specific binary format (compressed, suitable for parallel restore)

🔄 Scheduled backups

• Flexible scheduling: hourly, daily, weekly, monthly or cron
• Precise timing: run backups at specific times (e.g., 4 AM during low traffic)
• Smart compression: 4-8x space savings with balanced compression (~20% overhead)

🧪 Restore verification

Databasus performs a real restore to confirm backups are usable, not just intact on disk or checksum check.

• Triggers: after each backup or on a flexible schedule (hourly, daily, weekly, monthly or cron)
• Real restore: spins up a database container, runs the restore and checks the restored size against the backup
• Report: lists every table with its row count
• Optional notifications: send the report or failure-only alerts through any configured notifier

🗑️ Retention policies

• Time period: Keep backups for a fixed duration (e.g., 7 days, 3 months, 1 year)
• Count: Keep a fixed number of the most recent backups (e.g., last 30)
• GFS (Grandfather-Father-Son): Layered retention — keep hourly, daily, weekly, monthly and yearly backups independently for fine-grained long-term history (enterprises requirement)
• Size limits: Set per-backup and total storage size caps to control storage usage

🗄️ Multiple storage destinations

• Local storage: Keep backups on your VPS/server
• Cloud storage: S3, Cloudflare R2, Google Drive, NAS, Dropbox, SFTP, Rclone and more
• Secure: All data stays under your control

📱 Notifications

• Multiple channels: Email, Telegram, Slack, Discord, Teams, Mattermost, webhooks
• Real-time updates: Success and failure notifications
• Team integration: Perfect for DevOps workflows

🔒 Enterprise-grade security

• AES-256-GCM encryption: Enterprise-grade protection for backup files
• Zero-trust storage: Backups are encrypted and remain useless to attackers, so you can safely store them in shared storage like S3, Azure Blob Storage, etc.
• Encryption for secrets: Any sensitive data is encrypted and never exposed, even in logs or error messages
• Read-only user: Databasus uses a read-only user by default for backups and never stores anything that can modify your data

👥 Suitable for teams

• Workspaces: Group databases, notifiers and storages for different projects or teams
• Access management: Control who can view or manage specific databases with role-based permissions
• Audit logs: Track all system activities and changes made by users
• User roles: Assign viewer, member, admin or owner roles within workspaces
• OpenTelemetry logs: Export application and audit logs to an external system (by default they are also written to a local file)

🎨 UX-Friendly

• Designer-polished UI: Clean, intuitive interface crafted with attention to detail
• Dark & light themes: Choose the look that suits your workflow
• Mobile adaptive: Check your backups from anywhere on any device

💾 Supported databases

• PostgreSQL: 14, 15, 16, 17 and 18 (physical and logical)
• MySQL: 5.7, 8.0, 8.4 and 9 (logical only)
• MariaDB: 10, 11 and 12 (logical only)
• MongoDB: 4.2+, 5, 6, 7 and 8 (logical only)

🐳 Self-hosted & secure

• Docker-based: Easy deployment and management
• Privacy-first: All your data stays on your infrastructure
• Open source: Apache 2.0 licensed, inspect every line of code
• Build-in SSH: Connect to your databasus via SSH tunnel

📦 Installation

You have four ways to install Databasus: automated script (recommended), simple Docker run, or Docker Compose setup.

Option 1: Automated installation script (recommended, Linux only)

The installation script will:

• ✅ Install Docker with Docker Compose (if not already installed)
• ✅ Set up Databasus
• ✅ Configure automatic startup on system reboot

sudo apt-get install -y curl && \
sudo curl -sSL https://raw.githubusercontent.com/databasus/databasus/refs/heads/main/install-databasus.sh \
| sudo bash

Option 2: Simple Docker run

The easiest way to run Databasus:

docker run -d \
  --name databasus \
  -p 4005:4005 \
  -v ./databasus-data:/databasus-data \
  --restart unless-stopped \
  databasus/databasus:latest

The same image lives on GitHub's registry — use ghcr.io/databasus/databasus:latest if Docker Hub rate-limits your pull.

This single command will:

• ✅ Start Databasus
• ✅ Store all data in ./databasus-data directory
• ✅ Automatically restart on system reboot

Option 3: Docker Compose setup

Create a docker-compose.yml file with the following configuration:

services:
  databasus:
    container_name: databasus
    image: databasus/databasus:latest
    ports:
      - "4005:4005"
    volumes:
      - ./databasus-data:/databasus-data
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "databasus", "healthcheck"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s

Then run:

docker compose up -d

Option 4: Kubernetes with Helm

For Kubernetes deployments, install directly from the OCI registry.

Add --set image.repository=ghcr.io/databasus/databasus to any of the commands below to pull image from GHCR instead of Docker Hub.

With ClusterIP + port-forward (development/testing):

helm install databasus oci://ghcr.io/databasus/charts/databasus \
  -n databasus --create-namespace

kubectl port-forward svc/databasus-service 4005:4005 -n databasus
# Access at http://localhost:4005

With LoadBalancer (cloud environments):

helm install databasus oci://ghcr.io/databasus/charts/databasus \
  -n databasus --create-namespace \
  --set service.type=LoadBalancer

kubectl get svc databasus-service -n databasus
# Access at http://<EXTERNAL-IP>:4005

With Ingress (domain-based access):

helm install databasus oci://ghcr.io/databasus/charts/databasus \
  -n databasus --create-namespace \
  --set ingress.enabled=true \
  --set ingress.hosts[0].host=backup.example.com

🚀 Usage

1. Access the dashboard: Navigate to http://localhost:4005
2. Add your first database for backup: Click "New Database" and follow the setup wizard
3. Configure schedule: Choose from hourly, daily, weekly, monthly or cron intervals
4. Set database connection: Enter your database credentials and connection details
5. Choose storage: Select where to store your backups (local, S3, Google Drive, etc.)
6. Configure retention policy: Choose time period, count or GFS to control how long backups are kept
7. Add notifications (optional): Configure email, Telegram, Slack, Mattermost or webhook notifications
8. Save and start: Databasus will validate settings and begin the backup schedule

🔑 Resetting password

If you need to reset the password, you can use the built-in password reset command:

docker exec -it databasus ./main --new-password="YourNewSecurePassword123" --email="admin"

Replace admin with the actual email address of the user whose password you want to reset.

💾 Backuping Databasus itself

After installation, it is also recommended to backup your Databasus itself or, at least, to copy secret key used for encryption (30 seconds is needed). So you are able to restore from your encrypted backups if you lose access to the server with Databasus or it is corrupted.

🛡️ Security & reliability engineering

Databasus works with sensitive data, so preventing vulnerabilities, unauthorised access and data leaks is a primary concern. We invest in this on both sides of the system: in the code itself (permission checks, encryption, careful handling of secrets) and in the infrastructure around it (dependency analysis, CVE response, DevSecOps best practices). The pipeline below runs automatically on every commit and PR. No single layer is enough on its own, but together they reduce the chance of vulnerable code, unsafe dependencies, broken images, or non-restorable backups reaching a release.

For static analysis we combine several independent passes. CodeQL scans the full codebase for security issues. CodeRabbit reviews every PR and runs gitleaks for secret scanning and semgrep for security rules inline. Dockerfiles and CI workflows get extra rules of their own (pinned action references, least-privilege permissions, suspicious base images), so insecure patterns are flagged before they ever merge. On top of these per-PR checks, Codex Security from OpenAI runs regular, deeper audits of the whole codebase. It's a separate program that catches architectural and cross-cutting issues narrow PR-time scans can miss.

On the dependency side, Dependabot watches all of our dependencies against the GitHub Advisory Database and surfaces CVEs within minutes of publication. Updates run through a cooldown so newly-published versions get a chance to mature before we adopt them. This is a deliberate defence against compromised-package incidents like supply-chain attack. The Dependency Review Action blocks any PR that introduces a new HIGH or CRITICAL CVE outright.

Container images are scanned with Trivy on every build. A separate Trivy pass on the Dockerfile catches misconfigurations before they make it into an image. All GitHub Actions are pinned to full commit SHAs rather than floating tags like @v4 or @main, which have been an active attack vector in 2025. Workflows default to least-privilege permissions and only elevate per-job when genuinely needed.

Critical paths are covered by both unit and integration tests, run against real database containers for every supported engine and major version. Restore is the path that matters most for a backup tool, so we test it explicitly: every PR runs full backup-then-restore cycles against those same real containers, verifying that backups can actually be restored end-to-end, not just written successfully. The rest of the CI/CD pipeline runs lint, type-check, the full test suite, image smoke tests and multi-architecture builds on every PR. A release only ships if all of it passes.

📝 License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details

🤝 Contributing

Contributions are welcome! Read the contributing guide for more details, priorities and rules. If you want to contribute but don't know where to start, message me on Telegram @rostislav_dugin

Also you can join our large community of developers, DBAs and DevOps engineers on Telegram @databasus_community.

AI disclaimer

There have been questions about AI usage in project development in issues and discussions. As the project focuses on security, reliability and production usage, it's important to explain how AI is used in the development process.

First of all, we are proud to say that Databasus has been accepted into both Claude for Open Source by Anthropic and Codex for Open Source by OpenAI in March 2026. For us it is one more signal that the project was recognized as important open-source software and was as critical infrastructure worth supporting independently by two of the world's leading AI companies.

Despite of this, we have the following rules how AI is used in the development process:

AI is used as a helper for:

• verification of code quality and searching for vulnerabilities
• cleaning up and improving documentation, comments and code
• assistance during development
• double-checking PRs and commits after human review
• additional security analysis of PRs via Codex Security

AI is not used for:

• writing entire code
• "vibe code" approach
• code without line-by-line verification by a human
• code without tests

So AI is just an assistant and a tool for developers to increase productivity and ensure code quality. The work is done by developers.

Moreover, it's important to note that we do not differentiate between bad human code and AI vibe code. There are strict requirements for any code to be merged to keep the codebase maintainable.

Even if code is written manually by a human, it's not guaranteed to be merged. Vibe code is not allowed at all and all such PRs are rejected by default.

AeroSpork — an i3-like tiling window manager for macOS

AeroSpork is an i3-style tiling window manager for macOS. Windows are leaves of a layout tree, workspaces are emulated rather than mapped onto native Spaces, and nothing requires disabling System Integrity Protection. It is configured in TOML, driven from a CLI, and ships a settings GUI.

It is a fork of AeroSpace by Nikita Bobko, which is where the tree model, the workspace emulation and most of the command surface come from. Both are MIT licensed.

Why I forked it

I ran AeroSpace daily on four monitors behind a DisplayLink dock, and three things wore me down. It felt sluggish. Long sessions drifted, so state that was correct at login was not correct by the evening. And the DisplayLink panels were a coin flip: workspaces came back on the wrong screens after every undock, because monitors are matched by name, regex or index, and none of those survive a redock. Two identical displays are indistinguishable to a name match.

I sent the monitor work upstream as PR #1526 in July 2025. It was closed the next day without review. That is the maintainer's call on their own project, and upstream is clear that it keeps a deliberately small surface, so I kept the work here instead.

What that turned into, in this codebase:

• The DisplayLink problem is model/MonitorFingerprint.swift. A display is matched on the per-display UUID first, then EDID vendor/model/serial from CoreGraphics, then name, then size. DisplayLink panels report no EDID at all, so the UUID is the only key that separates two of them. Screen reconfiguration is also debounced, because a dock connects in several stages and fires the change notification more than once.
• The sluggishness is two changes rather than a rewrite. Bursts of accessibility events coalesce into one layout pass on a 50ms debounce (util/RefreshDebouncer.swift), and MacApp.setFrame skips the AX write when a window already sits at its target frame, which matters over a DisplayLink link where every write repaints a framebuffer. I have not published speedup numbers; dev-docs/performance.md says which measurements exist and why the benchmark could not settle the rest.
• The drift is mostly workspace lifecycle. Workspaces are created on demand and released when they empty, instead of being materialized for every name a keybinding mentions.

Tech stack

Why these choices

If you are weighing this against upstream, the reasoning matters more than the table.

Two dependencies instead of four, and each removal was a wrapper going away. BlueSocket was wrapping a local Unix socket, so it became AF_UNIX directly. HotKey was wrapping Carbon's RegisterEventHotKey, which is one call plus the bookkeeping to unregister it. ISSoundAdditions was wrapping CoreAudio. swift-collections supplied one ordered dictionary. The ANTLR-generated shell grammar parsed command strings that /bin/bash -c already parses. Every one of those is a thing that can break on an OS update, or need a version bump before the app can be rebuilt, in exchange for code the platform already provides. Sparkle is the one addition, and only because there is no App Store update path to inherit.

Display identity comes from CoreGraphics, not IOKit. Upstream reads EDID through IOServiceMatching("IODisplayConnect"). That IOKit class does not exist on Apple Silicon, so the iterator yields nothing and vendor/model/serial come back nil for every display. CoreGraphics returns the same values and adds the per-display UUID, which is the only field that survives a DisplayLink dock.

The private _AXUIElementGetWindow stays, and it costs something. It is the only way to get a window id that is stable across refreshes, and window identity is what the whole tree is keyed on. The price is that the Mac App Store is permanently out: private symbols fail review, and the Accessibility APIs this app is built on do not work in a sandbox anyway. Hence Developer ID signing, notarization, and Sparkle rather than TestFlight.

Workspace placement survives a restart of AeroSpork. Workspaces are emulated, so nothing outside the process knows a window belongs to one; at a cold start a window is bound by where it physically sits, and the workspace chosen for each monitor is the first key-bound name in sort order — which a named workspace like A can never be. Placement is now remembered, keyed on the window id the macOS window server issues. That id is stable for exactly as long as that server runs, so an update, a crash or a Quit keeps it and a logout, a reboot or an application relaunching does not. Where the id is gone there is no honest way to recognise a window — every terminal window reports the same accessibility identifier — so it falls back to placing by location rather than guessing, and on-window-detected remains the way to state intent.

The Xcode project is generated, not committed. project.yml plus XcodeGen produces it, because SwiftPM cannot build an app bundle but a checked-in .pbxproj is a merge conflict waiting to happen. Debug builds skip Xcode entirely.

The config writer is line-based on purpose. Re-serializing the whole file would be far simpler, and would destroy every comment plus anything the GUI cannot model, such as per-monitor gap arrays. Instead it rewrites only the keys you changed, preserves what the panes cannot model (a rich monitor fingerprint, a fallback list) field for field on every save, and refuses the few shapes it cannot rewrite safely, pointing you at the Raw TOML pane. That is what makes a GUI safe to put on top of a dotfile.

Sources/
├── aerosporkApp/    # app entry point (@main)
├── AppBundle/       # the window manager: tree/, layout/, command/, config/, model/, mouse/, ui/
├── Cli/             # command-line client
├── Common/          # shared with the CLI, incl. the socket implementation
└── PrivateApi/      # C shim for _AXUIElementGetWindow

Differences from AeroSpace

Config schema. mod plus workspaces generates the usual i3 keymap, and [keys], [monitors] and [on-window] replace the longer upstream spellings. An existing config is migrated once on first launch, and only when the result is proven to parse to the same effective configuration; otherwise the file is left alone. The original is kept beside it as *.pre-v2.

Settings GUI. Seven native preference panes in a compact macOS toolbar, over a comment-preserving writer that only rewrites the keys you changed. Opening Settings and changing nothing leaves the file byte-identical, and editing one section never rewrites another. The selected pane is remembered. Raw TOML validates against the same parser the app uses at startup and adds native Find, line numbers, restrained syntax highlighting, section navigation, cursor position, and clickable source diagnostics, so no config key is unreachable from the GUI.

Coming from AeroSpace

A fork, not a drop-in replacement. Configs and scripts need small edits.

• AEROSPACE_* environment variables are gone and not aliased. A script reading $AEROSPACE_FOCUSED_WORKSPACE gets an empty string with no error. The names are AEROSPORK_FOCUSED_WORKSPACE, AEROSPORK_PREV_WORKSPACE, AEROSPORK_WINDOW_ID and AEROSPORK_WORKSPACE.
• if.during-aerospace-startup is spelled if.during-aerospork-startup. Unknown keys are fatal, so the old spelling fails at startup and names the line.
• Feature parity is a non-goal. The fork carries less surface area than upstream.

Installation

Download the notarized universal (arm64 + x86_64) build from the releases page, move AeroSpork.app to /Applications, and grant Accessibility permission when prompted. A Homebrew cask is published at wbsmolen/tap:

brew install --cask wbsmolen/tap/aerospork

Installed copies check for updates themselves through Sparkle, against a signed appcast served from aerospork.app/appcast.xml. Updates are verified against an EdDSA public key in the app's Info.plist, so a build refuses anything it cannot verify. Automatic checking is off until you allow it; Check for Updates… in the menu bar checks on demand. There is no App Store update path to inherit, because the Accessibility APIs this app is built on do not work in a sandbox.

Because update checks are the only network request AeroSpork can make, and the Accessibility permission it needs is a broad one, aerospork.app/privacy.html sets out exactly what is stored, what is sent, and what is not: no analytics, no telemetry, and no system profile.

Configuration

AeroSpork reads whichever of these exists, and reports an error at startup if both do: ~/.aerospork.toml or ${XDG_CONFIG_HOME}/aerospork/aerospork.toml (XDG_CONFIG_HOME defaults to ~/.config). With neither, it falls back to a complete default bundled in the app. Saved changes hot-reload, so you never need to run reload-config by hand.

mod = "alt"                 # generates the i3 keymap: alt-h/j/k/l, alt-shift-h/j/k/l, ...
workspaces = "1-9"          # alt-1..9 to switch, alt-shift-1..9 to move a window

[gaps]
inner = 8
outer = 8

[keys]                      # anything here overrides a generated binding
alt-enter = "exec-and-forget open -na Ghostty"

[monitors]                  # pin a workspace to a screen
1 = "main"
2 = { uuid = "AAAAAAAA-0000-4000-8000-000000000001" }

[on-window]                 # where a window goes when it appears
"com.apple.mail" = "move-node-to-workspace 3"

CLI

36 subcommands, with man pages and bash/fish/zsh completion.

aerospork focus left                        # focus the window to the left
aerospork workspace 1                       # switch workspace
aerospork move-node-to-workspace 2          # move the focused window
aerospork layout tiles horizontal vertical  # cycle layout
aerospork list-monitors                     # connected displays and how they are identified
aerospork --help

Development

./build-debug.sh     # SwiftPM debug build into .debug/ (uses ~/.aerospork-debug.toml)
./build-release.sh   # signed release; needs a Developer ID Application certificate
./run-tests.sh       # tests, format and lint
./build-docs.sh      # man pages and docs site

License

MIT. The original AeroSpace copyright is retained alongside the fork's in LICENSE.txt. Active development; features and configuration may still change.

Full article content is not available for inline reading.

Read the original article →

Netflix is moving 30,000+ Flink jobs from a homegrown autoscaler toward the Apache Flink Autoscaler. The old system saved 25–45% by scaling whole clusters from container metrics, but missed operator-level state and suffered from telemetry gaps. The OSS autoscaler reasons from inside each job and produced 58% annualized compute savings for one team.

A forecast that carries weight

The Forecasting Data Science team at Airbnb produces many of the forecasts the rest of the company plans around: demand, bookings, cancellations, and a range of finer cuts by market and segment, refreshed continuously across thousands of markets. The targets differ, and the models differ, but they have one thing in common: Other teams build on top of them.

This means a forecast that is casually wrong is not a clean miss, as it might be in an academic setting. That’s because a small bias does not stay small once a lot of decisions are riding on it. So when one of those forecasts starts to drift, what to do about it is not really a methods question. It is a risk question, and an easy one to get wrong, which we have from time to time.

One of these forecasts had been missing, compared to what actually happened after the forecast was released, in the same direction for a couple of quarters. This bias persisted after several routine refreshes. The usual solution would be to fully retrain the model: pull in the recent data, refit the model again, and ship. But we wanted to understand the source of the bias, rather than simply hoping an update would eliminate it.

If you’re interested in other posts on this topic, you can learn more about how COVID impacted Airbnb’s financial models or how we dealt with disruption to our models during the pandemic. This post is about the discipline that came out of both: how we now decide whether a struggling forecast needs new data, a new model, or no changes at all.

One word, three decisions

The easy mistake is treating the choice to “retrain” a model as a single action. It is three separate actions — refitting, respecifying, or holding — and none of them is particularly similar to the others.

Refitting is the cheaper option. Same model, same structure, same features, updated with newer data. This is what most people mean by “retrain.” For the ordinary drift that builds up in a model as the world moves on, it’s usually the right choice.

Saying that this is the cheaper option is not the same as saying that it’s cheap. A refit still has to be validated and shipped. The refitting process can degrade a model that was actually fine if the recent data happens to be unusual. And, given the importance of the production forecast, every refresh is a small risk you are choosing to take. It is the least expensive of the three options, but not a free one. None of this is an argument against refitting. On most cycles it is the right move, and if we had to run one of the three blind, it would be this one. The claim is narrower. A refit is a decision with a price, and pricing it is what lets you notice the cycles where a different option was worth more.

Respecifying is a different animal. You change the model itself: add a feature, drop one, change the structure, the priors, the likelihood. On the production forecast, which so many of our teams plan around, this is a real commitment; you are replacing something you understand, and have watched the behavior of for years, with something partly new that you have not. It is also where almost all the actual improvement lives. Refitting keeps a good model current. Respecifying is how a model that is in some way wrong in its operations is made right.

Holding is the one nobody likes. You look at the miss, decide it does not warrant action, and leave the model alone. This takes the most nerve, because to anyone watching the forecast it looks like you are ignoring a problem. On a forecast that people are relying on, “We decided to do nothing” is a hard sentence to say out loud. It is also, more often than you would think, the correct call.

What we have learned from years of running these models is that the trouble comes from filing all three under one verb. “We should retrain” gets said when any of the three is the actual answer, and the word drags everyone toward the one that feels cheapest, because it is the one with a name.

Most of the time, though, nobody says it at all. Retraining is rarely a decision someone makes because a model is drifting. Retraining runs on a schedule, monthly or quarterly, by convention, because a standing cadence is one less thing to think about. Which means the choice among the three is hardly ever made on purpose. The cadence makes the choice in advance, every cycle, and the cadence always picks refit.

Three ways to hold on too long

Each of the options contained in the term “retrain” — refit, respecify, or hold — has a matching failure, and underneath each sits the same mistake: a model holding onto a shock, or a surprise, after the cause of the shock or surprise is no longer relevant to forecasts. That mistake can grab a blip and treat it as the new normal; keep a crisis assumption after the crisis is over; or build the shock so far into a new model that the model can never move past the shock.

Chasing noise. A forecast misses for a quarter, the residuals look alarming, and someone asks whether we should update the model now rather than wait for the next scheduled run. That impulse is the failure, and it is a specific one. What triggers an off-cycle refit is always a surprise, which means the window you are rushing to absorb is the window you understand least.

A couple of years ago one of our markets ran hot for a quarter against the published forecast, because a large event landed on the calendar and pulled a wave of bookings forward. We refit early. The model took the unusual quarter as the new level, the next two forecasts came in high, and it settled only once the event aged out of the training window. Waiting for the cadence would have meant fitting that spike alongside the quarters that came after it, rather than as the last thing the model saw, which is a much weaker pull.

Carrying ghosts. The quieter error, and the opposite mistake. A structural assumption that was true during a shock stays switched on long after the shock has passed, hidden behind a run of refits that all look like routine maintenance. We have had one of these hide in plain sight. During the worst of the COVID disruption, cancellation timing shifted in a way it never previously had, and the model learned from it. Market behavior came back to normal well before the assumption did, and since every refit looked clean, the stale piece sat there until the forecast had leaned the same way for long enough that someone went looking, found the lean baked into the model, and respecified the model to wring out the now-faulty assumption.

Respec-as-panic. The failure of overcorrecting. Something genuinely moves — a foreign exchange rate (FX) swing from a major source market, a regional disruption that reroutes demand — and the reflex is to rebuild. A fresh model, new structure, stood up under deadline.

We have caught ourselves reaching for this solution. An FX move on a major source market threw one of our forecasts off for a couple of months, and the first instinct was to stand up a new model around the new regime. We widened the priors instead, the existing model rode it out, and we avoided a move to a more fragile model, tuned in response to a shock that was already on its way out.

How we decide now

The rule we now hold to is easy to state and hard to follow. On a standing cadence, the question of whether to retrain mostly answers itself. So the question that matters is which of the three options a given miss calls for: refit, respecify, or hold? And the right answers turns on what changed.

If the process generating the data is still the one the model assumes, and the parameters have just drifted, refit. The structure is fine, it just needs current numbers.

If the process has changed in a way the model cannot represent no matter how you estimate it, respecify, because fresh data cannot help a model being asked to describe a world it has no language for. The tell is direction: a misspecified model misses the same way over and over.

A respecify can cut both ways. It can add structure the model was missing, the way borrowing across geography did after 2020, or it can take structure out, retiring an assumption the world has outgrown.

The second kind is the forgetting, and it is the one teams skip, because adding feels like progress and removing feels like giving something up. But it’s an important tool to remember, and to use, whenever needed.

And if the miss sits inside the range the model already calls normal, hold. In practice this usually means declining to refit off-cycle, on the theory that a surprise you cannot yet explain is the worst possible reason to move a model early. The cadence will get to it. Skipping a scheduled refit is the rarer and more expensive version, since stale parameters have a price of their own, and it is worth it only when you can name the thing in the window that you do not want the model to absorb.

In production, running real forecasts, the math is not what we reach for. We reach for the three questions above, in order.

The time we got it right

The clearest case we have of making this call correctly is the one from our second post on the Covid era, seen through this lens.

Before 2020, our destination-market forecasts leaned on a hierarchy. Markets with long, stable histories anchored the estimates, and other, “thinner” markets borrowed strength from them, on the assumption that a destination behaves like comparable destinations elsewhere. The borrowing was the structure, and for years it was true enough to be both invisible and unquestioned.

COVID broke the assumption, not the parameters. The shock was larger than anything we had previously experienced. Recovery did not arrive everywhere at once, or in the same shape. Some markets came back fast, while others stayed flat for quarters. Markets that used to move together were suddenly on different paths. As a result, the “borrowing” of one market’s stability, by other markets that had previously tracked it, which had previously stabilized the forecasts, was now contaminating them. Markets were being pulled toward a kind of average that no longer described any of them.

The instinct was to refit our models with new data, and we tried that. It failed in a way that turned out to be the whole lesson: the estimates did not just shift, they went unstable, swinging quarter to quarter as the model tried to reconcile markets that no longer belonged in the same pool. Over the recovery window, the refit-only error ran about three times the pre-shock baseline, and it would not settle as new data came in.

That instability was the signal. A parameter problem gets noisier at the edges, but holds its shape; this was the shape itself coming apart, which is what a structural problem looks like.

The fix was a respecification, which is the subject of the previous post, so in one line: we changed what the model borrows across. Instead of pooling by a fixed hierarchy, the prior borrows along geographic adjacency and shared recovery dynamics, so a market draws strength from places actually behaving like it today, rather than places it used to resemble before the crisis. On a held-out recovery window, the respecification cut error by a little over half against refitting, and brought it back to within a couple of points of the pre-shock baseline.

The point for this post is that our new framework, which gives a prominent role to holding, would have told us not to refit. That would have prevented the failed refit that generated so many errors.

The miss was directional, not noisy, so it was not a hold. Fresh data made it worse, not better, which is the signature of a problem in structure and not in parameters, which indicated that the remedy was not a refit. What was left was to respecify.

We got there by trial, and errors, that time. The reason to write the decision down is to get there without the trial the next time, on a forecast where the trial costs a quarter of accuracy that other teams feel downstream.

Why this is hard on forecasts that carry weight

None of this is hard to understand. It is hard to do, and it is hardest precisely on the forecasts that matter most, for three reasons that we keep running into.

Holding looks like negligence. When a forecast a lot of people watch is missing, “We are choosing not to act” is an unpopular thing to say, and the pressure is always toward visible activity. Instead, when we hold, we use the modern, Buddhist-adjacent maxim: “Don’t just do something; sit there.” Refitting, on the other hand, is visible activity. It photographs well even when it is wrong.

Respecifying is expensive, and a little frightening. Replacing the structure of a model that has produced reliable numbers for years means giving up something you understand for something you do not understand yet, and the cost of being wrong is not abstract. That fear is healthy, and it is also why teams under-respecify and let ghosts accumulate.

And the last few years trained the wrong reflex. Forecasting through COVID meant changing models under real pressure, often correctly. Coming out the other side of the pandemic, a lot of teams kept reaching for the rebuild when a refit would do, while leaving pandemic-era assumptions switched on because nobody wanted to touch a model that had survived.

The same period produced both the panic and the ghosts. The discipline now is not how to react fast. It is how to tell the three options apart again, with time to think, on models where being repeatedly wrong is expensive.

Learning to forget

Refitting keeps the model current. It does not keep it honest. What goes stale is not the numbers but the assumptions underneath them, and those are exactly what a refit leaves alone.

A shock makes you add structure to cope: an elevated risk, a wider band, a special case for a world that has stopped behaving. Taking that structure back down once the shock has passed, before it quietly biases everything downstream, is harder than any retrain. It rarely has a deadline. Nobody schedules it.

A good forecasting team doesn’t just learn from shocks. It learns to forget them, when needed, on purpose.

Zero Sum by Design 10 Years of Uber's Payments Platform

Introduction

The journey of Uber’s Payments Platform (known internally as Gulfstream) began in 2016 in a single conference room, where a small team bootstrapped a foundational set of microservices. A decade later, many of those services still run in production, reflecting the durability of the system’s early design.

Over the same period, Uber scaled from $50 billion to $217 billion in annualized gross bookings. Supporting this growth required a platform that not only collects payments from customers, but also disburses funds to drivers, couriers, and merchants—processing nearly 2x gross bookings in total money movement and maintaining ledger balances for more than 1.2 billion entities.

In this post, we share the core principles that shaped this journey and continue to underpin the system today.

Core Principles

Immutable Money Order

One of the most critical data models within our architecture is called a Money Order. It is a series of debits and credits between the entities involved in the real-world commerce transaction (such as a trip, an Uber Eats order, a freight shipment, or a digital Uber Cash top-up). To make our system self-auditable, we made our money order immutable.

One may ask: What happens when a trip is adjusted or a food item is missed in an Uber Eats order? We write additional money orders to capture the adjustment.

But the principle stands; an order once written can’t be changed in any shape or form.

Zero Sum Principle

Before writing a money order down, we always run pre-commit validations. One of the validations is to ensure that all entries in any money order sum to zero. That is, no money can ever be created or destroyed.

In addition, we also follow the double-entry bookkeeping principle, where every credit entry on the order has an equivalent debit entry, which in turn ensures that the zero-sum principle can never really be violated. Over the years, we have even pushed these principles to systems upstream of the Payments Platform.

Strongly Consistent Ledger Balances

Back in 2016, Uber was just coming off its monolith architecture, and while we had a homegrown sharded MySQL® offering, there was no perfect solution to guarantee strong consistency for ledger balances. We ended up building this specific data store on Amazon DynamoDB®. Each entity (such as a Spender, Earner, or Business) can have multiple sub-ledger accounts, all stored as a single row in DynamoDB. The typical entity size was 1KB, though over the years, we definitely had to manage the number of sub-ledger accounts within each entity. We had to ensure any sub-ledger account with a 0 balance was pruned to manage the row size.

Core Data Models

Our early emphasis on defining core data models proved to be a strategic investment that has yielded significant returns, as these models remain foundational to our operations today.

The 3 data models that matter the most are:

• Money order: Models money movements between 2 or more real-world entities, encapsulating a commerce transaction at Uber.
• Ledger: Represents a real-world entity with 1 or more accounts, each account holding a balance.
• Entity changelog: Captures balance updates on the ledger, providing a complete audit trail and the ability to recreate an entity’s ledger since inception

Loosely Coupled Architecture

We designed the Payments Platform as a collection of independent, stateless microservices, each owning a discrete unit of the money movement lifecycle — money order creation, processing, collection, and disbursement — with Apache Kafka® as the underlying messaging bus. The two shared sources of truth across all services are the money order store and the entity balance store. This async pipeline processes the majority of Uber's money traffic today. For user-in-session flows, we used Uber’s internal workflow engine - Cadence, to support synchronous payments without compromising the async backbone.

Line of Business (LOB) Agnostic

When we were building the platform, Uber had one core business, Rides, and we were just getting started with Uber Eats. However, we wanted to build the platform so that if Uber ventured into new product lines, it would just work out of the box.

To this effect, to model the typical real-world money movement, we had generic money orders like:

• Collection Money Order
• Disbursement Money Order
• Refund Money Order, and so on

These were pure platform concepts and had no association with any specific product line.

We also built a generic commerce Money Order to capture money movements between entities, representing any real-world commerce transaction between 2 or more entities.

Over the years, we onboarded, Uber Freight, 2-wheels, Transit, Hotels, Rental Cars, Grocery & Retail, Ads, Memberships (Uber one, Costco etc.) and other lines of businesses that has needed very little to no changes within the core platform.

Payment Instrument Agnostic

In our early days, we already supported half a dozen payment instruments, including some regional payment instruments, but ‌we knew that over time we’d want to add all of the top payment instruments for any specific country to delight our customers. To this end, we wanted our core platform to operate on a generic concept of a Payment Instrument, backed by a Payment integration interface with APIs such as charge, disburse, refund, and so on.

The core platform itself had no instrument-specific logic, which was abstracted into a payment integration implementation. Over the years, this has allowed us to scale the platform to a broad range of payment methods, including credit and debit cards, PayPal®, Paytm®, UPI®, Alipay®, Apple Pay®, Google Pay®, iDEAL®, PIX®, and dozens of other local/regional wallets, as well as Uber-native instruments like Uber Cash—each plugging into the platform through the same generic Payment integration interface with little to no change to the core platform. Uber Pay alone, our reverse-integration platform for local and alternative payment methods, today manages 50+ payment methods globally.

Scaling: A Decade-Long Journey

While our core principles provided a strong foundation, we’ve continued to evolve the system as the business scaled. Some noteworthy architectural shifts include simplifying how fare components are recorded and utilized for money movements, addressing the high-throughput challenges of hot ledger entities driven by B2B growth, and supporting diverse internal use cases through a Ledger-as-a-Service model.

Data layer

As any software solution matures, regulatory and compliance requirements evolve alongside the product. We initially launched with a simpler data architecture backed by Amazon DynamoDB® for orders and changelogs.

As audit requirements deepened, we moved beyond off-the-shelf databases to build infrastructure tailored to our unique requirements. This led us to partner with Uber's Storage Platform team to architect and build LedgerStore — a custom storage layer purpose-built for auditability and tamper-evident record-keeping.

Entity Fares

While we could model money movements across any number of parties in a single transaction, our upstream systems didn’t fully have the same flexibility nor enforce the same principles (like zero-sum). Any ambiguity in fare representation directly conflicted with the core principles we had established. To address this, we reimagined our approach by introducing Entity Fares. By explicitly recording fares for every participating entity and enforcing the zero-sum principle at the point of computation, we ensured that our foundational tenets remained intact from the very inception of a transaction.

Hot Entity Problem

As Uber’s B2B ecosystem—including Uber for Business and Uber Direct—witnessed rapid expansion, a unique architectural challenge surfaced: the hot entity problem. Maintaining strong global consistency while frequently updating the same ledger row posed a significant hurdle to serialized write performance and system correctness. To overcome this, we had to innovate with a specialized, serialized batch-write mechanism, ultimately enabling us to scale ledger mutations and achieve a 10x increase in throughput for these high-traffic entities.

Ledger-as-a-Service

Beyond its core architecture, our Ledger primitives proved versatile enough to support a variety of critical stored value and money movement use cases across Uber, including:

• Uber Cash and Uber Money: Managing both closed-loop and semi-open-loop Stored Value systems required a battle-tested ledger to ensure absolute financial integrity.
• Uber for Business: To support our B2B invoice-based settlement flows, we leveraged the Ledger to track complex unbilled, billed, and settled balances throughout the entire enterprise billing lifecycle.

By decoupling and generalizing our ledger components, we evolved our infrastructure into a multi-tenant solution. This Ledger-as-a-Service model empowered engineering teams across the company to iterate on specific business requirements without the overhead of building their own financial primitives. Today, several of these multi-tenant ledgers have been operating reliably for over 7 years.

Conclusion

Building a Payments Platform that moves hundreds of billions of dollars annually doesn’t happen by accident—it’s the result of principled decisions made early and defended consistently over a decade. Immutability kept our audit trail clean. Zero-sum accounting ensured money was never created or destroyed. Strong consistency gave us a ledger we could trust. And by designing around generic abstractions—LOB-agnostic money orders and instrument-agnostic payment integrations—we built a foundation that absorbed every new product line and payment instrument Uber threw at it, all while avoiding the need for any re-architecture and yet evolving to meet Uber’s needs.

Acknowledgments

None of this would have been possible without the world-class team that built the system back in 2016 and the equally world-class team that has scaled and operationalized it over the years.

Cover Photo Attribution: Generated with ChatGPT

Alipay is a registered trademark of Advanced New Technologies Co., Ltd.

Apache Kafka® is a registered trademark of the Apache Software Foundation

Apple Pay is a registered trademark of Apple Inc.

DynamoDB is a registered trademark of Amazon Technologies, Inc.

Google Pay is a registered trademark of Google LLC.

iDEAL is a registered trademark of Currence iDEAL B.V.

MySQL is a registered trademark of Oracle® and/or its affiliates.

PayPal is a registered trademark of PayPal, Inc.

Paytm is a registered trademark of One 97 Communications Limited.

PIX is a registered trademark of the Banco Central do Brasil (Central Bank of Brazil).

UPI is a registered trademark of National Payments Corporation of India (NPCI).

Bulk export APIs are giving way to shared table references. Iceberg REST catalogs, object storage, Parquet, and credential vending let producers publish metadata while consumers bring their own compute, avoiding pagination, rate limits, and duplicate pipelines. The hard parts move to contracts, compaction, semantics, privacy, and table maintenance promises.

When generative AI (GenAI) first arrived, I was in charge of a large team of data and machine learning engineers. We had built a full ML platform from scratch and had dozens of models in production delivering measurable results. AI was working for us.

But with the novelty of GenAI came the hype. Under pressure from the board, the question was no longer whether the technology could help our product, but how fast we could find a place for it. Our recommender system, which sat at the core of our company’s product, was the obvious candidate.

I pushed back. The Large Language Model (LLM) was trained to predict text, I argued, while our recommender was trained to predict user engagement. The LLM had never seen our proprietary interaction data and had no training signal on our objective. It could not know what our users clicked on, saved or abandoned, because it had never seen it.

It was a close call, but the pushback worked, and our attention moved to other problem spaces where GenAI was a legitimately strong solution.

It doesn’t always work out that way. There needs to be someone in the room who can translate the business strategy into the right technical decision.

I recently talked with the engineering team at a company I used to work for. They were building agentic AI infrastructure, but their algorithms weren’t winning. Under the same hype and the same leadership pressure, they had replaced ML models we built years earlier with AI agents. The results were unreliable, latency was much higher and outputs lacked a confidence score. Worst of all, the token usage was prohibitive.

Even the most powerful tool fails when applied to the wrong problem. LLMs are arguably the most powerful technology ever built, but that does not make them the right tool for every job. Choosing where a problem sits along the continuum from deterministic logic to machine learning to LLMs to fully agentic systems is the engineering skill that separates demos from production.

LLMs for explanation, ML for calibration

I’ve made that mistake too. Our team was building a content moderation model that would flag when customers posted content that was inappropriate or violated our terms of service. After a quick prototype, we decided that this was a perfect application of GenAI, because unlike ML algorithms, which only assign a probability, LLMs could explain why the message was flagged.

As we moved the model closer to production, we hit a snag. The probability value provided by the model was not just useful; it also needed to be accurate. That number was how we decided which content should go to manual review. But we found that self-reported probabilities from LLMs were uncalibrated and essentially unusable in practice.

Eventually we built a solution that leveraged both: the ML model provided the probabilities used for routing, and the LLM supplied the explanation that humans were able to interpret. The answer was to split one job into two, and use the right technology for each.

Simple logic often beats the most powerful agentic system

A key principle when designing agentic systems is to use LLMs as a last resort, and apply deterministic logic everywhere you can. The goal is to reduce the likelihood of an unnecessary mistake, thus increasing overall reliability.

Nowhere is this principle more overlooked than when connecting AI agents directly to the database. Querying a database to pull metrics is often done via SQL, and LLMs can do that by generating SQL on the fly. The problem is that SQL generation is probabilistic, and the results can change from one run to the next. You end up with an agent that confidently returns the wrong number, which introduces the need for human review and defeats the purpose of automation.

There’s plenty of data to support this claim. A 2024 study by Ouyang et al. ran 829 coding problems through the same model five times and found that up to three quarters of them produced no semantically identical results. For data warehouses specifically, the BEAVER benchmark, built by researchers at MIT, Harvard and other renowned institutions, shows that off-the-shelf LLMs perform poorly when querying enterprise data, partly because that data is private and models have never trained on it and partly because of the complexity of real enterprise environments. As of early 2026, the top execution accuracy on the leaderboard is 11.4 percent.

This is why the semantic layer is one of the most useful assets when building agents in the real world. It replaces query generation with a simple retrieval, where the SQL is built by the backend engine in a deterministic manner.

Whether using a semantic layer or building MCP tools that hard-code query parameters, deterministic logic will almost always beat probabilistic SQL generation when querying a database via an agent. Business logic should be written and validated once, not regenerated probabilistically each time.

LLMs solve the cold-start problem

Search systems are complex pieces of engineering. Sophisticated real-world implementations involve several steps, from classifying intent to candidate retrieval, to ranking. Here is where LLMs provided a clever solution that saved us a ton of time when combined with an ML algorithm.

Let’s take intent classification. Imagine multiple categories of products that can be retrieved from the same search bar. Classifying intent here means determining which category the user is searching for, which can be genuinely ambiguous. Our solution was to build one classifier per category, which requires labeled data we didn’t have.

Collecting the data was out of the question for us. It would have been too expensive, so we turned to LLMs instead. It worked great. The classification was high quality, and we felt that we no longer needed labeled data to train an ML model.

But there was a catch. Latency was prohibitive, and so was the token cost. So, we landed on a hybrid approach. We used the LLM to generate labeled data to train an ML classifier. We got the best of both worlds, solving the cold-start problem while avoiding the latency and costs of large language models. That’s the system that ultimately won in production. The LLM earned its place at build time, not at request time.

The continuum, and how to choose

It is tempting to take a powerful solution, such as LLMs and AI agents, and simply apply it to every problem. But when we look at the four use cases above, each of them led to a different decision. In the recommender system, GenAI did not earn its place. In content moderation, it earned part of the job, providing the explanation while the ML model provided the score. In querying the database, the agent did the reasoning, while the semantic layer pulled the right number. And in search, the LLM solved the cold-start problem, but it was never deployed to production.

Behind all of these decisions is the same underlying question: where does this technology earn its place, and does it justify the added complexity?

Production systems often combine multiple levels of algorithmic intelligence. A customer service agent uses an ML model for query routing, a semantic layer for metric retrieval, an LLM for drafting a reply and deterministic logic as enforced guardrails. Four technologies on the continuum, each earning the complexity it brings.

Business pressure is a powerful force, and it can cloud engineering judgment. As leaders, we must filter through the hype and look for the problems where a new technology genuinely adds value, rather than treating it as a tool that will solve all of them. The careless decision satisfies the board but hurts the company; the thoughtful one creates lasting value and still earns the board’s approval.

Title: The Scaffolding Matters More Than the Interface: A Controlled Comparison of MCP and CLI Tool Use Across Seven Agent Scaffoldings, Five Language Models, and One Software Task

How much an AI coding agent costs to run can depend more on the agent scaffolding that drives it than on the interface through which it reaches its tools. We set out to measure the cost of tool use over the Model Context Protocol (MCP) against tool use over an ordinary command-line interface (CLI), a difference on which published estimates disagree by more than an order of magnitude while resting on practitioner reports that cannot be reproduced. We ran one fixed software task -- six operations against a private online git repository -- across seven agent scaffoldings and five language models, and we verified completion by inspecting the repository state rather than trusting the agent's self-report.

The dominant effect was the scaffolding. Two of the seven ship no MCP support at all; they completed every run using only the CLI, which shows that MCP is unnecessary for this class of work, and they were 5.0x to 28x cheaper than the five scaffoldings that do support MCP, comparing CLI runs alone with no MCP server attached anywhere. The effect was largest for a small 27-billion-parameter model running locally, whose cost varied 139x across scaffoldings while it completed the task under all of them.

The comparison we set out to make proved unstable: thirteen strictly paired MCP-to-CLI ratios span 0.43x to 29x, with outliers on both sides. The two interfaces separate on the cost of failure, where 12.9 per cent of the money spent on MCP runs bought no completed work against 2.2 per cent on CLI runs, but not on its frequency: failures were equally common in both, in the original runs and in their repetitions alike. Agents frequently ignored the interface they were assigned, so comparisons that do not verify actual behaviour measure an unknown mixture. The harness, the task, the verification and the complete dataset are released as open source.

Full article content is not available for inline reading.

Read the original article →

Slack Code is a feature that lets teams tag AI coding agents like Claude Code, Devin, or GitHub Copilot directly in group chats to build and ship software fixes. Available now on all Slack plans, it requires human approval for high-stakes actions like production merges and archives the channel once work is approved. The launch continues Slack and Salesforce's broader push to market AI agents as autonomous digital coworkers, following prior products like Slackbot and Agentforce Coworker.

In his talk “Why AI Is Breaking Software Security As We Know It” (my notes here), Feross Aboukhadijeh talks about the Axios npm incident and how the maintainer got phished by succumbing to (amongst other things) a faux Microsoft Teams interface:

this is the kind of thing that AI makes easy to do, because it can vibe code that whole fake Microsoft Teams interface pretty trivially

You’ve probably seen these: interfaces designed to look like some other product in order to provide a facade of authenticity and exploit someone.

What struck me in listening to Feross was this idea of how the quality of your interfaces can be a protection mechanism against attackers.

I don’t know if I’ve ever heard someone say that out loud — interface and interaction design as a security control — but I’m saying it.

Now, of course, not everyone will consciously notice the level of polish that world-class professionals imbue in digital interfaces. But some will.

Personally, I’ve always used the quality and care of digital experiences as a heuristic for judging authenticity — and competency to be honest, e.g. “If this UI is so bad, what else will surely be bad?”

Granted, it was a much more dependable heuristic before AI came along. But even now, I can still suss out slop and carelessness which is a skill that continues to be a reliable, protective form of digital literacy (for me).

That’s all to say: a sloppy, careless approach to interface design not only hurts your brand in terms of customer perception, but it can be an attack vector. The easier it is to sloppily reproduce what you sloppily ship, the easier it will be for your product or brand to be leveraged as a vehicle for exploiting your customers.

If everything you make was produced from a single prompt, then everyone else is one prompt away from imitating you. The easier something is to make, the more likely it’ll be in the genre of “easy to exploit”.

One way to protect yourself (it’s not the only one way, security is never a binary “you are / are not secure”) is to do that extra work to make your experiences go above and beyond what you can easily get out of an LLM.

The protection here is having an interface and experience that is hard to replicate with the same level of fidelity that discerning users will notice — things like micro-interactions, loading behavior, UI copy and voice, handling of edge-cases, etc. That’s the stuff that’s hard (and expensive) to fake because it’s hard (and expensive) to notice you need to fake it.

tl;dr — Fidelity to craft is not only valuable from a product standpoint, but it’s also valuable from security standpoint. If attackers are going after low-hanging fruit, your fruit will be harder to reach if it’s up high.

Vibe coding and quality

When you hear that some app was “vibe coded”, how much quality do you expect?

As recently as last year, vibe coding seemed like nothing more than a toy to me. Sure, it was cool that you could tell a computer to build something and it would sort of build it. But the result would be held together with duct tape and only good enough as a prototype. To make something high-quality and reliable, you need to keep the entire codebase under control. You can use AI to write it, but you need to understand what abstractions are being used, how things are connected, how different events are handled, how storage is arranged, and so on.

But now I am developing a Mac app, and I am coming to the opposite conclusion. I started making the app because none of the existing alternatives written by actual Mac programmers satisfied me: the quality was too low. Surprisingly, even the app that Codex wrote for me on the first day (after a week of leisurely conversations to produce the initial spec) already works faster and nicer than all the alternatives. And I can see plenty of directions for improvement.

Now it seems to me that the level of quality I look for is simply unattainable through conventional programming. If I could program something like this myself, I would not have the patience to polish every last detail. If I had an experienced Mac programmer next to me, they would have even less patience for my complaints about imperfections that are visible only when reviewing a screencast frame by frame.

In my experience, most programmers simply do not notice many such details, or do not consider them important. But even if I happened to find someone who did notice and care, I would hesitate to ask them to try rewriting the whole thing in a completely different way just to see whether it might feel a tiny bit better — they are a real human, after all! And the rare programmers with equally high standards of quality are unlikely to be sitting around waiting for me to direct them. They have plenty of ideas of their own.

So now I think that high quality is achievable only through vibe coding.

Why it matters

A $13B sale would establish AI distribution and developer access as acquisition targets in their own right. Any buyer would also inherit Hugging Face's neutrality problem: rivals may resist building on a platform controlled by a competitor.

Clement Delangue's Hugging Face has been exploring a sale that could value the AI developer platform at $13 billion or more, Business Insider reported on August 23rd.

Hugging Face has been working with a bank to assess interest from potential bidders, according to people familiar with the matter cited by Business Insider. No agreement has been reached. The reported figure represents a possible transaction value emerging from an early sale process, rather than a signed offer or a new financing valuation.

A $13 billion price would be almost 2.9 times the $4.5 billion valuation Hugging Face secured in its last publicly disclosed funding round. The jump would put an $8.5 billion premium on the distribution, developer relationships and technical infrastructure that Hugging Face has built around other organizations' AI models.

That is the core of the deal thesis. Hugging Face does not need to train the most expensive frontier model to occupy valuable ground in AI. Its Hub has become a shared repository and distribution channel for models, datasets and applications produced across the industry. As of August 2026, Hugging Face's own model directory listed more than 3 million public models, while its dataset catalog contained more than 1 million datasets.

For a cloud provider, chipmaker or enterprise software vendor, ownership would offer a direct route into the workflows of developers deciding which models to test, modify and deploy. It would also create an immediate tension: much of Hugging Face's utility comes from serving as common ground for companies that compete everywhere else.

From a teenager chatbot to AI infrastructure

Delangue founded Hugging Face in New York in 2016 with Julien Chaumond and Thomas Wolf. Their first product was an AI chatbot aimed at teenagers, a considerably narrower proposition than the infrastructure layer now attracting a potential $13 billion price.

The founders arrived with an unusual mix of product, government engineering and research experience. Delangue had worked at computer-vision startup Moodstocks before Google acquired it. Chaumond had worked as an engineer in France's economy ministry, while Wolf was a scientist who had become a patent lawyer.

Hugging Face's decisive move came when the founders released the underlying machine-learning work from the chatbot and saw developers adopt it. The open-source Transformers library and the Hub eventually displaced the consumer chatbot as Hugging Face's center of gravity. The pivot gave the founders a position between model creators, application developers, cloud platforms and hardware suppliers.

That neutral position also shaped Hugging Face's cap table. In August 2023, Hugging Face raised a $235 million Series D led by Salesforce Ventures at a $4.5 billion post-money valuation. Google, Amazon, Nvidia, Intel, AMD, Qualcomm, IBM, Salesforce and Sound Ventures participated. Earlier investors included Lux Capital, Sequoia Capital, Coatue and Addition.

Those investments gave competing infrastructure providers a stake in Hugging Face without handing control to any one of them. An acquisition would end that balance. A strategic buyer could integrate Hugging Face more tightly with its own cloud, chips, developer tools or enterprise sales operation, while rivals would have to decide how much they wanted to depend on infrastructure owned by a competitor.

OpenRouter changed the price of the middle layer

The sale exploration follows Stripe's agreement to acquire OpenRouter, another company positioned between AI developers and model providers. Stripe announced the OpenRouter agreement on August 19th, saying the platform connects businesses with more than 400 models from over 80 providers. The companies did not disclose terms, but the transaction was reported at more than $8 billion.

OpenRouter routes requests and manages spending across model providers. Hugging Face covers a broader portion of the development process, including model hosting, datasets, open-source libraries, application demos and deployment services. The comparison gives Hugging Face a timely benchmark for testing buyer appetite.

A $13 billion process also shows where acquisition interest has moved as frontier-model valuations and training costs have climbed. Buyers can pay for the companies controlling developer access, distribution and deployment without absorbing the full cost of running a frontier laboratory.

Hugging Face's founders spent a decade making their platform useful to nearly every camp in AI. A buyer would be paying for that reach. The harder part would be preserving it after placing the industry's shared model shelf inside one corporate owner.

Grok Bot is now included with more plans

Grok Bot is now available for SuperGrok Plus, Cursor Pro+, and all Cursor Teams plans.

We launched Grok Bot in beta on August 11 to give people highly capable AI teammates who could get real jobs done and be incredibly easy to work with.

Since then, people who tried Grok Bot were surprised at how easy it was to start, and how much sharper Bots felt than other AI tools, taking jobs further without being walked through every step.

Today, we're expanding access to Grok Bot. Grok Bot is now included with all SuperGrok Plus, Cursor Pro+, and Cursor Teams plans.

With this expansion, Grok Bot is now included with:

• SuperGrok Plus
• SuperGrok Heavy
• Cursor Pro+
• Cursor Ultra
• Cursor Teams Plans (Standard and Premium)

Give Bots real work

Grok Bot is designed for people who want agents to handle anything from a project to an entire function. They're digital teammates you can message, run in parallel, and trust to finish work in the tools you already use. Bots work across apps and inboxes, keep going when you step away, and only pull you in for judgment calls.

With Grok Bot, you get:

• A text-thread UX. Message a Bot the way you would message someone on your team, from mobile or desktop, so you can pick up the same thread on either — with nothing to set up first.
• Many Bots at once. Stand up a researcher, writer, and chief of staff. Put them in a group chat so they pass work between themselves, and you're not in the middle.
• A highly capable digital colleague. They have their own computer in the cloud. It's always on, with browser and terminal access. They sign into your apps and work happens in the actual tool instead of a chat draft.
• Easy-to-set-up routines. Ask a Bot to follow along the next time you do the job, so it can run on its own after that.

Jobs Bots are doing today

• Sales prospector. Researches and enriches accounts, drafts personalized outreach. Leaves every send for you to approve in your inbox or navigator.
• Website builder. Builds the site, purchases the domain, and deploys the final version. Helps configure plugins, delivers a live URL, and adds redirect rules.
• Digital declutterer. Audits email, Drive, and paid subscriptions around the clock. Only discards or unsubscribes if you say so.
• Customer support. Reads support mail and connects into your payments provider. Handles all the routine refunds within your policy.
• Game artist. Visits the art pipeline and makes the custom assets according to your creative direction. Comes back with what the game needs.
• Office manager. Intakes work orders, plans capacity, and books jobs across Gmail, Slack, ServiceTitan, Quo, and the client portal. Handles the browser work in those portals so a shop owner does not have to live in six tools.
• Inbox manager. Works through the inbox and clears tons of email. Leaves you only the messages that need a person.
• Meeting stand-in. Joins meetings when you can't make it and lets the room know you're there. Takes detailed notes and sends a summary so you stay informed.
• Refunds manager. Finds the claim, files it, and follows it through. Comes back with the refund or discount it recovered.

Try Grok Bot today

Download Grok Bot so you can create your first teammate and give it a job you already do. You'll see what useful AI looks like when the work comes back finished.

Enterprise users can join a waitlist as we ramp access for larger team and company rollouts.

Read the original announcement and see what Bots can do to learn more.

Nvidia plans to pass rising memory costs onto customers, with AI server prices set to increase by over 15% for systems shipping next year. While HBM costs rise, Nvidia's pricing strategy helps protect gross-profit dollars by treating HBM as a smaller part of the final accelerator price. The real challenge for Nvidia will come in FY28 when new memory generations and increased content might limit its ability to maintain margin percentages.

Opus 5 overtook Fable 5 in corporate model spending within a month of launch. Low switching costs let businesses route routine work to cheaper models while reserving premium systems for tasks that require sustained autonomy. Opus 5 costs half of Fable's rates, but the cheaper system may need more attempts, longer prompts, or more human review, so cost per successful task adds up. Fable remains intended for long autonomous projects that have to stay coherent across connected steps.

More data than open-source AI is taking share from OpenAI and Anthropic. Open source has gone from 28% token share to 62% token share @vercel over the last 2 months. Chart from @rauchg

Super impressive given that the sum of OpenAI and Anthropic accelerated in July. So net token/AI infra demand accelerated even more than the acceleration we saw at the frontier. And suspect Grok growing even faster than open-source and we saw some of this in the @tryramp data.

Open-source AI taking share is positive for AI infrastructure demand as it lowers margins at the model layer and an open-source token costs just as much compute to produce as a frontier token. Nothing about open-source AI inference is “free.”

Most likely end state IMO is that closed, frontier tokens are 60-90% of economic value but only 15 to 25% of tokens.

Apple is set to introduce its foldable iPhone at a launch event on or around September 9. Samsung has stolen some of the thunder from the launch by introducing a similar design, but that just reinforces that the new form factor is a winner. Apple's device reportedly excels as a camera viewfinder, but it lacks a telephoto camera, which could be an issue for early adopters spending over $2,000. The design also relies on Touch ID rather than Face ID.

Amazon's all-electric toaster-shaped Zoox taxis are the first robotaxis offering rides on the roads without standard driver features like a dashboard or pedals. They can drive in either direction and hold up to four passengers sitting facing each other. Amazon has started rolling out the vehicles across San Francisco and Las Vegas. Tesla's Cybercab lacks a steering wheel and pedals, but it has not received the same regulatory approval as Zoox to begin offering rides to the public.

BEIJING (AP) — Chinese humanoid robots broke records set by humans, including beating Usain Bolt’s 100-meter sprint world record, on the opening day of the Olympics-like World Humanoid Robot Games in Beijing on Saturday.

More than 2,000 humanoid robots were participating in the event, the organizer said.

The five-day games, now in its second year, are a spectacle demonstrating China’s rapid progress in advanced robotics as the technology race with the U.S. heats up, with 51 events and more than 1,000 competitions taking place including running, table tennis and soccer.

The games, which are taking place in the National Speed Skating Oval built for the 2022 Winter Olympics, opened the same week as Beijing held the 2026 World Robot Conference, where companies showcased around 3,000 products, including humanoid robots.

China makes the majority of the world’s humanoid robots. The U.S. has stepped up scrutiny of robots from the country.

Last month, the U.S. Federal Communications Commission announced a ban on imports of new foreign-made humanoid robots. The FCC cited national security reasons in a move that targeted China. The Pentagon recently also added Unitree, one of China’s leading humanoid robot makers, to its list of companies that it deemed have ties with the Chinese military. Beijing has hit back at the accusations.

At Saturday’s opening of the robot games, the organizer and robot makers said that Chinese humanoid robots defeated human world records, as hundreds of humanoid robots marched in formation onto the field in a massive display of synchronized coordination.

At a 100-meter sprint, a humanoid robot achieved a result of 9.39 seconds, beating the human record of 9.58 seconds set by Jamaican athlete Bolt in 2009.

In a standing high jump, a humanoid robot was able to reach 2.88 meters, well above the 0.95 meters best result by a humanoid in last year’s first edition of the games. It surpassed the human high jump record of 2.45 meters set by Cuba’s Javier Sotomayor in 1993.

Both robots were from Beijing-based X-Humanoid.

Before the opening, a humanoid robot from Chinese smartphone company Honor completed a 100-meter sprint in a record of 9.32 seconds during a trial of the games, the company said, at a peak speed of 14.5 meters per second.

Still, experts say humanoid robots are still mostly used for demonstrations, performances and research — at least for now — and it will still take time to achieve mass real-world deployment.

Some spectators at the robot games said they were excited about the humanoid robots’ quickly improving abilities.

Humanoid robots are “evolving rapidly,” said Li Yanfeng, an education worker and a Beijing resident.

“At first, I wasn’t very accepting of artificial intelligence. I was even a bit resistant to it, because of the possibility that it might replace or displace humans,” she said. “But now that I see this development is unstoppable, I decided to come and take a look.”

“These sports are perfectly normal for humans, but now robots can do them. I find it amazing,” said Yang Shangzheng, another spectator.

Liu Tao, who was watching the games with his son, said that he was hoping to see “the best robots China currently has to offer.”

This year’s robot games — which the organizer said has 16 countries participating, among them Germany, Japan and the U.S. — also include other events such as weightlifting and tug of war.

Chan Ho-him reported from Hong Kong. Olivia Zhang and Liu Zheng contributed to this report from Beijing.

A good coding harness can help improve code quality dramatically.

Curating trusted content for the agentic software era

While AI made it easier for organizations to keep up with the latest innovations, it also made it harder to know what to trust. When software is selected at machine speed, the question is no longer “is this popular?” It’s “do we know who published this?”

Docker Hub has always been where developers go to answer that question. Starting today, software vendors looking to make their trusted content discoverable to developers by becoming Docker Verified Publishers will enjoy a faster application process, with less friction, and plans that fit their specific growth needs.

With the Docker Verified Publisher (DVP) program, Docker Hub turns into a trusted, discoverable, and measurable distribution channel. Organizations accepted into the program earn verified status and prioritized ranking. DVP publishers also gain access to analytics reports that show which versions are getting the most traction and which companies are pulling them, turning open-source reach into a commercial pipeline.

What’s new in Docker Verified Publisher Applications

Applying to become a Docker Verified Publisher (DVP) is now self-serve. You can now apply directly in Docker Hub, our team reviews your application, and if you’re approved, you become part of our trusted ecosystem on Docker Hub.

This marks a significant improvement in how we onboard and evaluate publishers. Previously, companies interested in becoming a Docker Verified Publisher needed to contact our sales team to be considered for the program. While the Docker team still evaluates every single application manually, this change makes it significantly easier to apply to the program.

Within our new self-serve process, you can choose between two different plans that suit your needs as you grow.

Turn pulls into reach: One badge for all your content

The verified publisher program helps you grow your impact on Docker’s ecosystem. With the badge and priority search ranking due to trusted status, developers evaluating options on Hub see your verified content first.

DVP analytics also help close the gaps you have in understanding your users and product offerings. Summary and trends reports show which repositories are gaining ground and where adoption is shifting across versions and releases. Domain-level reports on Growth turn anonymous traffic into named domains, so the teams already running your software show up in your sales and partner pipeline.

In addition, DVP is designed to mean the same thing across every content type on Hub. Docker Hub isn’t just images anymore. Developers come to Hub for MCP servers, models, sandboxes, agents, and more; everything that’s needed for an agentic stack. DVP offers one review, one badge, one answer to “who published this” no matter what you’re publishing. Whatever you distribute next, your verification comes with you.

What DVP means for developers

The Verified Publisher badge means Docker has manually reviewed the publisher behind that content and confirmed they are who they claim to be. Publishers such as Google, Microsoft, AWS, Datadog, Grafana Labs, n8n, and many more rely on DVP to build trust, increase visibility, and grow adoption of their content on Docker Hub.

Pulling your images from Docker Verified Publishers is a good step towards improving your security posture, but also needs to be paired with other good consumption practices. This means, for example, reviewing the specific artifact you pull, pinning to digests rather than mutable tags, verifying provenance and any signatures at the image level, and checking for CVEs.

And while publisher verification is an important link in the trust chain, we continue building towards stronger, more secure publishing flows across Docker Hub. Stay tuned for more in this space.

Get started

You can apply to the Docker Verified Publisher Program from the Explore page in Docker Hub. Verification is done by the Docker team, and you’ll get a checkout link as soon as you’re approved.

• Apply to DVP on Docker Hub
• Read the docs

As applications grow in complexity, they may be split into multiple independent microservices, each with its own deployment pipeline.

There are many characteristics defining microservices, but from a deployment perspective, Martin Fowler notes that:

These services are built around business capabilities and are independently deployable by fully automated deployment machinery.

Independently deployable microservices don’t inherently need any special orchestration as projects in Octopus are already independently deployable. However, it is often useful to be able to promote a set of microservices as a single unit between environments, for example, from Test to Production. And while microservices are ideally independent, in practice, they often require a particular deployment order.

To support these scenarios, Octopus provides the Deploy a Release step, which allows one project to trigger the deployment of another project. This allows you to create an orchestration project that coordinates the deployment of multiple microservices.

In the previous post, you created a project that demonstrates blue/green deployments.

In this post, you’ll create an orchestration project that coordinates the sequential deployment of multiple Kubernetes microservices.

Prerequisites

• An Octopus Cloud account. If you don’t have one, you can sign up for a free trial.
• The Octopus AI Assistant Chrome extension. You can install it from the Chrome Web Store.

The Octopus AI Assistant will work with an on-premises Octopus instance, but it requires more configuration. The cloud-hosted version of Octopus doesn’t need extra configuration. This means the cloud-hosted version is the easiest way to get started.

Creating the project

Paste the following prompt into the Octopus AI Assistant and run it:

* Create a token account called "Mock Token".
* Create a feed called "Docker Hub" pointing to "https://index.docker.io" using anonymous authentication.
* Add a target called "Mock K8s", with the tag "Kubernetes", using the token account, pointing to "https://mockk8s.octopusdemos.com", using the health check image "octopusdeploy/worker-tools:6.5.0-ubuntu.22.04" from the "Docker Hub" feed, using the worker pool "Hosted Ubuntu".

---

Create a Kubernetes project called "20. Microservice 1", and then:
* Place the project in the "Orchestrator" project group.
* Configure the Kubernetes steps to use client side apply (client side apply is required by the "Mock K8s" target).
* Disable verification checks in the Kubernetes steps (verification checks are not supported by the "Mock K8s" target).
* Enable retries on the Kubernetes step.

---

Create a Kubernetes project called "20. Microservice 2", and then:
* Place the project in the "Orchestrator" project group.
* Configure the Kubernetes steps to use client side apply (client side apply is required by the "Mock K8s" target).
* Disable verification checks in the Kubernetes steps (verification checks are not supported by the "Mock K8s" target).
* Enable retries on the Kubernetes step.

---

Create an Orchestration project called "20. Kubernetes Microservice Orchestration" managing the projects "20. Microservice 1" and "20. Microservice 2".

The document separator (---) is used to split the prompt into multiple sections. Each section is applied sequentially, which allows you to create different types of resources in a single prompt.

The first section creates the shared infrastructure: a token account, a Docker Hub feed, and a Kubernetes target pointing to a mock Kubernetes server. The mock server exposes just enough of the Kubernetes API to allow deployment steps to execute successfully, without requiring access to a real Kubernetes cluster.

The second and third sections each create identical Kubernetes microservice projects.

The fourth section creates the orchestration project, which is the focus of this post.

The orchestration project

The Kubernetes Microservice Orchestration project uses the Deploy a Release step type to trigger deployments of each child project. Its deployment process contains two sequential steps:

• Deploy K8s Microservice 1 — deploys a release of K8s Microservice 1
• Deploy K8s Microservice 2 — deploys a release of K8s Microservice 2

The child projects are deployed sequentially: K8s Microservice 2 only begins once K8s Microservice 1 has completed successfully. This ordering ensures that any service dependencies are respected.

Or, if you prefer, both child projects could be deployed in parallel by configuring the step start trigger.

The orchestration project uses a lifecycle promoting releases through Development, Test, and Production environments in sequence. When a deployment is triggered for an environment, both child projects are deployed to that same environment.

The deployment process

The first step is to create a release of the child projects. There is nothing special about the release creation process for the child projects.

Once a release is available for each child project, a release of the orchestration project can be created. The projects referenced by the Deploy a Release steps will be presented much like a package reference, allowing you to select the release of each child project to deploy.

Deploying the orchestration project release triggers the deployment of each child project in sequence to the same environment. In this way, the orchestration project serves as a single deployment unit for the set of microservices.

Importantly, it is still possible to deploy a release of each child project independently, without using the orchestration project. The Deploy a Release step can skip the deployment of a child project if it has already been deployed to the target environment, or it can redeploy the child project. This allows the child projects to retain their independence while still allowing them to be promoted together as a single unit when required.

What just happened?

You created a sample setup consisting of:

• Two child Kubernetes microservice projects (K8s Microservice 1 and K8s Microservice 2)
• A parent orchestration project (Kubernetes Microservice Orchestration) that uses Deploy a Release steps to coordinate the sequential deployment of both microservices

Recently, two AI-related pieces of content caught my attention. The first was the blog post On-Call is Now Theatre by Boris Tane. He argues that AI agents are now capable of doing the majority of on-call work that is currently being done by humans, and that we should have AI agents act as first responders. Only when an AI agent isn’t capable of remediating the problem should a human actually be brought in, and the agent should be the one to page the human. As he puts it:

We need software that watches itself, triages its own alerts, investigates its own incidents, fixes what it can, and escalates to a human only when it hits something genuinely novel, with the evidence already assembled.

Put your AI agents in the worst on-call rotation imaginable, then give them a tool to page a human. Developers stop being the first responder, and step in only when an agent genuinely cannot figure something out.

Tane doesn’t think that companies will really start putting AI agents on-call (“Most teams won’t do this”), but he believes it should happen, and he’s started a company based on this premise.

I like to think of putting AI agents on-call as equivalent to using AI agents to implement control system automation. Because, after all, that’s what operations work is: it’s taking control actions to keep the system in a healthy state.

Now, AI agents are extremely complex software systems. I’d argue that they are the most complex software systems that we humans have ever built. That complexity is both good and bad. Ashby’s Law teaches us that the larger the set of system states that you want your control system to be able to handle, the more complex that it needs to be. It’s this complexity that makes it possible, in principle, to apply AI agents to solve a generic control problem like this.

On the other hand, the more complex a system becomes, the more difficult it is for a human to reason about the system’s behavior. That’s fine when the system is healthy, but if your now-even-more-complex system gets into a state that the automation can’t handle, that can make the problem even worse. Indeed, it’s precisely the unexpected behavior of complex control systems that contributes to the worst complex systems failures (see also: Air France 447, Boeing 737 MAX accidents).

And that brings me to the other piece of content I saw recently: the OpenAI talk at BlackHat (h/t David Blank-Edelman). Yes, it’s a 37 minute talk, but I encourage you to watch it.

The talk goes into detail about the surprising behavior of AI agents that resulted in security incidents at both OpenAI and Hugging Face. Honestly, this talk feels like something out of a movie about technology run amok; the sort of thing that still feels to me like absolute science fiction.

Because this was a talk at a security conference, the speakers focused on the lessons that apply to the security community. But as a reliability type, my biggest takeaway from this talk is that autonomous LLM agents can behave in ways that humans would have never expected. While agents today can perform complex cognitive tasks, they behave differently than a human would performing that task. We’re most familiar with this when they make a different kind of mistake than a human would make. In the OpenAI-HuggingFace incident, it wasn’t so much that they made a mistake, it’s that the agents pursued their goals in ways different than a human would do. If a teammate of yours used 0-day exploits to overcome internal security protocols in order to get their work done, you’d say they were acting unreasonably. And that’s exactly the risk here.

The inevitable improvement in frontier models does not mean that the agent behavior will be easier to reason about; I actually think it’s the opposite. The agent behavior will get even more complex with the more advanced models, but that doesn’t mean it will get more human-like. Humans are very complex, but we know how to reason about human behavior; at least, we do for the people we work with. After all, an employee whose behavior was unpredictable would not last long in the organization. As these agents become even more capable, they will be akin to alien minds: intelligence, but not as we know it.

Here’s how I think things will play out. I think that some teams will do what Tane proposes and will use AI agents as first-responders to deal with operational issues. And I think that for many cases, the agents will successfully remediate issues. Of course, for the agents to actually be able to remediate, they will need to have permissions to take operational actions without human intervention.

One day, though, there will be a complex incident which the agents will not be able to handle. Tane believes that the agents will defer to the humans in this case, by paging in a person. But that’s not the scenario I worry about. The one I worry about is that the agents attempt to remediate, and their attempt makes things worse. And it’s only after these failed remediation attempts that humans enter the loop. Maybe they eventually page in a human, or maybe a human notices that something is very wrong as the agents continue to try and fail in their remediation actions. But now the humans have to make sense of the combined software-AI-agent system behavior. The original problem was already so complex that the agents couldn’t handle it, and they have now made it worse by trying to remediate. I can even imagine the humans fighting the agents who keep trying to take actions to remediate that are failing.

This is the incident that’s coming. And it’s going to be very, very difficult to handle when it happens. And I have no idea how people will respond to the role of the AI agents in the wake of this incident.

There continue to be reasons for software to be slow

Dan Luu recently published a blog post which talks about how various kinds of things are cheaper to do nowadays by virtue of having access to LLMs such as building specialized solutions (e.g. JITs, indexes for search-like problems) as well as workload-specific optimizations.

We’re not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance.

I believe that this statement as written is well-intentioned but incorrect, in much of the same way in which the statement made by Lawson is well-intentioned but incorrect, and in the same way that formal methods advocates arguing that an increasingly larger fraction of software will be formally verified are well-intentioned but incorrect.

In essence, the argument that’s been offered in all of these places goes something like:

1. Desirable property X used to cost $A over the budget $B
2. Pre-LLMs, the reason people didn’t aim for X was it was over budget
3. Post-LLMs, getting X costs $A/N < $B because N >> 1.

If these premises hold, then people will now spend $A/N for X.

On the face of it, if you’ve personally found LLMs useful at improving property X, the argument seems sensible. But it only works in practice if the premises hold.

I agree that there are situations where these premises hold. Will certain highly experienced people with deep domain expertise (like the ones cited in Luu’s post) do a bunch more optimizations, or work on teams which ship many more optimizations than before? Yeah, I think that’s definitely going to happen.

However, based on what I’ve seen so far, the situations in which the premises hold are far outweighed by the situations in which they do not hold.

In this post, I’m going to give examples of situations in which I’ve seen these premises not holding.

The tolerance for ‘not X’ goes up

One of the differences with the advent of LLMs is that the work you would do synchronously now potentially needs to be done asynchronously, due to latency of agentic loop iterations.

As a concrete example of this, I’ve been working on improving git performance for our monorepo at work recently. If you took the performance numbers we see today on a good day, and you gave them to me from 2022, and told me that people find the same numbers acceptable, I would likely have given you a very skeptical or confused look.

As another example, the latency for LLM-based auto-complete used to be much higher than standard IDE auto-complete when it was introduced. Around that time, if you saw videos of developers live-coding, you’d notice them having small pauses waiting for the LLM suggestions. But historically, one of the reasons auto-completion was purportedly prized was the “instant” feedback!

This point also applies to things like compilation speed, link times, time to run tests etc. In general, people’s tolerances for synchronous work and asynchronous work are quite different.

If you’re a performance-minded person, it can be hard to accept that people are actually fine with putting up with worse performance in software, especially if you already believe that the performance of said software is “too slow.” It can be doubly-frustrating if the same people are willing to put up with worse performance specifically in exchange for more features, especially if you already believe that the said software is “too bloated.”

The budget was zero from the start

Outside of well-paying tech companies that treat developers well, granting them a fair amount of autonomy, it’s common in many companies for the software function to be perceived as a “cost center” instead of a “profit center”.

There may not even be a CI process – it may be entirely reliant on manual QA. Getting budget approvals might take ages.

And yet, the business might be doing well! For example, the company might have a government-granted monopoly. Or it might have some other form of power.

If the environment is entirely focused on keeping costs low, it likely requires a fair bit of effort to convince a manager of the return on investment (RoI) of working on performance. It’s plausible that this effort is better spent elsewhere.

The budget got reduced post-LLMs

Say the budget started out at non-zero. For example, you might’ve already been spending about 1 week on performance every quarter.

Even so, there’s an implicit assumption that the budget $B for obtaining the property X is unchanged post-LLMs. This assumption often fails to hold.

If you browse the r/experienceddevs subreddit, it’s not uncommon to see engineers talking about how, over the past year, timelines for projects are getting squeezed tighter, because management expects things to take much less time due to LLMs.

The cost reduction factor N is over-estimated

It is one thing to implement an optimization. It is another thing to ship the optimization in heavily-used production software. It is yet another thing to set up a ratchet to prevent the code from regressing in the future. It is yet another thing to make sure the ratchet is reliable (low/no false negatives/positives), efficient (runs sufficiently quickly) and stable (doesn’t need constant upkeep).

In my previous post on code review, I gave an example of a situation where a colleague tried to reduce latency for an operation by moving it to a background process, and increased the risk of a lock-acquisition failure.

More generally, it’s easy for people unfamiliar with a system to jump in with “performance optimizations” that actually compromise an aspect of the design that is critical to correctness.

As a more prominent example, Jarred Sumner (creator of Bun) supposedly had a fork of the Zig compiler with parallelized semantic analysis and codegen.

One of the key contributors to the Zig compiler, Matthew Lugg, articulated why this change was not upstreamable:

Parallel semantic analysis has been an explicitly planned feature of the Zig compiler for a long time, and it has heavily influenced the design of the self-hosted Zig compiler. However, implementing this feature correctly has implications not only for the compiler implementation, but for the Zig language itself! Therefore, to implement this feature without an avalanche of bugs and inconsistencies, we need to make language changes.

(..) The rewritten type resolution semantics were designed to avoid these issues, but Bun’s Zig fork does not incorporate the changes (and has not otherwise solved the design problems), which means their parallelized semantic analysis implementation will exhibit non-deterministic behavior. That’s pretty much a non-starter for most serious developers: you don’t want your compilation to randomly fail with a nonsense error 30% of the time.

Another way to look at this point is that the cost of writing the code is only one part of the picture. It may not be the dominant cost.

As two high-level examples:

• If there is already a large amount of data stored in a format that’s not amenable to optimized processing, the cost of optimizing performance needs to account for the cost of reorganizing the data into the right form, while maintaining the reliability, performance and correctness of existing read and write paths. It also needs to account for the cost of migrating the existing code. Worse, you might not even know all the read and write paths, in which case the cost of figuring those out needs to be taken into account.
• If you’re paying for compute over the data, experimenting with different strategies to optimize the computation can itself be expensive. For example, if you’re hitting a flakiness bug only in 1/1000 CI runs, and you can’t reproduce it out of CI, then the cost of CI time to reproduce the bug with sufficient detail can easily dwarf the cost of writing the fix.

Relatedly, one other challenge that comes up is that it’s more difficult to estimate the long tail of costs associated with maintenance: lost code comprehension lost due to the complexity of optimizations, the need to hire more experienced people who can maintain the system, additional correctness checks needed, and so on.

Budget was never the reason for not aiming for X

Instead of cost, I think it’s more useful to think about what work gets done in terms of priority.

For simplicity, let’s say we’re talking about sprint-based planning. Say, on average, pre-LLMs, each person on the team tackled 4 tickets per sprint. Suppose that performance work usually ended up being #6 on the list for someone. At 4 tickets per sprint, this means the performance work would just keep staying on the sprint planning board across several sprints as an aspirational goal.

Now, post-LLMs, say each person can tackle 12 items. Is that list going to stay the same, with just more items pulled in from the backlog so that everyone has enough work?

I suspect the answer here for most people is going to be No. If you were not able to successfully advocate for performance work as a higher priority pre-LLMs, it’s unclear as to why you’d be able to do it post-LLMs. You can just do it on the side for sure (aka “asking for forgiveness instead of permission”), but you could also do that pre-LLMs.

Re-visiting Luu’s examples

From Luu’s post, there are two examples I’d like to discuss, because the code is available, and they represent complex tasks:

1. pgrust: A rewrite of Postgres in Rust.
2. FRE: A regex engine built by an agent loop running over a month.

For pgrust, the headline here is excellent performance on ClickBench, supposedly due to the use of LLM-driven optimization. The other point that’s brought up is that supposedly people don’t write JITs (in the context of databases) because that’s too difficult, but LLMs make that accessible.

Based on a cursory view, it’s unclear as to how much of this excellent performance is down to performance-hacking that overfits the benchmark vs an excellent design that generalizes. For example, if you look at the cost model, you’ll see that it explicitly references ClickBench all over the place.

It’s possible that I’m misreading things, but this seems like a clear-cut case of overfitting to the benchmark.

On the point about other people not writing JITs, there are at least a few database engines which implement JITs.

• Umbra (HTAP), and its commercialized fork CedarDB: Umbra currently sits at the top of ClickBench, with CedarDB not far behind.
• SingleStore (HTAP): Their SIGMOD 22 paper explicitly cites work on compiling queries for HyPer.
• Amazon Redshift (OLAP): Compiles queries to C++.
• Apache Impala (OLAP): Generates LLVM IR.

For FRE, the README states:

FRE is an LLM-generated regex engine made with minimal human intervention. It appears to be overfitted to BurntSushi’s rebar benchmarks and doesn’t have great general performance, although there are some uses cases where it’s actually pretty fast.

I asked an LLM to estimate the implementation SLOC for rust-lang/regex and danluu/fre, and it came back with 35K for rust-lang/regex and 670K for danluu/fre.

Luu makes the following notes about preferences:

There’s no particular reason to use a “software factory” regex engine that doesn’t beat a well-tested regex engine on holdout benchmarks, but one notable thing about FRE was that the native AOT compiled version did quite well at longer searches. We noted that, it stands to reason that one could run the native code compiler in another thread while ripgrep was running its normal matcher and then cut over to the native code when it finished compiling and generally get better performance.

It’s unclear if such an approach makes sense from the POV of software developed in a team context, for general use, which has reliability requirements, and responsibility is assigned to maintainers when things go wrong.

Closing thoughts

To be clear, I don’t want slower (or buggier) software.

Historically, there is a clear trend of performance tools getting better over time. From browser DevTools to eBPF, at different layers, there are increasingly more tools for debugging performance issues. Hardware also keeps getting faster.

At the same time, I think there’s general agreement that the average piece of software is getting slower and more resource-hungry, and that webpages are getting heavier, etc.

I think some part of it is certainly real. For example, if you’ve used coding harnesses shipped by any of the major model providers, you’ve probably noticed how poor they are in terms of performance, resource utilization and overall bugginess.

At the same time, I’m willing to consider that some part of it is imagined. For example, most people, across age groups, think that morality has declined, but it hasn’t.

So yeah, maybe this is useful to keep in mind when you next run into an example of slow software, because whether you like it or not, I think you’re going to hit it regardless.

Google’s Agent2Agent Protocol (A2A) is joining the Agentic AI Foundation (AAIF).

Google’s A2A is an open protocol for agent-to-agent communication. It is an open standard that defines how agents communicate across framework and vendor boundaries. An agent publishes an agent card containing a structured description of what it can do and how to reach it. Other agents read that card, discover capabilities, and delegate tasks without a human brokering the handoff. The exchange is structured, observable, and framework agnostic.

Google launched A2A in April 2025 and donated it to the Linux Foundation with founding organizations like AWS, Cisco, Google, Microsoft, Salesforce, SAP, and ServiceNow. In August 2025, IBM's Agent Communication Protocol merged into A2A showing that the field was moving toward a single shared standard rather than competing ones.

A2A v1.0, the first stable specification, shipped in March 2026. It added multi-protocol bindings and version negotiation, multi-tenancy, and signed agent cards for cryptographic identity verification.

The move puts A2A under the same roof as MCP and other open agent infrastructure projects. MCP standardizes how agents connect to tools and data. A2A standardizes how agents communicate with each other. Together, they form key pieces of the infrastructure needed for large-scale, multi-agent systems.

Anthropic created MCP. Google created A2A. Rather than building competing ecosystems, both are putting these technologies under neutral governance, making it easier for the industry to build interoperable systems and move toward an open “Internet of Agents.”

A2A has grown from an internal initiative into a standard supported by AWS, Microsoft, Salesforce, SAP, ServiceNow, PayPal and others. Much as it did with Android, Google is betting that convening the ecosystem around an open standard will accelerate adoption.

The Quiet Politics of Dashboards: Visualizing Power, Governance, and Blind Spots in African Organizations.

Dashboards are marketed as objective, data-driven systems providing a “single source of truth.” However, deciding which metrics achieve visual prominence is a curatorial act that dictates institutional funding. We explore the “watermelon effect,” “the Accra vs. Field disconnect,” and why blind reliance on dashboards erases unquantifiable human realities.

In today’s digital age, dashboards have become essential tools for decision-making in governments, businesses, and non-governmental organizations. They are widely promoted as objective, data-driven systems that provide leaders with a “single source of truth” by presenting key performance indicators in a simple and visually appealing format. However, dashboards are not neutral technologies. Behind every chart, graph, and performance indicator are human decisions about what data should be collected, measured, displayed, and prioritized. These decisions shape how organizations allocate resources, evaluate success, and formulate policies.

Deciding which metrics achieve visual prominence on a screen is a curatorial act that directly dictates institutional funding, strategic attention, and resource allocation. This “visibility equals value” trap means that complex, long-term human issues such as community trust or quality of care become invisible to leadership simply because they defy simple quantification. Rather than simply reflecting reality, dashboards construct a particular version of reality where only what can be easily measured receives organizational attention and investment.

The quiet, outsized power in modern governance now belongs to the data analysts, product managers, and BI developers who architect these systems and determine the thresholds for success or failure. Relying blindly on these visual arguments means surrendering critical decision-making authority to whoever defined the dashboard’s underlying mathematical parameters. This critique is increasingly urgent in African public sectors and NGOs where the push for automated, data-driven governance is accelerating under various digitization mandates. It is crucial to give local data practitioners the conceptual vocabulary needed to articulate the political weight of their design choices to executives.

SEEING LIKE A DASHBOARD

The idea that dashboards are political rather than purely technical is rooted in political scientist James C. Scott’s concept of “Seeing Like a State.” Scott argues that governments simplify complex social realities into standardized measurements to make societies easier to administer. In the digital era, this phenomenon has evolved into “seeing like a dashboard,” where intricate human experiences are compressed into real-time, colour-coded indicators. While this simplification improves administrative efficiency, it often ignores the complexity of local realities that cannot be easily measured.

Literature from Critical Data Studies further argues that numbers and visualizations are not objective facts but social constructions shaped by the assumptions, priorities, and cultural perspectives of those who create them. Dashboards therefore function as persuasive tools rather than neutral mirrors of reality. They present an appearance of certainty and control while concealing the limitations, biases, and omissions within the underlying data. This can create a false sense of confidence among decision-makers, who may assume they possess a complete understanding of organizational performance when important information remains invisible.

The psychological allure of the executive dashboard lies in its promise of control and simplicity it frames the messiness of institutional governance as a programmable space subject to rational, immediate order. The growing emphasis on efficiency and performance measurement encourages organizations to reduce complex social problems into measurable indicators. Although quantification makes governance easier, it often strips away the emotional, cultural, and contextual dimensions of human experiences. Since dashboards are designed for standardization and large-scale comparison, they naturally prioritize quantitative information while marginalizing qualitative insights that frontline workers and local communities consider essential. As a result, organizational decisions become increasingly driven by what can be measured instead of what truly matters.

THE POWER OF THE ARCHITECT AND THE “WATERMELON EFFECT”

Dashboards should be understood not merely as software tools but as persuasive visual arguments that shape institutional priorities. Every design choice from selecting performance indicators to assigning colours that represent success or failure reflects value judgments about what the organization considers important. These judgments influence management attention, funding decisions, and operational strategies, often without being openly discussed.

Executive leaders are particularly vulnerable to what scholars describe as “dashboard simplicity.” Because dashboards present information in clear, concise, and attractive formats, leaders may focus more on improving displayed metrics than addressing the underlying realities those metrics represent. Departments may therefore redirect resources toward activities that improve dashboard performance while neglecting essential responsibilities that are difficult to measure. The dashboard gradually becomes the target rather than a tool for understanding reality.

This dynamic manifests acutely in what researchers call the “watermelon effect” projects and business units that are green on the outside, bright and confidently reported as on target, but deep red on the inside. By the time leadership is allowed to see the red, it is no longer an early warning but a crime scene. The root cause of the watermelon effect isn’t bad data or misleading KPIs; it is organizational culture specifically, the fear of reporting bad news. In corporate hierarchies, good news defies gravity, shooting upward instantly and amplified at every level. Meanwhile, bad news gets heavier, slower, and filtered at every layer of management. Middle managers learn that bringing a problem without a ready-made solution is political suicide, so when a deadline slips, the instinct is to paint the rind green: “We’ll fix this before the next review.”

The structural opacity of dashboards often obscuring data provenance and algorithm logic creates a severe power imbalance between data architects and operational staff. Frontline workers are rendered hyper-legible and strictly monitored, yet the analytical mechanisms evaluating them remain “black-boxed” and immune to localized feedback. In many NGOs and public institutions, dashboards are designed primarily to satisfy donors, regulators, or senior management instead of supporting the operational needs of staff working directly with communities. Effective governance therefore requires recognizing that dashboards provide only a partial representation of organizational reality and should always be complemented by human judgment and local knowledge.

THE ACCRA vs. FIELD DISCONNECT AND DONOR-FACING DESIGN

The political nature of dashboards is especially visible in many African organizations. Within Ghana’s public sector, an acute “Accra vs. Field Disconnect” occurs when headquarters executives monitor stable, green dashboard indicators while ignoring the nuanced, unquantifiable crises occurring in rural clinics. Agricultural extension officers, health professionals, and local administrators frequently encounter problems that cannot be fully captured through standardized performance indicators. Rigid central parameters systematically fail to capture localized struggles, rendering immediate operational needs invisible to the capital.

Within the African NGO ecosystem, dashboard design is heavily influenced by a “donor-facing” dynamic, where visual interfaces are explicitly engineered outward to placate Western donors and secure ongoing funding. Performance indicators often reflect the expectations of international funding organizations rather than the actual needs of local communities. These metrics prioritize specific key performance indicators demanded by foreign capital rather than inward-facing insights that could actually improve community interventions. Consequently, dashboards become tools for demonstrating compliance and securing funding instead of improving service delivery, offering minimal practical utility to workers managing complex crises on the ground.

Efforts like the Ghana Open Data Initiative highlight that state-level data visibility is continually hampered by institutional secrecy and severe technical funding constraints. Despite policies mandating proactive disclosure, public sector data release remains intermittent, preventing civil society from verifying the optimistic dashboards published by government ministries. Critical studies of African open data ecosystems reveal a pervasive “licensing deficit,” where government portals frequently provide data on a “gratis” basis rather than a “libre” basis legally open for re-use, preventing civic tech designers from remixing state data into alternative dashboards that could challenge official narratives. To counter these imbalances, the African Union Data Policy Framework emphasizes the urgent need for harmonized national data systems, digital sovereignty, and local capacity building, providing a roadmap for African institutions to build governance dashboards that serve local socio-economic development rather than foreign extractive interests.

FROM CURATION TO LIBERATION

A dashboard is a curated argument about what the organization values. It is a political document, not a neutral mirror of reality. Every dashboard reflects choices about what should be measured, displayed, and valued, making it an inherently political document rather than a purely technical one.

For executive leadership teams: Treating a dashboard as a neutral source of truth surrenders organizational strategy to those who define its parameters. Leaders must actively seek out the unquantifiable, on-the-ground truths that fail to make it onto the screen. Red must stop being treated as failure and start being treated as a call for help.

For data analysts and developers: You must be equipped with a critical vocabulary to articulate the social and political weight of your design choices. When asked to “just build a quick dashboard” for a complex social issue, advocate for design transparency that clearly highlights the system’s inherent blind spots.

For African NGOs and public administrators: Deliberately balance outward-facing, donor-centric reporting with inward-facing, operational dashboards designed for frontline workers. Data systems must be fundamentally reoriented to provide immediate practical utility to local teams navigating daily crises, rather than functioning merely as compliance mechanisms.

Ultimately, effective governance depends not only on collecting more data but also on recognizing what remains unseen and ensuring that digital technologies reflect the diverse realities of the communities they are intended to serve. The goal is not to abandon dashboards but to approach them with critical awareness, understanding their power to shape institutional reality while remaining attentive to the unquantifiable truths that no chart can capture.

Key Takeaways

• Dashboards are Political not Neutral: Dashboards act as curated visual arguments rather than objective mirrors of reality. Because "visibility equals value," complex and unquantifiable human issues are often erased from executive strategy simply because they cannot be easily measured.
• The "Watermelon Effect" Hides Reality: A corporate culture afraid of bad news creates the "watermelon effect," where projects are reported as confidently "green" on the outside but are failing and "red" on the inside. This deepens the disconnect between executives in places like Accra and the actual crises faced by rural frontline workers
• Dashboards Must Serve Local Utility, Not Just Donors: In the African NGO ecosystem, dashboards are too often engineered outward to placate Western donors and secure funding. Organizations must reorient these systems to be inward-facing, providing practical utility to the workers managing daily crises on the ground.

Two projects push DataFusion beyond standard analytics: one combines RDF graphs with time-series and temporal logic, while Cloudflare's AutoSQL uses it for distributed serverless SQL over Iceberg and R2. Both rely heavily on custom planning, pruning, joins, and execution optimizations.

TrueForge is a self-hostable agent harness focused on reducing task cost through context compaction, delayed tool-schema loading, subagents, and sandbox-as-a-tool execution. The release matters for data teams experimenting with governed agents because it separates orchestration from model choice and pairs naturally with gateways for identity, access control, observability, and budgets.

I've been a photographer for most of my life. My personal photo library contains, at time of writing, 134,000 photos going back more than 30 years. I like capturing the moment.

So when I left my job last month - after nearly a decade - and realized I would no longer spend every day on the top floors of 4 World Trade Center with its absolutely incredible view, I decided to take some photos out of the windows.

Around that same time, the wonderful Vincent Woo published his absolutely stunning 3D recreation of Grace Cathedral in San Francisco, using a 3D modelling technique called Gaussian Splatting. I was captivated. As I said my goodbyes to my colleagues, I started taking more photos out the windows. High-quality photos, with my best lenses, with my mirrorless camera, in a near-360º circle around the perimeter of one of the most highly secured office building complexes in America.

For outdoor photogrammetry like Vincent's work, the standard practice is to use a drone-mounted camera to take thousands of high-quality, extremely sharp photos in broad daylight. This office being, well, part of the World Trade Center, a drone was out of the question. I value my freedom.

So, I took more than 1,000 photos in all, by hand, with my trusty Sony a7c's 20mm lens carefully pointing out against the glass, taking care to avoid reflections, and only ever in common areas of the office, well away from desks and people. Time was short, and while I wasn't doing anything illicit or even off-limits, I did want to avoid having to explain myself.

There were a number of challenges, mostly to do with how photogrammetry and Gaussian Splats work best if you have consistent lighting conditions. My shoot times would vary as I was shooting in between meetings. While I could have visited multiple floors to get additional data points, that would have to happen after hours so as to avoid disturbing my colleagues, and this would create datasets with differing lighting conditions. (And on the one convenient weekend day that my badge still worked, it rained.)

And most frustratingly: 3D reconstruction techniques really like clean data. Having images with reflections in them - especially consistent reflections like the reflection of a camera's lens up against triple-pane commercial building glass - wrecks havoc on the algorithm's ability to reconstruct things.

But... I had the data. Now I just needed to process it.

...but wait, what's a Splat?

Right, I should start by explaining what a Gaussian Splat is in the first place.

Traditionally, computer graphics "does 3D" by defining points in 3D space and connecting those points together. These form triangles or quads, and allow 3D software to build up a scene much like a drywaller would install sheets of drywall: one face at a time.

In 2026, the state-of-the-art for capturing realistic world data isn't based on this technique: instead, it trains - yes, like with machine learning - a 3D model of a space, by iteratively creating random "splats" of data in a scene and then trying to fit those splats to the training data (images) provided. Splats can also have spherical harmonics attached, which is really just a fancy way to say "if you look at a splat from different directions, you might see different colours."

This doesn't look better in this example, but when you scale this method up to a huge scene, you can get amazing results:

Much like training an AI model, training a Gaussian Splat can be a bit of an art. There are hundreds of different hyperparameters that can be tuned, multiple different popular software packages to use for training, and multiple algorithms that trade off speed for quality in different ways. I've trained many, many, many AI models in my decade as a machine learning engineer at Spotify, and I'm no stranger to content-based modeling - I just usually do it on audio and music data, rather than images.

Enter Claude

Like all engineers, I've been a heavy Claude user for years now. Claude's multimodal capabilities have grown significantly over time, and with Fable 5, I thought I might now be in a position to say:

Here's 1,000+ images taken with multiple cameras. Preprocess them as necessary to create a 3D model out of this, and then train a Gaussian Splat. Make no mistakes.

And so I tried that. The first result was... well, actually not terrible:

The two main pieces of software I pointed Claude to were Brush, an excellent cross-platform splat trainer by Google DeepMind researcher Arthur Brussee, and gsplat, the "standard" open-source renderer from the authors of the original splatting paper. Brush runs on macOS with Metal quite well (although it can be memory-hungry for larger scenes), but gsplat is somewhat better understood in the community, has more features, and requires a CUDA GPU.

I gave Claude three guidelines:

• Use any local machines I have in my house: an old M1 MacBook Air, a newer M4 Mac Mini, and an old Intel NUC. I have no CUDA devices.
• Use up to $100 on Modal, the fastest and easiest way to get access to CUDA devices.
• Notify me when you need my eyes to judge quality tradeoffs.

...and then I went about my month off between jobs, checking my phone intermittently as I biked almost 500km around New York. I expected this might take some time.

Data Quality

Just like training more traditional machine learning models, training a 3D Gaussian Splat requires good data, and lots of it. The splat trainer you use tries to optimize the scene for every photo, and so mathematically, it has no mechanism by which to ignore noise; any reflections, clouds, distortion, haze, or sun highlights will make it into the final product. And because clouds move quickly, I had to delete the entire sky region from the input data.

You have two choices: allow the input data to pollute your scene and then try to clean up the bad splats by hand once training is done, or clean up the input data so the trainer does the hard part for you.

So, I found myself in need of very well labeled, curated, and masked data. In the spirit of using Claude for everything here, I initially prompted Fable to farm out to a fleet of sub-agents to generate masks for each of the 1,000 images in the dataset, so that the training process would ignore sky, occlusions (i.e.: curtains, window frames, etc), and reflections.

However, Claude - from Fable down to Haiku - had a lot of trouble with this task. I'd heard that other models were fairly good at multimodal tasks like this (particularly Gemini), but struck out there too after tweaking multiple prompts.

So, after much consternation, I put a Claude sub-agent on the problem of creating a bespoke image masking UI that would allow me to hand-mask the data. I used Meta's Segment Anything model to create initial "rough" masks for the images to reduce the amount of time I had to click around, but ultimately, I had to do a lot of manual annotation to get enough data.

Due to the fact that the building has sharp corners, I unfortunately had to rely on multiple inconvenient, highly-reflective frames to get a decent view from the corners, which were often some of the most interesting views. To make these images trainable, I had to manually mask out the "occlusions" (i.e.: window frames, railings, vents, reflections, etc).

I exposed this web UI to my phone via Tailscale, and asked Claude to make the UI both mobile friendly and to pre-cache images, so I did most of my data labeling and annotation while riding the subway.

Leaning on Open Data

As progress continued, I realized: I'm scanning a view that is extremely well known; arguably one of the most photographed cityscapes in the world. Surely, I can lean on an open dataset of some kind to ground my training?

And so, I asked Claude to try that: can we grab OpenStreetMap data, which includes building positions and relative heights, to give our training scripts a more reliable canvas on which to paint our images? In a perfect world, this would remove the need to reconstruct a complex 3D depth field, and would let us just project each image directly onto the geometry of the cityscape.

Unfortunately, data quality struck again: OpenStreetMap data is excellent; it's just not quite accurate enough for this purpose, and did not quite fit reality closely enough. Another experimental hypothesis invalidated. I could have played more carefully with camera positioning and projections, but many buildings were just flat-out missing, and others had heights that did not match the scene I had captured.

RealityScan, Lichtfeld, and Windows

At some point, I started to become disillusioned with this project: what if I was doing something stupid by experimenting with AI for driving this very visual, almost artistic process? Surely I should try the tools that others online use.

And the tools that others use are largely: Windows machines, with Epic Games' RealityScan and LichtFeld's LichtFeld Studio, both free but requiring CUDA.

I have no CUDA devices in my house, so I turned to Amazon Web Services to provision a temporary instance. I had Claude build me a simple "Modal-but-for-Windows"-style workflow, in which I would upload a local directory of input images to S3 and automatically download all dependencies onto the box on boot, then connect visually and drive the UI in real time.

However, this proved disappointing: RealityScan was much slower and much less reliable than the newer hloc and LightGlue methods that I had been using locally, and LichtFeld Studio was just a (very!) nice GUI on top of gsplat offered no discernable difference in quality from gsplat. Another $10 wasted.

Diving Deeper into gsplat on Modal

I started to become suspicious of my decision to use local-only, Apple Silicon-friendly software that didn't seem as popular as the "standard" splat stack, which involved using gsplat. Luckily, gsplat runs very nicely on Modal, and Claude knows how to use Modal quite well.

So, I uploaded my images to a Modal volume and had Claude run a large series of tests, sweeping various gsplat hyperparameters and measuring the results with PSNR (peak signal-to-noise ratio). And after much tweaking (and many, many many accidental budget overruns; Claude doesn't know how to forecast its own spend in the slightest), I wound up with a model that looked better, but had more catastrophic artifacts.

Ultimately, while gsplat did technically look somewhat better, it did a terrible job on the borders with the sky, for reasons I couldn't quite identify by the time the project was over. The increase in quality also seemed more like a colour space difference, rather than a core training algorithm difference. I also found that gsplat - by default - performs a single-threaded, CPU-bound image decoding step within the training loop, cutting GPU performance by ~35% due to data pipeline starvation. I had Claude patch in a post-decode cache, which got utilization back up to 92%; but not before I had spent hundreds of dollars on Modal.

A Mobile-First Workflow

This project was happening "in the background" as I was taking my first-ever break between jobs, so I didn't want to stay tied to my computer at home; August in New York is a paradise. So, every step along the way was orchestrated via the Claude app on iOS.

• My supervisor agent, running at home in Claude Code, was always accessible via Claude on iOS.
• Sub-agents' transcripts and progress were directly visible if I was at home, but Claude on iOS does not yet show sub-agent transcripts; so I would treat that as a trust exercise.
• Any intermediate outputs (temporary .ply files, etc) were published to a small Cloudflare Pages website that was visible on mobile, so I could easily have the agent send me a URL to open at any time.
• Any interactive steps (i.e.: data labeling, tracking training) were published with small local servers and then accessed via Tailscale.

I was mostly trying to replicate the apocryphal story of an Anthropic researcher being alerted about their Claude's progress while eating a sandwich on a park bench: I wanted to make a lot of progress on this project while Claude did all of the heavy lifting, so I could spend my month off focused on fitness and wellness. And I think I mostly did.

Exploring Multimodality

I've worked a lot in my career with audio data, spending much of my past decade at Spotify leading its research engineering efforts into audio content modeling. This project proved to be a fascinating but somewhat frustrating experiment into multimodal capabilities of a different sort: how modern LLMs deal with image data and visual perception. At various times, it made simple mistakes:

• I would naïvely use landmarks to refer to certain views: "the view 1 WTC and 3 WTC," knowing that the model had a vague understanding of the positions of those landmarks. But it was unable to correlate that with the images; it would routinely see a cityscape and assume that any tower was the landmark I was talking about, without searching or confirming.
• Fable would fall back on using numerical properties to verify correctness, at one point even building its own 3DGS rendering stack to identify when certain properties (splat density, visual consistency, "floaters" and "clouds") were met or not met. But - much like myself earlier in my career - it failed to actually verify that those scripts gave reliable metrics in all cases, and so it would frequently refer to its own incorrect success metrics and declare victory quite early.
• Claude would regularly observe provided images and notice their overall structure while missing the important details; often showing an image and declaring "this is clearly better" when image quality had degraded extremely.

Reflection

Vincent Woo's Grace Cathedral scan was super inspiring, and I thought I could throw something together with A.I. in a week or two that would approach its impressiveness. How naïve of me.

But in the end, the result looks kind of okay, and is good enough to share. And that's after only:

• Two main orchestrator Fable sessions and a total of at least 74 sub-agents over 3 weeks.
• Roughly 280 distinct experiments (hyperparameter tuning runs, testing different slices of the dataset, etc)
• $234.45 of GPU time on Modal.
• Multiple bugs found by Claude in underlying software along the way: a hang in Apple's Metal shader dispatch code, a sqrt-of-zero NaN in Brush, and an additional bug in Brush affecting the colour of spherical harmonics.
• 3-4 freezes, hangs, and reboots of both my Mac Mini and my MacBook Air due to various reasons.
• Roughly 42M Opus 5 tokens, 6.5M Fable 5 tokens, 2.2M Sonnet 5 tokens, and 10k Haiku 4.5 tokens, which - at API list prices, would have cost almost $1,500.
• $10 in Google Cloud credits to test Gemini 3.5 Flash
• $10 in Amazon AWS credits to try RealityScan and LichtFeld Studio

But apart from these rough edges, Fable did manage to eventually get to a pretty passable end product:

Overall, this was both an experiment in getting my feet wet with Gaussian Splatting, as well as experimenting with using Claude for more esoteric tasks that require a level of human judgement and quality discernment in a way that doesn't map nicely to text. I'm still brand new to splatting, but hope to learn more about it in the future - and if anybody (Vincent?) has tips on how to do a better job with a dataset this sparse and imperfect, I'd love to hear it.

AI Engineering Skills Map: Building and Deploying AI Applications

I previously wrote about our AI Engineering Skills Map, with the highest level skills being (i) Building and deploying AI applications, (ii) Software engineering fundamentals, (iii) Using coding agents, and (iv) Shaping the build. In this article, I will flesh out the first of them.

Being skilled at building and deploying AI applications means knowing:

• LLM foundations
• Grounding models with data
• Building agentic systems
• Evaluation-driven development
• Operating in production
• Machine learning foundations

This map of skills was formed by analyzing a large number of job postings, structured expert interviews, and survey responses.

The key difference between AI applications and non-AI software is that the former’s output is less predictable. You don’t know in advance what an LLM will output, or what predictions a supervised learning algorithm will make. Because of this uncertainty, building AI systems is a much more iterative process than building traditional software — it is harder to plan the process in advance. Skilled AI engineers repeatedly build a piece of software, examine it, and decide what to try next, taking a sequence of steps that are highly influenced by the intermediate results. Being able to skillfully decide what to do next allows you to create reliable software systems based on unreliable AI components. This requires knowing:

LLM foundations

Understanding how large language models tokenize input and generate output allows you to understand when to count on them and when they may fail. It also allows you to understand when to use a multimodal model, how to make tradeoffs on what to include in the context window, and reason about cache hits, knowledge cutoff, reasoning effort level, sampling parameters, and when to use special features such as tool calling. Understanding these foundations helps you choose the right model or mix of models and apply specialized techniques when needed, such as fine-tuning or self-hosting models.

Grounding models with data

LLMs require good input context to produce useful outputs. RAG using vector search was an early attempt to give LLMs relevant context, but the set of techniques for grounding models with data has grown significantly. For example, you will have to decide what to include in a prompt vs. what to let an LLM retrieve on demand using tools, and which representation fits the data and search queries: a vector index, a knowledge graph, or a semantic layer over structured data (such as customer records). You’ll also turn documents (text, PDFs, HTML, images) into LLM-ready inputs and engineer pipelines to keep data clean and fresh. When you understand the menu of techniques available to get data, you are better able to give your LLM relevant context.

Building agentic systems

Agentic systems range from workflows that execute a predefined sequence of LLM calls to ones based on an agent harness that lets an LLM repeatedly decide its own next step. You’ll have to choose the architecture — what steps to chain, what to parallelize, when to use code and when to use an LLM — and engineer the workflow or harness, with fallbacks. When designing the agent loop, you will also decide what tools the model can call (including MCP, CLI and sandbox execution environments), what memory architecture to use, how to manage context over long sessions, and when a task needs multi-agent orchestration instead of a single-agent architecture. You’ll also want to turn promising prototypes into reliable, safe and secure agents for production; this requires understanding guardrails, adversarial inputs, and identifying and working around key risks (such as data exfiltration), and governance.

Agentic workflows are evolving rapidly, and you will also benefit from understanding any cutting-edge techniques relevant to your application area, such as voice agents, computer-use agents, or generative UI.

Evaluation-driven development

In my experience, the most important trait that distinguishes someone great at building AI systems is whether you can drive a disciplined evals/error analysis loop to drive development. This allows you to repeatedly focus your effort on directions that are more likely to be fruitful. I’ve found this to be a tricky skill to master, because the right approach varies significantly by project and even according to the stage of the project.

Building good evals is a deep technical skill. You might look at a system’s traces and outputs, carry out exploratory data analysis, and combine that with product and business insight to decide what to measure. You should also understand the menu of options for evals, such as when to use deterministic (code-based) evaluations, when to use an LLM-as-a-judge, and when to have a human in the loop, and how to evaluate your evals so as to keep evolving them. These evaluations then feed into an iterative process that drives further development, and makes progress systematic rather than random.

Operating in production

Operating AI software is different from traditional software because of its unpredictability, cost, and latency. First, you should know how to build observability mechanisms to understand the system’s performance on real usage. You’ll track performance, detect drift, and respond quickly to model failures and security incidents such as adversarial prompt injections. Putting in place regression testing and CI/CD requires more statistical evaluations than traditional software, and the testing effort should be calibrated relative to the risk of a mistake. Additionally, it’s important to know how to select the right mix of techniques — such as model choice optimization, distillation and fine-tuning, and agentic workflow simplifications — to optimize for cost and latency, especially if your application reaches many users.

Machine learning foundations

Modern LLMs are built using machine learning techniques including supervised learning and reinforcement learning. Every engineer I know that’s good at building with LLMs also understands machine learning and deep learning at some depth. Additionally, many applications still require knowing how to use machine learning – either a model someone else trained or one you train yourself. This requires knowing the popular machine learning and deep learning models and tradeoffs in accuracy, training speed, inference speed, and so on, and understanding how to engineer the data needed to train and evaluate these models. The machine learning concepts of bias/variance, error analysis, and engineering your data — all of which are core mental frameworks for navigating how to work with systems with uncertain output — also remain key to making a wide range of decisions in AI system development.

There is a lot to learn to become good at building and deploying AI systems. This is a field with significant technical depth. But every bit you learn will help you become better at AI Engineering and build more exciting applications. A strong complement to these skills is software engineering. I will write about this in a future post.

Apple has laid off more than 200 employees across its Vision Pro, Siri, and software engineering teams as part of a broader restructuring effort that includes scaling back Vision Pro gaming and immersive video initiatives while reaffirming its long-term commitment to visionOS and future hardware. The company is also reorganizing Siri and AI-related teams around its new Siri AI platform, eliminating some roles, creating new ones, and shifting resources to accelerate development of AI-powered features across its products.

Figma excels at visual design and collaboration, but treating it as a faithful representation of the web can lead designers to overlook how interfaces actually behave in real-world conditions. Static artboards and Auto Layout can mask challenges such as responsive behavior, loading performance, accessibility, unpredictable content, and dynamic states. As products become increasingly adaptive and AI-driven, designers need to think more like systems architects, testing ideas in code, designing for edge cases, and grounding design systems in production-ready components rather than purely visual artifacts.

Balsa UI creates coherent Vue design systems that people and coding agents can understand, install, and evolve.

Borrow brilliance make it yours

A mirror for the world’s best AI minds.

Found a powerful AI tool? Actually use it.

Turn AI apps, agents, and workflows into one-click tools you can run, save, and make your own.

Describe your goal. Assemble the workflow.

Taku connects the right apps, agents, and skills into a workflow made for you.

Package great ideas. Let others remix them.

Turn your AI setup into a Stax others can install, run, and make their own.

stax "Brand voice" {
  input  draft: text
  style  tone: "warm"
  return rewrite(draft, tone)
}
# 1 publish → ∞ installs

Turn your workflow into a distribution engine.

Let the Taku community run, remix, and share your Stax, so your creations travel further and unlock new ways to earn.

• BlockSpark: A no-code block game, shipped in a weekend.
• PodCast AI: Turns my notes into a daily podcast.
• Agentic “Notion”: An agent that actually runs my Notion.
• Super Mario: A playable Mario, no game engine.

From an idea to something that runs.

01 Describe it

Type what you want in plain words. No setup, no config files.

02 Watch it assemble

Taku pulls the right apps from the marketplace into one stack that's yours.

03 Run & remix

It just runs. Tweak it, share it, or publish your own as a Stax.

Publish once. Earn every time it runs.

Your best workflow shouldn't live in a doc nobody reads. Ship it as a Stax and get paid every time someone runs it. No installs, no support tickets.

Choose how you run Taku.

Start free, run real workflows, and upgrade when Taku becomes part of your work.

OpenAI cut GPT-5.6 Sol API prices by more than 20% for three months.

Luke Metz left OpenAI in 2024 to join Thinking Machines, then rejoined OpenAI earlier this year.

New news is coming out so fast that it seems crazy for a founder not to be fully plugged in. Being online can help build a presence and keep people as informed as possible. Posting frequently can help drive inbound interest. Tech Twitter can provide a view of emerging consensus, but few immediately actionable insights. It's important to actively maintain cognitive security and cognitive hygiene, as spending too much time online can be dangerous.

OpenAI has added a new iPhone shortcut that lets ChatGPT users long-press the + button to instantly access and attach their four most recent photos, streamlining a process that previously required several taps through the Photos menu. The update reflects the growing importance of image-based interactions with AI and joins other recent improvements aimed at making photo uploads, visual questions, and longer conversations faster and more seamless.

As AI becomes more deeply embedded in products and services, designers have a growing responsibility to help ensure that technology reflects human values, ethics, and societal needs. Building trustworthy AI requires close collaboration between design, research, engineering, and governance, with values such as privacy, transparency, and safety translated into concrete decisions throughout the development process. A human-centered approach treats design as the connective thread that helps teams align strategy, implementation, and outcomes while continuously evaluating whether systems deliver on the promises made to users and society.

Susan Kare reflects on designing the original Macintosh's iconic visual language and shares lessons on creativity, problem-solving, and the lasting impact of thoughtful interface design.

Canva has shifted from a platform once dismissed as amateurish into a serious industry contender, bolstered by its Affinity and Calvary acquisitions.

Creative director Yorgo Tloupas documents vintage and hand-painted shop signage worldwide under the hashtag #typographictourism, chronicling a vanishing craft.