The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We're not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance.

In response to a comment in my last post that the cost of formerly specialized performance work has dropped by many orders of magnitude and performance work that used to require a person or team that had a rare set of skills can be done by anyone who can type a few sentences, which means that you can do all sorts of optimizations that used to be too expensive to be worthwhile for all but the largest scale or most lucrative projects, Marc Brooker responded with

Completely agree with your closing point. Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome. (Which comes with all kinds of fun risks and opportunities of its own). Kind of reminds me of FFTW. And a ton of weird old demoscene techniques which were all about being super fast and small on a very particular problem (and often very particular hardware). For example, I remember a demo that re-used its code as textures to get great cache locality.

And Michael Malis has noted

There’s been a meme circulating about how AI doesn’t help because “code was never the hard part.” I think that’s true in some domains, but in others, writing the code absolutely was the hard part. JIT compilers are a great example of that. For many pieces of software, a JIT compiler would help a lot with speeding up the code. The rarity of JIT compilers makes me believe that implementing a JIT compiler historically was too difficult for it to be worthwhile. LLMs have lowered the barrier to entry and made it much easier to write a JIT compiler. This is the thesis behind pgrust. Databases historically were the hardest piece of software to build and were limited because of that. Now, with AI, we can be more ambitious about the type of software we build.

Optimizing for a class of workload

Let's try this out with FRE, the regex engine we built in the last post. Recall that it was created by having an agent loop for a month on improving regex engine performance with access to the rebar regex benchmark suite. This resulted in FRE being heavily overfit to rebar until we warned our agent that we had a holdout benchmark, which caused the agent to generalize the optimizations enough that performance was ok-ish on our holdout. There's no particular reason to use a "software factory" regex engine that doesn't beat a well-tested regex engine on holdout benchmarks, but one notable thing about FRE was that the native AOT compiled version did quite well at longer searches. We noted that, it stands to reason that one could run the native code compiler in another thread while ripgrep was running its normal matcher and then cut over to the native code when it finished compiling and generally get better performance. Of course this will generally result in worse performance for short queries as we lose a thread to compilation, but I care a lot more about how long ripgrep takes when it runs for many seconds or minutes than when it runs for a few seconds, so I'm ok with that tradeoff.

In the same way we could build a regex engine in a few minutes of human time, we can also just try this experiment in a few minutes of human time. I typed a few sentences and an agent went and did the work to allow this to happen (which would be a decent chunk of code surgery for a human) and it ran the benchmark on actual ripgrep queries that come from my codex history. For longer queries, we see a 2x-4x performance improvement here for a few very simple queries. But most queries are more complex, and when we run on representative holdout queries, for queries where AOT should be enabled, we get about a 7% speedup. Not an earth shattering result, but also not a bad outcome for spending a few minutes typing to codex (and it's still doing more optimization and will presumably speed things up further).

Build an index?

This is arguably a silly thing to do, since if we're repeatedly searching for text on a computer, the obvious thing to do to speed that up isn't to write a native code compiler for regex matching, it's to create an index. But the point here is just that this kind of technical work, which used to take a fair amount of time and expertise, can just be done trivially now. And if we wanted to build a text index, it just so happens that I worked on BitFunnel, the Bing search index that was specialized for constant/fast text ingestion that won Best Paper Award at SIGIR, so I can think of a few experiments to try if we're going to build a fast local index of our entire machine.

If I were working at an AI lab and had access to things like SOTA models running on Cerebras chips or other accelerators that greatly increase tok/s and therefore load/demand for search, I might actually survey the existing indexers to see if they're fast enough or if I'd want to build something custom myself. While the open source version of BitFunnel "only" contains a bytecode interpreter and one JIT, the Bing version contains multiple JIT compilers. A project that did that level of optimization used to be a major undertaking, but "I could do that in a weekend" is now actually true for some of these kinds of projects. With my lowly $200/mo account, I think a somewhat faster ripgrep plus any off-the-shelf index is fine, so maybe this fast-ingesting whole-machine index project can be left as an "exercise for the reader (who works at an AI lab)".

Optimizations are cheap

The drastic reduction in the cost of optimizations has been true going back to November 2025 and maybe even somewhat before then with public models. For an example from the GPT-5.1 or 5.2 days, with no knowledge of game AIs, I tried building an Azul AI. This ended up being the strongest AI in the world for the game by a pretty large margin. From reading the thesis that describes the 2nd strongest AI, I think my AI is probably a bit better on the "AI" side of things, but the main place it wins is on optimization despite spending what looks like maybe two orders of magnitude less time and also mostly working on my laptop vs. having a cluster of machines to use. For example, that other AI is single-threaded and my AI is multi-threaded. Since I have a native code version as well as a heinous shared wasm memory + javascript version, and two different search architectures for two different versions, this would've been a fairly large undertaking if done by hand.

There's a bunch of standard stuff it makes sense to do to debug and verify a multithreading algorithm for something like this, like implementing replay from debug logs that can reproduce bugs despite the algorithm being nondetermistic. Doing that alone would've probably been days to a week of work had I done it by hand, but it's exactly the kind of thing an agent can trivially do in a loop. A lot of the tedium it used to take to get a tricky optimization like this working is gone.

This also applies to a lot of other tricky optimizations. From having written CPU microcode, done CPU verification, worked on optimizing a search engine index, etc., I have a lot of experience looking at optimizations and thinking "hmm, this would increase performance by 2%, but it's going to take N person-days to verify that this tricky optimization works" and making a call to go ahead or not based on whether or not it's worth the time to get the optimization working. Now that this N has dropped by a tremendous factor, the number of these kinds of optimizations it makes sense to do goes way up.

Going back to the game AI case, at least for the AI I tried, it seems like you gain about 100 Elo for every doubling in speed. Just adding multithreading alone is enough to wipe the floor with an otherwise comparable AI on a large machine. If you stack in 10-20 more optimizations that seem too annoying for most people to do by hand, the difference in strength is tremendous and it's not really reasonable to try to keep up with a hand-written AI.

To pick another example, as part of preparing for performance interviews, Jamie Brandon tried Anthropic's now public performance takehome. After trying it, he had Claude pick up where he left off and it got a much better result. When he looked at what Claude did that he didn't, he said a lot of the optimizations were things that occurred to him but he hadn't gotten to yet, and "[o]thers were just crazy shit that I would never try unless I was working on this for weeks".

Workload-specific optimization

Coming back to this part of Marc Brooker's comment:

Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome.

This seems pretty inevitable. In another response to my post, Michael Malis of pgrust said something similar:

[discussion of pgrust optimizations] ... I think it's easy enough to create these optimizations that we could look at a customers workload and add them as needed

Without having any kind of framework or setup, right before I started writing this post, I had an agent do workload-specific optimization for my ripgrep queries, which took about 2 minutes for me to launch. The optimizations run on a set of queries, and then there's a later holdout set of queries to run against. After one pass of optimization, the workload optimized version is 2% faster than standard ripgrep on the holdout and it's still getting faster. 2% isn't a big deal for my local ripgrep usage, but considering that this took minutes of time, I'd take a 2% win here.

Appendix: There's no reason for software to be slow anymore

I've been on the record for a long time as strongly disagreeing with the general sentiment that the developers of X are bad and should feel bad for writing slow code because there are a lot of different kinds of programming expertise and not only is it not the case that most programmers don't have performance expertise, it probably doesn't even make sense for them to develop, so of course most projects will have very poor performance compared to what a performance expert can do. I can see why a performance expert would look at the growing gap between how fast a program can be and how fast programs actually are and think that it's ridiculous.

I still don't think someone is bad and should feel bad if their software has poor performance, but I do think that someone who doesn't know anything about performance and is a reasonable user of LLMs should generally be able to create software that has decent performance. If you just tell an LLM to optimize, it will often do all sorts of incorrect things that are really bad that you have to catch, but that's generally true of using the LLM effectively in the first place, so getting decent performance is no longer a specialized skill.

Appendix: How is codex running ripgrep?

Here's some information about the distribution of riprep queries on my machine. The p50 is 55 unicode code points, which is already longer than things I grep for by hand, and the p90 is 119!

But some are funny numerical constructions. The entire pipeline for that was

cargo clippy … | rg 'crates/fre-aot-regex/src/module.rs:' | rg NUMBER_REGEX | head -250

which might be an odd thing for a human to do, but agents seem to do this kind of thing all the time.

On another topic, if we look at how long ripgrep queries took, there are quite a few slow queries, e.g., p99 is almost 1 minute! And p999 is almost 10 minutes! And the maximum query over this time period is approaching 2 hours!

On a draft of the last post, Peter Geoghegan noted

It's also possible for a regex implementation to be faster by supporting fewer features. Some implementations don't support back references, etc.

which is also true here. The workload-specific optimizations done here were fairly superficial because I just gave codex some short instructions and let it do whatever it wanted, but with a more detailed plan, more focused optimizations supporting the common use cases for my queries could be expected to yield larger gains.