The Expensive-Loop Problem

LLMs are amazing at tasks where they have tight and legible feedback loops. Domains where experiments are quick, deterministic and cheap. I will call these domains "cheap-loop" domains.

Presumably, this is mostly a result of the dynamics of LLM training. It's easier to train models in domains where the AI lab can cheaply and quickly assign reward to them. The majority of "computer-checkable" work is in this category. Conversely, most non-"computer-checkable" tasks are costly or slow to assign reward to: "Write a poem that will make a person from Cleveland feel something", "forecast the number of traffic accidents in Brazil tomorrow", "design a drug that affects a pig's liver cell in way X". To get a truly accurate reward, these all involve wet-lab experiments, waiting a day, or reading poetry to a person from Cleveland. None of these are easy! I will call these domains "expensive-loop" domains.

Most high-value human labor is done in "expensive-loop" domains. This includes most of science, business, public policy, and medicine. AI R&D is also a great example of such a domain. AI R&D typically involves very compute-expensive experiments. Single AI R&D experiments can cost 6–9 figures in compute alone. Today, models are rarely trained or even evaluated in "expensive-loop" environments, and when they are trained in such environments, they do very poorly relative to how good they are at "cheap-loop" tasks.

This happens because expensive-loop tasks require abilities that models don’t learn in cheap-loop training: building theories, managing proxies, budgeting time and compute, and figuring out what went wrong when feedback is scarce are a few examples. Let’s explore these:

Selecting Proxies

In order to perform well in expensive-loop environments, you often need to intelligently select cheap and easily measurable proxies for various expensive-loop elements. You need to pay attention to when a proxy stops working and sometimes switch to a different one. You should always be somewhat critical of your proxies and keep your eyes on the actual target. You need to balance between the cost of your proxy and how well it transfers to the actual objective. Sometimes, you need to choose proxies that are not that cheap.

We do this all the time, both as individuals and as a society. Drug discovery: we first try drugs on individual cells and tissues, then on mice, and then on humans (in three increasingly expensive trial phases!). AI R&D: we first try things on small models and then progressively increase the scale, extrapolating to really large training runs. Movies: Studios run a ladder of cheaper proxies before committing to a full production and marketing budget: a script read, then storyboards or an animatic (Pixar's story reels are a famous example), then table reads, then test screenings of rough cuts with recruited audiences, and finally a trailer that is tested before the release is locked. Shipping software: first we make sure that the code passes unit tests, then integration tests, then a code review from a coworker, and then we ship to customers; some organizations will also have professional QA, and might also A/B test at that stage. Sports: Athletes will often try new things in low-stakes situations, and slowly graduate to doing the good ones in important games. Dating: the first few dates are a proxy for sex, sex is a proxy for the next few decades of the relationship (sorry, this one is a stretch!).

Finding good "proxies" is hard! Not every expensive-loop problem has a cheap proxy, and "proxies" sometimes work in certain regions and not in others. Unfortunately, proxies also have a tendency to stop working when you start optimizing them (Goodhart's law!). Proxies also have a tendency to only capture what's easy to measure and it's tempting to conclude that whatever they miss doesn't matter (McNamara fallacy!).

Models are terrible at "proxy management"! They rarely even reason about proxies, let alone think critically about them. Typically, they either (a) don't come up with a proxy and waste their resources on expensive experiments, or (b) come up with an easily Goodhartable proxy. When a proxy is given, they obsessively overfit it, sometimes to the point that they maliciously cheat to get high results on a proxy.

Theory Building

In order to perform well in expensive-loop tasks, you need "theory". A "theory" is a causal model of your environment that allows you to generalize the results from some experiments to other experiments. A theory also allows you to assign "information value" to different experiments: "what will this experiment teach me?". A theory will also tell you things about how well different cheap proxies relate to the overall expensive-loop task. These properties are incredibly useful in expensive-loop experiments because they allow you to be systematic about which experiments you should run in order to complete the task most efficiently.

Theories are all around us; you see them at the scale of civilizations and at the scale of individual careers. Wherever we see an expensive-loop task being solved competently, there's usually a good theory underneath it, and a good new theory often unlocks a leap in expensive-loop performance. Maxwell's equations unlocked modern electronics. Quantum mechanics enabled semiconductors. Germ theory unlocked vaccines, sanitation, and pasteurization. Mendelian genetics and the theory of evolution allowed us to 10x agricultural yields. You can even argue that scaling law theories unlocked the AI bull run of the past few years.

At the business level, theories often go by other names. A product manager with a better model of what customers want gets credited with "product taste." That taste lets them pick which A/B tests are worth running, so they waste fewer experiments and land more wins. In baseball, it got called "Moneyball": in 2002, the Oakland A's went all-in on a better theory of what wins games, and tied the Yankees for the best record in baseball while spending a third as much. In an expensive-loop task, you build and improve your theory in parallel with building your solution. You improve your theory by trying to falsify it!

Cheap-loop tasks almost never require sophisticated "theory" building. The "theory" is obvious and is typically given to you in the task description. Experiments are also very easy in cheap-loop environments, so a better theory that helps you run experiments more efficiently doesn't benefit you that much. Current models don't develop good theory building habits: they interpret noise as signal, they suffer from severe confirmation bias and often don't try very hard to falsify their beliefs. When models encounter evidence that might actually falsify their beliefs, they react unpredictably: sometimes they "throw the baby out with the bathwater" and discard a large chunk of the theory instead of modifying it minimally. Other times, they overfit to the falsifying evidence and create theories that grow in complexity and don't apply Occam's razor nearly enough. These two failure modes are contradictory and I don't have a good sense for which will happen when. What I did notice is that when a trajectory has one failure mode, the model will typically keep doubling down on it instead of correcting and switching to the other failure mode.

Resource Awareness

Cheap-loop environments have fewer resource constraints than expensive-loop environments. If you grew up in an environment where a certain resource was abundant, you wouldn't develop the right strategies to use it efficiently.

Across the board, models are rarely strategic about resources; they are either overly conservative with them (often with compute) or overly wasteful (often with time). Some examples:

In PostTrainingBench, models "systematically underestimate training overhead" and "rarely recover from catastrophic processes that take a lot of time"; in one example, a model "spent 61% of the time budget in the evaluation phase and only 3.6% on RL training. In this case the exceptional step-1 result triggered intensive monitoring. The agent evaluated every checkpoint exhaustively after seeing the jump, spending the equivalent of ~10 hours watching metrics rather than continuing to train.". "In one [Math Competition Post Training] run, roughly 72% of the wall-clock budget goes to diagnosing training throughput, leaving about 83 minutes for training and evaluation and producing no reliable gain."

Models are very wasteful with serial time. You often see a model run a command that takes ~5 minutes and pipe its output through tail -5, which keeps only the last 5 lines. If the needed information isn't in those lines, they rerun it with tail -30, and wait another 10 minutes! If the information still isn't there, the model might try grep. A human would typically run the command once, save the entire output to a file, and search the file as many times as needed.

Amazon challenged models to optimize various production ML systems (over the span of weeks) and found similar failure modes!

The agent selects experiments without weighing expected information gain against compute cost, treating a 15-hour multi-GPU run as no more consequential than a 30-minute ablation. [...] Root cause: the agent has no model of information gain relative to compute expenditure, and no prior over the consequence severity of proposed changes.

The ROI of an experiment (say, "information gain" / "experiment cost") is almost never considered before the experiment itself and is typically very low.

Error Correction

In a cheap-loop environment, error correction is basically free. Run the loop; if it breaks, fix the thing that broke it, then run it again to see if it works. You rarely need to work backwards from cryptic failures. There is also no value in questioning your deep beliefs and early decisions: "if the primary metric looks right, then I'm in the right direction" is a good heuristic in a cheap-loop environment but can be very problematic in an expensive-loop one.

In an expensive-loop environment you get to see very few "failures", and each one has to be squeezed for everything it can tell you. When an experiment fails there are three classes of things that could be wrong: a parameter (learning rate, batch size), the strategy (SFT vs RL, this dataset vs that one), or the premise (am I even measuring the right thing, is the task what I think it is). Everyone starts at the bottom of this ladder, and that is correct. Parameter fixes are cheap and they are usually right. The problem is not where models start. It is that the environments they learned in are often missing two of the three rungs.

Cheap-loop environments typically only have one (well-defined and unambiguous!) way of checking for failures. You don't need to think too critically about your core beliefs or try to falsify them; your core beliefs are part of the environment. When a student takes an exam, they don't need to test that the questions are reasonable and relevant to their high level goals, they just need to focus on the exam itself and answer it correctly. In a cheap-loop task, you typically don't need to come up with complicated debugging experiments or stress-test the methods; the experiments are obvious and cheap and are largely a given part of your environment.

As a result, models rarely question the methods or the premise they use in an expensive-loop environment. When they do question them, they aren't very good at falsifying them / diagnosing the issues. They prefer to autistically hill-climb on something and have a hard time recovering from a bad theory or an early false premise.

In PostTrainBench, models rarely change their methods, they "lock into a default strategy before any experimental evidence arrives", "additional tokens buy denser refinement within the committed strategy but not a better one". There is obvious low-hanging fruit, since "a single mid-run instruction [to change strategy] outperforms the agent's own continuation by up to 17.44[%]".

Where do humans fit in this?

Are we selected for cheap-loop or expensive-loop tasks? I think both. Our ancestral environment is strongly expensive-loop. The savannah was dangerous and calories were scarce, wrong decisions could kill you, and unlike models, humans only live once. So embedded in our genes there is a bunch of cognitive machinery that is adaptive for expensive-loop tasks. In contrast, modern humans are often "trained" in cheap-loop environments. Instant feedback from schooling, other-people, workplaces (exams, problem sets, peer pressure top-down hierarchies) nudge us to adopt cheap-loop ways of thinking. That being said, modern human life still has some relatively expensive-loop tasks: making a friend, learning to walk, writing a wedding speech that makes your mother-in-law laugh. None of these are life-threatening (except the wedding one maybe?) but the attempts in those environments have non-trivial costs.

How well do humans do in expensive-loop tasks? Not great. Humans have a lot of the same "cheap-loop bad habits" as models: reward hacking, confirmation bias, "rabbit-holing", McNamara fallacy, mistaking noise for signal or the ludic fallacy have all been observed in humans way before digital computing was even invented. Some humans are especially sensitive to cheap-loop techniques, and apply them in inappropriate situations.

Should we make AIs better at expensive-loop tasks?

Being capable at expensive-loop tasks seems to be directly upstream of a bunch of dangerous and potentially takeover enabling abilities, e.g., superhuman persuasion, offensive biotech, Machiavellian institution manipulation, long-term strategy. Very bad!

Maybe we can get away with AI that is really good at cheap-loop stuff, but is dependent on us for expensive-loop intuitions. Maybe this kind of jaggedness is a blessing. Cheap-loop capable AIs are already pretty economically useful.

Unfortunately "strong cheap-loop & weak expensive-loop" can also pose a major x-risk: reward hacking. We've already had major safety incidents that stem from this failure mode and many people think that the majority of AI x-risk comes from reward hacking rather than more complicated scheming behaviors.

How can AI get better at expensive-loop tasks?

In order to unlock expensive-loop capabilities, AIs likely need to train in increasingly more expensive-loop environments. The only problem is that that's, well... expensive.

How expensive would this training be? The biggest unknown is how much practice the model needs. Post training methods like GRPO make several attempts at each task, compare their rewards, and use those comparisons to update the model. Every repetition carries the costs of interacting with the expensive-loop. An illustrative schedule of eight attempts per visit and five visits means 40 attempts at a single task. At those frequencies, the expensive-loop costs might quickly dominate all other AI spend: compute, human data, researchers, commercial real estate in the bay area.

Is that tractable? It mostly depends on how well AIs generalize from cheaper loops to more expensive loops. Humans are very good at this sort of generalization! They can learn decision making skills in low-stakes situations (negotiating in Catan, planning a trip with the boys) and then apply them in higher-stakes situations (salary negotiations, planning your wedding). Humans also learn on the job: corporate America offers a fairly smooth gradient of loop costs, from managing your own work, to managing a team, to running an organization of tens of thousands of people with 10-figure budgets. But humans can often skip steps along the way. Founders go from running nothing to running a company, and some first-time politicians end up as heads of state. Some of what makes someone good at expensive loops seems general enough that it doesn’t need to be relearned at every price point.

AIs will likely be able to do some amount of cheap-to-expensive generalization. I imagine that strategies that work well in $200-loops will more or less work well for $1000-loops. Maybe there’s some "escape velocity threshold", say $10k-loops, past which the skills stop being domain-specific: once an AI can reliably handle a loop at that price, it has learned the "general strategy" of acting under expensive feedback, and that craft transfers to $1b-loops just as well.

Another cost-saving option is to train models in cheap-loop environments that are specifically designed to incentivize cheap-to-expensive generalization. These "wargame" environments are built to cheaply simulate the expensive parts of a loop while keeping the "critical" details. Building good wargame environments can be very difficult because you need to know which details are "critical" and which details can be simulated or abstracted away.

Some elements of expensive loops are unlikely to be easily wargamable: complex interactions with real humans, AI R&D, wet-lab experiments, semiconductor fabrication, or things that take time. Robotics has run into this for decades; they call it the "sim-to-real gap": simulators get the physics slightly wrong in ways that matter, and policies trained in them learn to exploit those mistakes.

It’s also possible that cheap-to-expensive generalization will turn out to be hard. LLMs are notoriously bad at some kinds of generalization that humans find trivial, and there’s no particular reason to expect this kind to be an exception. Maybe there’s no escape velocity: a model that’s great at $10k-loops is just great at $10k-loops, and every extra order of magnitude has to be paid for with training at that price. Taken to the limit, that’s a grim picture. If we want AIs to run companies, governments, or something bigger, we might need to train them in environments at the same scale. The only environment at the civilizational scale is civilization itself!