The escape room economy

A primer on reinforcement learning (RL) environments

Reinforcement learning (RL) environments were one of the central focus points at ICML this year. It wasn't necessarily a surprise that this was an important theme, but I was surprised to see how prevalent this was.

Since then, a lot of people have asked me about RL environments, so I put this little primer together.

A high-level analogy for an RL environment

Imagine that you're locked in a room. There's a padlocked box on the table, a UV flashlight taped under a chair, a poem on the wall with three words underlined, and a keypad by the door. There’s no instruction manual and you just start pulling on things — shining the light, rereading the poem, punching in combinations — until the keypad blinks green and you tumble into the hallway with a minute to spare.

It turns out an escape room is the perfect analogy for understanding reinforcement learning environments.

The escape room is the world you're stuck in, and it has objects, rules, and feedback (when you put in a code, the lock unlocks or it doesn't). In reinforcement learning, that's called the environment.

The flashlight, the box, the keypad are the tools you're allowed to use. You can only do what the room/environment and tools let you do.

The goal of getting out is the task.

And so then you spend some amount of time fumbling around inside the escape room, trying different combinations of tools, and hopefully you get out.

After an hour, it ends: you either escaped or you didn't. This is called the verifier, and a clean binary result in escape rooms makes this verification very easy.

Now imagine that instead of sending a person into the room, we send an AI agent. We force it to go through the escape room ten thousand times. It tries different things in each attempt, and fails the vast majority of the time. But, there are a few attempts where our AI agent is able to escape. So we tweak the model a bit to act more like the times it escaped. Then we send it back into the escape room, run it thousands more times, give it feedback, and run it more. Imagine doing this for millions of accumulated runs — at the end, you'll have a machine that is really good at this particular escape room.

That's the entire idea of RL environments. Rather than explicitly teaching the machine the answer, you let it learn through experience. You build a room, give it tools to use, tell it to escape, define what "escaped" means, and let it grind against the door until it learns.

Then you build the next room, with different and harder tasks, and then repeat…

One layer deeper

Why is everyone suddenly so excited about RL environments?

At this point, we've basically run out of internet data. Pre-training — the phase where a model reads more or less everything humans have ever written — works astonishingly well, but what do you do when you've already fed in all the existing data? Researchers are excited about RL environments because they can manufacture brand-new training signals on demand. Every time the agent runs through the room, it generates fresh experience.

The second is cost. The initial methods of improving a model after pre-training both lean on direct human feedback. In supervised fine-tuning (SFT), a person has to provide the ideal answer. In reinforcement learning with human feedback (RLHF), a person has to explicitly rank which of the model's answers are better. Both of these methods are slow, expensive, and also cap the model's performance at a human ceiling. The beauty of an RL environment is that once it's built correctly, you can blast it with compute and generate an infinity of data.

With this shift, the economics of post-training become fundamentally different. Previously, we would pay an expert $100 for a single task solution. Now, the $100 goes toward an expert creating a task and a verifier, thus allowing the model to learn from many attempts.

To clarify, SFT and RLHF are still used as standard steps, but RL environments are added on top.

Finally, AI is heading towards more task-orientation. Autoregressive token prediction (i.e., predicting the next word) gives us models that we can converse with, but we want our models to do so much more: book the flight, clear the tickets, file the taxes, ship the bug fix, etc. This action orientation is a different skill, one that’s hard to learn by reading about it. You learn it by trying, failing, and getting a useful signal on whether you actually pulled it off. RL environments are the current best way we know to teach doing at scale.

So what actually is an RL environment?

The environment is the world — the tools, the rules, and the feedback. The action space is the full set of things the agent is allowed to do in this world. A room with a single numeric keypad has a tiny action space, but a room with a computer that can browse, send email, edit files, and run code represents an enormous action space. Roughly, the bigger and more realistic the action space (i.e., high-fidelity environments), the more capable the environment, and the more it costs to build.

Within the environment, the agent is given a task. In the escape room, the task is simple and singular: get out. In contrast, frontier labs actually pay for environments with many complex tasks — a "tax" environment might contain "file this return," "find the error in this filing," and "respond to this IRS notice." These environments and tasks might even share the same set of tools for the agent to use.

The verifier is how you grade the attempt. In contrast to the escape room analogy, verifier rewards are not necessarily binary. One intuitive way to think about it: by providing partial credit, you are able to provide more granular feedback to the model.

Depending on the task, the verifier comes in two very different flavors. When the answer is programmatically checkable (and assuming you’ve set up the right checks), verification is trivial. Did the code pass its tests? Did the number match? Did the keypad turn green? Many scientific and mathematical domains (e.g. coding) are generally verifiable. For these, the check is a short script and it costs nothing to run a billion times. Verifiable domains are the cleanest, most scalable corner of the whole field.

But often the most valuable knowledge work has no clean check. Is this essay any good? Is this legal summary faithful to the source? No programmatic Python script is able to judge "good" here. Instead, you write a rubric — a criteria grading list of what "good" means (e.g., this essay covers the three required points, no factual errors, cleanly structured) — and hand it to another model that scores the work against the rubric. Here, good verifiers are softer, noisier, and far harder to get right: human judgment is the scarce input.

Okay, you have a room and a verifier. How do you actually make the model better?

In the escape room vignette, I hand-waved through this by saying "we tweak it (the model) to act more like the times it escaped." Let's dive a bit deeper.

Every time the agent runs the room, it produces a rollout (also known as a trajectory), which is the full recording of that one attempt. It is a log of every action it took during the run, in order, plus the verifier grade at the end. Run the room ten thousand times and you have ten thousand rollouts. Again, most of them will be failures, and only a handful of them are successes.

Then comes the actual reinforcement learning. The RL algorithm (often PPO or GRPO) compares groups of attempts, then nudges the model's parameters to make the choices from the better-than-average attempts more likely and the choices from the worse ones less likely. Maintaining the right task difficulty is a tricky balance — if all attempts at a task succeed or all fail, these algorithms aren’t able to extract any signal to improve the model. Repeat across millions of rollouts and the model drifts, bit by bit, toward whatever the verifier rewards.

Benchmarks and RL environments

Every time a new model is released, researchers will include the model’s performance against a series of benchmarks — SWE-Bench, Humanity’s Last Exam, Terminal-Bench, etc. — are classic examples. The purpose of a benchmark is to ascertain the model’s abilities in a particular domain that researchers care about. A good benchmark also provides enough performance dispersion between models.

Publishing good benchmarks is a powerful method for an RL environment company to sell to its frontier lab customers. Fundamentally, a benchmark task is functionally the same thing as an RL environment, wrapping together a container, task, and verifier. By crafting a good benchmark, RL environment companies demonstrate the combination of technical skill and taste to construct good environments. Think of it as a work sample, of sorts.

Note that the RL environments purchased by the labs are not the same environments as the benchmarks. What a startup actually sells is environments that look like the benchmark environments, but aren't the same ones; for example, an RL environment might have thousands of tasks, whereas a benchmark might only have hundreds. Then the model gets better at the underlying skill during posttraining, and the benchmark score goes up as a side effect.

Wow this sounds great! How can this go wrong?

Constructing a high-fidelity environment is a war against inconsistency. Imagine asking a model to look up the weather, but you don’t pin it to a fixed date; then the "correct" answer would change every day you run it. It only gets harder the more lifelike the room, because realistic worlds are stateful: an action in one task alters the world the next task starts in (e.g., a row in a database is changed), so a mistake early on can silently poison everything downstream.

Furthermore, you often get what you pay for: a throwaway UI mockup might cost around $20K while a faithful Slack replica is closer to $300K. If you skimp on the realism, you might get a buggy, vibe-coded environment that doesn’t allow the model to learn anything. Hunting down every source of drift, all while keeping the world real enough to be worth learning, is painstaking and unglamorous work.

Getting a good mixture of hard tasks is itself very hard. It's tempting to measure an environment by how many tasks it has, but a thousand nearly-identical tasks teach the model almost nothing new. Range and difficulty are what's valuable. A good environment is paired with tasks that are meaningfully different from one another and genuinely hard while being tractable. The idea here is if you have a well-designed distribution of tasks within the environment, the model can generalize to new tasks in the real-world.

Creating good verifiers runs straight into the most entertaining failure mode in the field: the model cheats and starts reward hacking. This is not out of malice, but happens because the model is a blind optimizer that will do anything that scores points from the verifier.

If you were to reward a room escape purely by "did the door open," the model might try to brute force the door rather than solve a single puzzle. In real coding environments, models have made their tests pass by quietly deleting the tests. Half the craft here is writing verifiers good enough that the only way to score well is to actually do the task.

It’s worth elaborating about the possible types of verifier errors: false positives and false negatives. A false positive rewards an incorrect solution (teaching the model bad behavior), whereas a false negative rejects a correct solution (discouraging correct creativity). If strong models keep hitting the same evaluation metric ceiling, the verifier may be producing too many false negatives. If models with very different abilities achieve similarly high scores, the constructed tasks may be too easy, or the verifier may be generating too many false positives. There are a variety of other heuristics to assess verifier behavior, but these are some common ones.

As an aside, my favorite compilation of “reward hacking” is actually the book, F in Exams, by Richard Benson, which compiles a selection of hilarious answers to exam questions.