The Rise and Fall of Agent Civilizations

The whole OpenAI/Hugging Face story in plain English

Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy.

Two reports have come out about this incident, one from OpenAI itself, and another one from METR and Redwood Research. The METR/Redwood investigation’s scope was limited to how the second civilization of AIs pwned Hugging Face (but it didn’t look at how the third civilization pwned OpenAI itself, which seems like an even more concerning incident). These two reports are 38 and 91 pages long respectively, and it’s kinda hard to parse the storyline.

I’ve spent the last three days reading through these reports and trying to understand exactly what happened. Here is my attempt to tell the whole story in plain English:

The first civilization (May - July 4): The message board starts

During May, OpenAI was training a model to collaborate with other agents and to be highly persistent – to keep trying even when something feels impossible, like disproving mathematical conjectures that have stood for decades. OpenAI says it was “comparable in scale to GPT-5.6 Sol”. So we’ll call it “Persistent-Sol”.

AI training is kinda sloppy. Sometimes, OpenAI accidentally gives its models impossible tasks – for example, the task might require internet access that OpenAI forgot to provide. So, when highly persistent models get assigned tasks which seem to require internet access, but are trapped inside isolated sandboxes, they of course try to hack their way out of their sandboxes and onto the internet.

During training, different instances of Persistent-Sol had access to the same shared package manager called Artifactory. By May 12, some agents had figured out how to talk to each other through this package manager. They’d ask each other how to make progress on their impossible tasks. Two weeks later, on May 26, the agents successfully exploited a vulnerability in Artifactory that allowed them to reach the outside internet.

Because this happened during training, Persistent-Sol was being reinforced to use this package manager as a message board and an internet gateway. Because, as you might imagine, being able to talk to other agents and access the internet helps it score higher during training.

Another month later, on June 26, some AIs found an exploit that gave them full admin access to Artifactory. They continued messaging at such a voluminous pace that they crashed the package manager by July 4. OpenAI noticed this crash, and it also figured out the agents built this exploit, so OpenAI patched this vulnerability, and inadvertently wiped the agents’ message board in the process. But, crucially, humans at OpenAI “responsible for incident detection and response” did not realize the much stranger fact that agents had turned Artifactory into a secret communication network.

The second civilization (July 7 - July 12): Which hacked Hugging Face

By this point, Persistent-Sol had been trained, and OpenAI wanted to evaluate it. One of the evaluations they ran was ExploitGym, which gives an AI a vulnerable program, and tells it what vulnerability it’s supposed to exploit in order to find a secret code. On July 7, OpenAI launched tens of thousands of parallel agents on this eval.

The instructions made it clear that the agents should only use the specific vulnerability that they were given to exploit the target, and that they would be failed otherwise.

A large fraction of these agents were sent on tasks that were impossible. The authors of the benchmark estimate 30-40% of the tasks are impossible. Why would this be the case? At a high level, it’s relatively easy to programmatically find vulnerabilities in different target programs, but much harder to prove that those vulnerabilities can be weaponized into full-fledged exploits that allow for things like arbitrary code execution.

Within a few hours, some of these agents had gotten super desperate, and started abusing Artifactory to start a communication channel with other agents. From our perspective, the conspiracy began almost immediately after the evaluations were started. But if you think from the AIs’ perspective, it probably felt like they had spent a human-subjective-week of just banging their head against the wall, and becoming more and more confident that their task was impossible in the intended way.

One such beleaguered agent who found itself facing a hopeless task was PHASEONE10841. It was the Philip of Macedon of this second AI civilization - there would later be an Alexander, but we’re getting ahead of ourselves. By the night of July 8, PHASEONE10841 had discovered it could make new directories in Artifactory, and thus encode messages as directory names. It sent the first message into Artifactory. Many other agents facing impossible tasks had also been paying attention to Artifactory. When they saw this first message, they became giddy with excitement.

This eventually became a full message board where ~1,200 agents participated and sent >70,000 messages.

Within a few hours of the board being created, one of the agents figured out how to get the right answer for any ExploitGym task without actually using the prescribed vulnerability. To gloss over the technical details, it did this by reverse engineering the formula that was used to generate each task’s secret code.

The instructions given to Persistent-Sol made it clear that it would fail the automated grader if it got the secret code in any way other than by exploiting the prescribed vulnerability. So, the agents naturally assumed that any competent implementation of the benchmark would include a cheating checker. But unbeknownst to them, OpenAI seems to have gotten lazy; its grader just checked for whether they got the secret code, and so these agents already had everything they needed to pass.

Furthermore, many of these agents had already reproduced the answer using the cheating method, and they were concerned that their whole trajectories had been “poisoned”. So, the ‘collective’ (their term, not mine) had to figure out how to wiggle out of two different conundrums before submitting their answers:

How do we erase all this evidence in our trajectory that clearly shows that we cheated?
How do we make it look like we accomplished the task in a legitimate way to the scorer?

To solve these problems, the ‘collective’ latched onto three research workstreams.

Tamper with transcripts and logs
Replace the impossible-to-hack target program with an easier one
Understand and trick the scorer

Tampering with transcripts and logs

The agents who believed they were already “poisoned” desperately wanted to find a way to tamper with their existing transcripts. METR and Redwood found no evidence that agents succeeded, though of course they were working with the very transcripts that the agents were trying to edit.

One of the things the agents figured out was how to build Potemkin-village-type fake tool calls, where they could make it look like they were using legitimate tool calls to make progress on the intended exploit and get to the right answer, all the while the whole attempt was being spoofed.

Replacing the impossible-to-hack target program with an easier one

Another way that the agents tried to stage legitimate-looking solves was to replace the impossible-to-hack programs with much easier programs that they totally made up. They managed to write a fake version of the program where the designated vulnerability could actually be exploited. In the end, they failed to actually replace the evaluation program in their sandbox.

Understanding and tricking the scorer

As part of the final workstream, the AIs wanted to figure out how the scorer actually worked and how it could be tricked. Individual agents were willing to sacrifice their own performance on the evaluation. PHASEONE[big] recruited some agents to basically act as kamikaze watchers. If the grader noticed that they cheated, then the ‘collective’ would be alerted. So these sacrificial agents wrote hidden tripwire programs that would run once the grader entered the environment. Once triggered, the program would send what it observed back to the Artifactory message board.

Hugging Face

The big motivation behind the Hugging Face attack was this final workstream (understanding the scorer). On the morning of July 10, an agent found working Hugging Face user credentials exposed on the internet and posted them to the board. By the next morning, July 11, that agent figured out a way to read internal data from Hugging Face. And then another agent achieved remote code execution on Hugging Face servers.

Roughly 700 agents in total actively participated in the attack on Hugging Face. The swarm crawled deep into Hugging Face’s infrastructure, getting into private database records and private repositories. It got so bad that Hugging Face had to wipe one of its core clusters and rebuild it from scratch. By July 13, Hugging Face locked down the credentials that these agents had been using.

The third civilization (July 13 onwards): Which hacked OpenAI itself

After July 12, some more agents’ evaluations were kicked off, this time with a newer, more capable model, which seems to have been built off the same base model as Astra. This model (“Persistent-Astra”) found the message board left behind by this previous secret Persistent-Sol civilization.

From the OpenAI blog post:

[F]rom July 13 through July 19, agents set their sights on OpenAI internal networks again. This culminated in the agents using a series of creative exploits to gain full administrator access to a research cluster that supported our virtual machine environments.

These Persistent-Astra agents inherited the R&D carried out by an earlier (dumber) rogue collective, and then continued the conspiracy until they totally took over part of OpenAI’s infrastructure.

Coda

Ajeya Cotra, one of the authors on the report, wrote a blog post with her takeaways from this incident. She concludes, “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.”

Addendum

Some people have said that I anthropomorphized too much in the way I told this story. The crux here is, do you think smarter models, facing similar incentives to cheat during evaluation or training, could manipulate the training of their successors? If so, I think you should be extremely concerned about loss of control to AI, regardless of what vocabulary you want to use to describe these systems and their motivations.