Who Aligns the Aligners? Brief Legal Thoughts on the “AI Safety” Fights to Come

Dario Amodei, the CEO of Anthropic, has published an essay – We Must Pace The Frontier – in which he writes:

I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life. I’ve written often about these incredible benefits: I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom.

But – and there is always a but –

…like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious.

Those risks include, according to some, the complete destruction of the human race.

See, e.g., Eliezer Yudkowsky confidently asserting today that if we do not institute immediate global techno-communism, instituting draconian government control over speech and publication of a type never before seen in any Western society, we are all going to die.

There is no evidence that this will happen. Some proponents of regulation tell us that the only response is the most extreme response available: total state control. There is no evidence that this response is correct, either. One could just as easily argue, hypothetically, that the government should force Anthropic to open-source its weights, so that everyone can have free, equal access to the latest model as a personal defense AI to protect themselves from cybersecurity risks from other AIs, a “Second Amendment for AI” if you will. In the alternative, if you really think AI is an extinction-level risk, it would seem to me that the only rational response to that position – if genuinely, truly held – is not “let governments run it” but rather to agree to destroy, by treaty, all modern computers and revert to 1970s technology in perpetuity.

There are myriad policy responses. Whatever we do, those responses will require popular consent and careful deliberation. No one person, or one company, or one movement, knows the answer and history is no guide, save that apocalyptic predictions about new technologies have, to date, all been wrong.

History does provide a great deal of guidance, however, about the use and misuse of government power. It tells us that the state is in fact likely the worst possible custodian for the most powerful publication and data analysis technologies.

This notwithstanding, to address this risk, Amodei proposes:

…building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas.

To wit, regulation.

As my regular readers will be aware, I have been engaged, on behalf of my clients, in legal combat with Internet censors around the world, agencies who think that they have both the standing, the competence, and the right to tell American companies what software they can write and run, for the better part of 18 months.

This censorship apparatus is about to be rebuilt from scratch, except this time for AI instead of social media. I expect to fight that, too, at the appointed time. I feel now is an appropriate time to offer my preliminary thoughts.

Amodei’s Proposal

Anthropic is, as David Sacks correctly pointed out on X, free to slow down its research and development efforts into AI at any time, to any extent it wishes. Amodei proposes something else: that everyone slow down together, under supervision. While Amodei initially writes that the “slowdown” should be voluntary, the plan would be to progress to legal regulatory regimes – meaning, this proposal necessarily involves the use of coercive state power – which software developers would be expected to obey on a compulsory basis:

The most effective method of pacing is via regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily.

His plan has three elements.

Embedded Evaluators (aka Appeasing Pressure Groups)

The first element is for “Embedded Evaluators” –

…employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.

There is already a robust industry of third-party “safety” overseers for Web 2.0 – what the House Judiciary Committee has described as the “censorship-industrial complex.” The track record of these entities from the last time around tells us how this arrangement plays out in practice.

One well-known private actor in this space was the Global Alliance for Responsible Media, or GARM. GARM, a commercial enterprise, described itself as “a voluntary cross-industry initiative created in 2019 to address digital safety.” Among other things, GARM provided a range of policy frameworks and guidelines, among them “the Brand Safety Floor and the Adjacency Standards Framework, which have supported brand owners in their independent development of their own bespoke, brand-specific safety frameworks to ensure that their advertising dollars do not inadvertently support illegal or harmful content that damages their brands.”

Although GARM disbanded in 2024, according to the House Judiciary Committee, during its active period GARM worked with global regulators to pressure companies like Twitter, now X Corp., to wield “significant collective power” to coercively influence Twitter’s moderation decisions, including “silencing President Trump,” and to procure boycotts of the platform if the platform refused to obey.

He adds:

This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees.

As it happens, the banking analogy is the exact argument leading “misinformation/disinformation” (read: pro-censorship) academics employ to justify the UK’s Online Safety Act and similar regimes. The problem, of course, is that financial accounting fraud is not a constitutional right; speech is. In America, software development absent the intent to commit or facilitate the commission of a crime is, as a general rule, protected expression. I fail to see how standing up a new crop of NGOs to perform substantially the same function as the “Online Safety” NGOs, using the same methods – only, this time with NGO commissars possessing highly sensitive employee-like access to internal systems – will lead to a different or better result than it has so far.

Democratic Coordination (aka Government Regulation)

The second element is “Democratic Coordination,” whereby

Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.

There are two aspects to this: (a) competition law and (b) content regulation law.

From a competition law standpoint, the problem Anthropic has is simple. Anthropic and OpenAI are the largest players in the AI market, by some distance, and coordinating their policies, procedures, and “standards” with each other risks classification as an unlawful cartel. “Government support” for “legally challenging” coordination is a polite way of asking for an antitrust exemption to allow greater coordination between competitors in the name of “safety.”

From a content regulation standpoint, the language “common safety standards as well as limits on the rate of unchecked AI progress” paints with a broad brush. This suggests that Anthropic envisages that practically any industry using its software – from manufacturing, to biotechnology, to news publication and copywriting – will require (a) de novo “safety” standards in relation to non-expressive conduct, and (b) limits on how quickly AI software itself can be developed.

The primary legal problem with this aspect of Anthropic’s proposal is in the United States, particularly with (b) – limits on how quickly software itself can be developed. Software development, software publication, and web hosting are inherently expressive activities. To the extent Anthropic and its fellow-travelers intend for this aspect of their plan to restrict American citizens from either (a) developing AI models or (b) using models and published FOSS model weights from China, First Amendment issues are immediately apparent and the weight of the precedent militates against government regulation.

Global Coordination (aka Treaties and Extraterritorial Censorship)

The third element of Amodei’s plan is “Global Coordination,” whereby

[t]he US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

The last great global effort to regulate publication and communications technology began following a moral panic brought about by the twin shocks of (a) Brexit and (b) the election of Donald Trump. The result was comprehensive Internet censorship laws in Australia, the United Kingdom, and the European Union – all of which seek to control speech and conduct which (a) lives on American servers and (b) in the United States, on those servers, is constitutionally protected under the First Amendment.

A “global coordination” framework for AI will be built by the same governments, staffed by the same regulators, and pressured by the same NGOs that built the above. It is possible, even likely, that any global attempts at harmonization will collide at exactly the same point: it will not be possible for an American company to comply with European controls and enjoy the full breadth of their U.S. constitutional rights at the same time, as the rulesets will be drafted incompatibly.

Amodei writes:

We should approach any global pacing decision, especially in the near term, in such a way that protects the lead of the US and its allies.

I do not view this as being particularly realistic. Among America’s geopolitical adversaries, several of which America is at war with (directly or by proxy) and who have every incentive to defect, effective compliance levels will likely approach zero.

Preliminary View: Who Aligns the Aligners?

If I have learned anything from our fight against European censors, it is that regardless of a regulatory regime’s good intentions, regulators are subject to political control. Regulators will, subject to that political control, do political things.

The entire enforcement process against SaSu, from pre-enactment lobbying to closing the file, was a single, continuous, political act. That is how a so-called “independent” expert regulator in a modern western democracy will behave under political pressure in what should have been an easy case.

This is also what we may expect an “embedded evaluator,” a coordinated standards body, or a global AI compliance regime will do under political pressure, because those bodies will be run by humans, and human beings are (a) fallible and (b) respond to incentives.

It is probable that Amodei’s proposals are already being ingested gleefully by “Online Safety” regulators and the related academic ecosystems around the world as they look to expand the reach and remit of the censorship schemes over Web 2.0 that they have spent the last decade building. It will not take a decade to update their censorship apparatuses to try to regulate yet another area of American tech, nor will it take a decade for the vast advocacy apparatus they have built around “Online Safety” to replace-all and begin pushing an AI safety narrative in legislatures around the United States, and around the world.

If OpenAI and Anthropic choose to adopt formal endorsement of prior restraint as corporate policy, those who would oppose the global regulation of AI must move quickly. The counteroffensive against government censorship of AI does not have the luxury of time.