Back to Devoured - November 02, 2026
DEVOURED
Supporting a shared AI training control for websites

Supporting a shared AI training control for websites

AI policyweb Google
Google is adopting a standardized robots.txt 'policy token' to allow website owners to opt-out of AI training across multiple providers simultaneously.
What: Google Search Advocate John Mueller announced that Google plans to support an IETF-standardized 'AIPREF' signal in robots.txt by 2027. This will allow publishers to issue a single opt-out instruction that covers multiple participating AI providers, while maintaining specific overrides for individual tools like Google-Extended.
Why it matters: This move aims to simplify the increasingly complex landscape of web crawling and AI training permissions, potentially reducing the friction between content publishers and AI labs.
Takeaway: If you manage a website, keep an eye on the IETF AIPREF working group specifications; you will soon be able to replace multiple provider-specific block rules with a single unified directive in your robots.txt file.
Decoder
  • robots.txt: A text file that provides instructions to web crawlers about which pages or files they can or cannot request from a website.
  • IETF (Internet Engineering Task Force): An open standards organization that develops and promotes voluntary Internet standards, particularly the standards that comprise the Internet protocol suite.
  • AIPREF: A proposed standard mechanism for websites to declare machine-readable preferences regarding the use of their content for AI model training.
Original article

Supporting a shared AI training control for websites

Today, we're announcing our plans to support a shared AI training control in robots.txt. This will help websites express their AI training preferences across multiple AI providers in a single step and is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF). This builds on our long history of providing website owners with clear controls, including through Google-Extended, the AI training control we've offered since 2023.

How the new control will work

Instead of managing separate instructions for every AI developer, website owners will be able to use a shared 'policy token' within robots.txt to set broad preferences across participating AI providers.

When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended. Other AI providers can handle it similarly, making it easy for website owners to opt out across all providers, or to pick and choose.

Existing company-specific controls will continue to function alongside this new joint signal. So website owners can easily express a broad choice using the industry signal, while also tailoring their instructions to individual AI developers like Google through controls such as Google-Extended. Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

The new mechanism brings website owners the benefit of the AI training definition developed at the IETF while multistakeholder discussions continue on the rest of the AIPREF vocabulary. As more industry players adopt this mechanism, it gives website owners early and expanding access to the preference signal.

This draft proposal reflects feedback from publishers, civil society groups, regulators and other stakeholders that have called for industry-wide controls on AI training. Support for this proposal across major AI providers will enable a future standard that is cross-industry, robust, and durable. This proposal is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act that recognizes robots.txt as a primary method for web publishers to express machine-readable preferences. We'll continue to work with these stakeholders to refine and standardize this through the IETF working group.

We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress.

Explore all 69 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi