🔍 Read the full analysis: OpenAI, Google, Microsoft And Anthropic Back One Opt-Out: A User Guide on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will support a shared robots.txt signal for publishers who do not want their content used to train generative AI models. The opt-out is voluntary, does not block all AI uses of a site, and is not yet an official internet standard; companies have set different implementation timelines.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will support a shared robots.txt signal for website publishers who want to opt out of having their content used to train generative AI models. The move could spare publishers from maintaining separate training opt-outs for each participating company, but the signal is voluntary, does not cover every use of web content, and is not yet an official internet standard.
The proposed rule uses the user-agent label __usage_ai-training followed by Disallow: / in a site’s robots.txt file. It expresses a publisher’s preference against training use. The shared signal is described in an Internet-Draft titled “Machine-Readable Training Preference Signal,” published by Anthropic standards lead Kevin Kelley on October 30.
The companies have not set one common launch date. Anthropic says it has started implementing support, though it says adding support across its services will take time. OpenAI says it aims to implement the signal “in the coming months.” Google is aiming for 2027, with its crawler documentation specifying early 2027. Microsoft says it will honor the shared training preference from February 1, 2027, as it crawls websites.
Microsoft also announced a separate page-level preference for AI Grounding—the use of web content to inform answers in Bing, Copilot and Microsoft 365. Its announcement says both preferences take effect February 1, 2027. Microsoft says its existing NoArchive and NoCache tags will stop counting as preferences for training or grounding after March 1, 2027. Existing controls such as GPTBot, Google-Extended and ClaudeBot remain available, according to the source material.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
One Rule, Several Company Timelines
For publishers, a shared signal could reduce the work of expressing the same training preference to multiple AI companies. Instead of relying solely on separate crawler-specific rules, site operators can add a common line and, as each company implements it, have that preference recognized across participating services. The practical benefit depends on adoption: dates differ, and the participating firms have not all completed implementation.
The distinction between training and other uses matters. The shared signal is for foundational model training; it does not, by itself, block crawling or prevent content from being used to produce AI answers based on live web information. Microsoft’s separate grounding preference addresses one such use in its own products. The source material also says the shared opt-out does not create payment, guarantee traffic, or determine inclusion in search results.
For readers and publishers, this is a new way to communicate a preference, not a technical lock. The cited announcements describe companies choosing to honor the instruction. They do not establish enforcement or a mechanism that can prevent every crawler from accessing a page.
Products Worth Considering
Jackery Explorer 1000 v2 Portable Power Station
LG 34GX90SA-W 34-inch Ultragear WQHD OLED Curved Gaming Monitor
Apple Watch Series 12 GPS Smartwatch, 46mm
As an affiliate, we earn on qualifying purchases.
The proposal is connected to the AI Preferences working group at the Internet Engineering Task Force, which develops internet standards. The draft offers a common machine-readable preference, but it remains an individual Internet-Draft and has no formal standing in the IETF standards process, according to the source material. At a working-group meeting on October 21, one chair said, “We haven’t declared consensus on anything yet.”
Company announcements refer to the same draft but describe different commitments. Google says that if a site uses the new token and has no Google-Extended rule, it plans to treat the new signal as if the publisher had disallowed Google-Extended. OpenAI characterizes its support as a voluntary effort, while Anthropic says implementation has begun. Microsoft’s announcement adds its distinct grounding control and a transition date for its existing tags.
Meta spokesperson Andy Stone said the company supports publishers’ ability to express preferences and called a common signal an important step. He did not commit Meta to honoring it. The source material reports no statement from Apple, Amazon, xAI or Perplexity about whether they will adopt the signal.
““has already started implementing support for this draft””
— Anthropic
Adoption and Coverage Remain Open
It is not yet clear whether the draft will gain formal support through the IETF process, or which companies beyond the four named will honor it. Meta has expressed support for the idea but not a commitment, and the source material identifies no adoption statements from Apple, Amazon, xAI or Perplexity.
The precise rollout across each company’s services also remains unsettled. Anthropic says implementation has begun but will take time, while the other firms describe future targets. The signal records a publisher preference; the cited material does not describe a technical enforcement system or establish how disputes over compliance would be handled. It also does not remove content from models already trained, and it does not cover all uses of content in AI-generated answers.
Publishers Face Staggered Rollouts
Publishers can add the shared training rule while keeping existing company-specific crawler instructions until each provider has implemented the new signal. Those seeking to control search, training and AI answer uses should treat them as separate preferences rather than assume one rule addresses all three. Microsoft’s stated February 1, 2027 start date and March 1 deadline for its legacy NoArchive and NoCache tags are dates site operators may need to track.
The next developments to watch are whether the IETF working group reaches consensus on the draft, whether Meta or other AI companies make explicit adoption commitments, and whether the four companies meet their announced schedules. Google’s documentation points to early 2027, while OpenAI has given a less specific target of “the coming months.”
Key Questions
What does the shared robots.txt signal do?
It expresses a website publisher’s preference that participating companies not use the site’s content to train generative AI models. The proposed rule uses __usage_ai-training with “Disallow: /”.
Does the signal block AI companies from crawling a website?
No. The source material says the signal is a training preference, not a block on crawling. It also does not cover every use of web content, including AI answers based on live information.
When will the companies honor the signal?
Anthropic says implementation has started; OpenAI aims for the coming months; Google aims for 2027, with its documentation saying early 2027; and Microsoft says it will honor the preference from February 1, 2027.
Has Meta agreed to honor the opt-out?
Not based on the statement cited here. Meta spokesperson Andy Stone voiced support for publishers expressing preferences and for a common signal, but did not make an implementation commitment.
Is this an official internet standard?
No. It is an Internet-Draft connected to the IETF’s AI Preferences working group. The source material says it has no formal standing in the IETF standards process at this stage.
Source: ThorstenMeyerAI.com
