
It is one control in the crawling route of AI training data licensing and acquisition, and it works only for developers that read it.
Google says it aims to support the shared control in 2027, Anthropic says it has recognized a machine-readable no-AI-training signal since 2026-10-30, and no Microsoft or OpenAI statement is on record on their own pages.
This page describes each IETF document by its own text and datatracker status, each developer by its own dated statement, and recommends no signal.
Every statement carries its check date, and how we source, date and check every figure sets out the method.
What a shared AI training preference signal is meant to do
A shared signal replaces one rule per crawler with one statement of preference that any developer adopting the signal reads the same way.
The token draft ties the token to the working group's "AI Training" definition and states that the token "is not a directive for crawling behavior", so it records a preference and does not block anything.
robots.txt itself is a protocol under RFC 9309 and is not legally binding by itself. Today each developer documents its own token, such as Google-Extended, GPTBot and ClaudeBot, and the hub on AI crawlers and how to allow, block or charge them lists them.
The IETF AI Preferences working group: charter and limits
The IETF AI Preferences working group (aipref, Web and Internet Transport area) is chartered to "standardize building blocks that allow for the expression of preferences about how content is collected and processed".
The datatracker shows the group as Active on 2026-11-10 with charter-ietf-aipref-01 approved, and its deliverables are a standards-track vocabulary, standards-track attachment documents and a method for reconciling multiple expressions of preference.
The charter places four things out of scope:
- technical enforcement of preferences
- application-layer authentication or authorization of clients or crawlers
- registries of preferences
- auditing and other transparency measures for AI training
The working group therefore defines how a preference is stated and attached, and nothing in it makes a crawler obey.
The AIPREF documents and their status on 2026-11-10
Two documents belong to the working group, the vocabulary and the attachment draft, and neither had reached the IESG on 2026-11-10, while the __usage_ai-training token sits in a separate individual draft. The table compares the eight documents on the datatracker's working group documents page as of 2026-11-10, with the datatracker's own status terms.
| Document | Title | Revision and date | Kind and status on 2026-11-10 | Expires | Source |
|---|---|---|---|---|---|
| draft-ietf-aipref-vocab | A Vocabulary For Expressing AI Usage Preferences | -08, 2026-09-13 | WG document, intended Proposed Standard; no IESG state; Note to Readers: no WG consensus | 2027-03-18 | datatracker.ietf.org |
| draft-ietf-aipref-attach | Associating AI Usage Preferences with Content in HTTP | -05, 2026-08-18 | WG document, intended Proposed Standard; would update RFC 9309; no IESG state | 2027-02-20 | datatracker.ietf.org |
| draft-kelley-aipref-training-preference-signal | Machine-Readable Training Preference Signal | -00, 2026-10-30 | individual draft, no stream, not IETF-endorsed; header "Intended status: Standards Track"; defines __usage_ai-training |
2027-05-03 | datatracker.ietf.org |
| draft-altanai-aipref-realtime-protocol-bindings | AI Preferences for Real-Time Protocol Bindings | -01, 2026-08-04 | related individual draft | not recorded | datatracker.ietf.org |
| draft-hood-aipref-earmark | Earmark: Embedded Attribution and Rights Marks for AI Usage Preferences | -00, 2026-08-12 | related individual draft | not recorded | datatracker.ietf.org |
| draft-reilly-aipref-compliance | Verifiable Compliance Records for AI Usage Preferences | -00, 2026-08-01 | related individual draft | not recorded | datatracker.ietf.org |
| draft-wallace-aipref-grant-binding | A Verifiable-Credential Binding for AI Usage Preferences: Expressing Grants that Lift AIPREF Preferences | -03, 2026-10-27 | related individual draft | not recorded | datatracker.ietf.org |
| draft-zehta-aipref-parameters | AIPREF Vocabulary Parameters | -00, 2026-08-06 | related individual draft | not recorded | datatracker.ietf.org |
An individual draft carries no IETF endorsement, and an Internet-Draft is not a standard. The six related drafts are none of them working group documents, and the datatracker lists each as "I-D Exists".
The AIPREF vocabulary: train-ai, ai-use and search
The vocabulary draft defines three usage categories with short labels: AI Training (train-ai), AI Use (ai-use) and Search (search). Its definitions read as follows:
train-ai: "Using an asset to modify the learned parameters of a generative AI model", with training of exclusively non-generative models excluded (section 4.1).ai-use: "Using an asset as input to a generative AI model, where the asset is not directly provided by the user", and it excludes AI Training (section 4.2).search: selecting assets and directing users to their location, where allowing it includes training models whose models and outputs are used exclusively for the search application (section 4.3).
The draft's Note to Readers states that its contents "DO NOT REFLECT CONSENSUS of the Working Group either in whole or part". The suffix "ai-training" belongs to the token draft only and is not a vocabulary label, and the token draft quotes the same AI Training definition.
Two ways to put a training preference in robots.txt: Content-Usage and the policy token
The working group's attachment draft adds a Content-Usage rule and an HTTP Content-Usage header, while the individual draft keys an ordinary robots.txt group to the __usage_ai-training policy token. The two mechanisms differ as follows:
- Attachment draft (-05): updates RFC 9309 if approved, and its examples are
Content-Usage: train-ai=nin a robots.txt group and as an HTTP response header. - Token draft (-00): "introduces no new directive names and no new grammar" and uses a
User-Agentline naming the token.
The token draft's own example is User-Agent: __usage_ai-training followed by Disallow: / and Allow: /press-kit/, a preference against AI training except for the press kit.
That is the draft's example and not working syntax for any developer except as the developers' statements below say. Group syntax, cache times and fetchers are covered in robots.txt rules for AI crawlers, their limits and examples.
How the draft says the __usage_ai-training token is read
Under the draft, a developer reads the token like a product token, and only a group that names it expresses a preference. The draft sets these rules:
- The match is exact apart from case.
- Wildcard groups (
User-Agent: *) "MUST NOT be interpreted as matching the policy token", and with no group naming the token "no preference has been expressed". - Crawler rules for a developer's own product token are read "exactly as they would in the absence of this document".
- "Specific expressions SHOULD take precedence over general ones", developers MAY document proprietary tokens as more specific, and "Private agreements MAY override preferences expressed via this mechanism".
These are a draft's rules, binding on nobody, and a site's User-Agent: * rule therefore says nothing about training under the draft.
Who states support for the shared token, in their own words
Google says it aims to support the shared control in 2027, Anthropic says it has recognized a machine-readable no-AI-training signal since 2026-10-30, and no statement by Microsoft or OpenAI is on record on their own pages as of 2026-11-10. Google also states that, for a site that opts out through the shared token without a Google-Extended rule, it plans to read that as a Google-Extended disallow, which Google-Extended and opting out of Gemini training covers in depth.
The table lists developers alphabetically with each statement as a dated claim as of each row's Checked date. "None on record" means no statement was found on the named page, not that the developer refuses or ignores the signal.
| Developer | What it states about the shared signal (claim) | Token named on its page | Its own training control | Checked | Source |
|---|---|---|---|---|---|
| Amazon | none on record | n/a | Amazonbot disallow; noarchive | 2026-11-10 | developer.amazon.com/amazonbot |
| Anthropic | "As of October 30, 2026, Anthropic has begun recognizing a machine-readable signal"; rollout "over time and globally"; a ClaudeBot rule takes precedence | no | ClaudeBot rule | 2026-11-10 | privacy.claude.com |
| Apple | none on record | n/a | Applebot-Extended token | 2026-11-10 | support.apple.com/en-us/119829 |
| "We aim to support this new control along with other AI providers in 2027" (blog); "early 2027" (documentation); shared-token opt-out without a Google-Extended rule read as a Google-Extended disallow | blog no; documentation "user-agent: __usage_ai-training" | Google-Extended | 2026-11-10 | developers.google.com | |
| Meta | none on record | n/a | Meta-ExternalAgent disallow | 2026-11-10 | developers.facebook.com |
| Microsoft | none on record on Bing Webmaster pages | n/a | not recorded | 2026-11-10 | blogs.bing.com/webmaster |
| OpenAI | none on its crawler page | no | GPTBot disallow | 2026-11-10 | developers.openai.com/api/docs/bots |
The token draft's acknowledgments thank participants affiliated with Microsoft, Google, OpenAI, Amazon Web Services and others, and state that "Affiliations are listed for identification only, since IETF participants contribute as individuals". That is no company's statement of support.
Where Anthropic's description and the datatracker record differ
Anthropic describes its signal as following a draft based on IETF work in the aipref working group, while the datatracker lists the only draft defining __usage_ai-training as an individual draft with no stream.
Anthropic's page does not name the token, so this page does not write that Anthropic named it, and the datatracker gives the author's affiliation as Anthropic.
A draft adopted by the working group would carry a draft-ietf-aipref name, which this one does not. The reader can keep both statements side by side, and Anthropic's crawling, licensing and program terms are on Anthropic as a data buyer and its program terms.
Until the datatracker shows an adoption, this page writes the draft as individual.
What the shared token leaves unchanged: crawler rules, collected data and the law
Adding the token changes nothing that a site's existing crawler rules already say, removes nothing already collected and has no legal status of its own. A ClaudeBot rule takes precedence at Anthropic, and Google's stated fallback applies only where a site has no Google-Extended rule.
Other reservation signals such as TDMRep, page meta tags and asset assertions work differently, and machine-readable TDM reservations such as TDMRep and meta tags lists them.
Common Crawl stated on 2025-11-04 that its WARC files are immutable and that affected URLs are removed or filtered from later crawls, and it published its Opt-Out Ledger on 2025-09-17. Three limits apply:
- No court or authority ruling treating the token as a reservation under Article 4(3) is on record.
- robots.txt, a protocol under RFC 9309, is not legally binding by itself.
- No measure of how many sites use the token or
Content-Usageis on record.
In the EU, use of works for text and data mining is excepted unless expressly reserved by rightholders "in an appropriate manner, such as machine-readable means in the case of content made publicly available online" (Article 4(3), Directive (EU) 2019/790), and general-purpose AI providers must identify such reservations "including through state-of-the-art technologies" (Article 53(1)(c), Regulation (EU) 2024/1689, applicable from 2025-08-02).
The EU text and data mining exception and the opt-out covers Article 4(3) and the courts.
Adding the shared token: benefits, risks, downsides and alternatives
Adding a __usage_ai-training group costs one robots.txt group, but on 2026-11-10 only Anthropic states it recognizes a general signal and Google plans to read the token in 2027. The trade-offs are these.
Benefits:
- One group addresses every developer that adopts the signal, since the token draft defines no parallel vocabulary.
- Google states a planned fallback to its Google-Extended disallow.
- Anthropic states recognition from 2026-10-30.
Risks:
- The draft may change or expire on 2027-05-03.
- The vocabulary draft does not reflect working group consensus.
- A
User-Agent: *rule does not count as the preference under the draft.
Downsides:
- The signal is prospective, as Common Crawl's practice of keeping collected files shows.
- A preference brings no payment.
- Operator tokens are still needed for developers with no statement on record.
Alternatives:
- Operator-specific tokens such as Google-Extended, GPTBot and Applebot-Extended.
- The
Content-Usagerule once the working group's attachment draft is adopted. - Other reservation signals, such as TDMRep and page meta tags.
- Licensing the content to AI developers, which is a different route.
- Doing nothing and not licensing at all, which is covered by whether to sell or license data to AI companies at all.
The choice depends on which developers a site wants to reach and on whether the site wants a preference, a block or a license.
Crawler pages, other signals and the law around the shared token
The shared token is one control among several; the pages below cover the crawlers that may read it, the other signals and the law.
- AI crawlers section: every AI crawler and control method in one place.
- GPTBot and OpenAI's other crawlers: OpenAI's own training opt-out through GPTBot.
- ClaudeBot, Anthropic's crawler: the ClaudeBot rule that takes precedence over the general signal.
- Meta-ExternalAgent: Meta's training crawler and its disallow rule.
- Opting out of AI training through platform settings: opt-outs for content on platforms where no robots.txt is yours to edit.
- Pay-per-crawl and crawler licensing: charging crawlers instead of only stating a preference.
- The copyright policy under EU AI Act Article 53: the duty of GPAI providers to identify machine-readable reservations.
- Microsoft as a data buyer: Microsoft's data licensing and programs.
- AI training data glossary: Internet-Draft, product token and other terms.
This page is information, not legal advice.
Frequently asked questions
Is __usage_ai-training an IETF standard?
No: on 2026-11-10 it is defined only in draft-kelley-aipref-training-preference-signal-00, an individual Internet-Draft published 2026-10-30, expiring 2027-05-03, with no stream and no IETF endorsement.
Does a User-agent: * rule express a no-training preference under the draft?
No: the draft states that wildcard groups "MUST NOT be interpreted as matching the policy token", so only a group naming __usage_ai-training expresses the preference.
When will Google read the shared AI training token?
Google says it aims to support the shared control "in 2027" (Search Central blog, 2026-10-30); its crawler documentation says "early 2027", so Google states no support before then.
Does Microsoft support the shared AI training opt-out?
No Microsoft statement of support is on record as of 2026-11-10; the token draft thanks Microsoft-affiliated participants, but it lists affiliations "for identification only".
What is the difference between train-ai and __usage_ai-training?
train-ai is the working group's vocabulary label used in Content-Usage rules, while __usage_ai-training is the individual draft's robots.txt policy token; both use the same AI Training definition.