DataPacts

The Shared AI Training Opt-out (__usage_ai-training): IETF AIPREF, Google, Anthropic and Microsoft

Checked

At a glance

The shared AI training opt-out is a single robots.txt token, __usage_ai-training, meant to tell every participating AI developer that a site prefers its content not be used for AI training, and on 2026-11-10 it exists only in an individual Internet-Draft.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

The __usage_ai-training token sits in an individual draft; the vocabulary and attachment drafts are working group drafts; none has reached IESG review or become an RFC. Google aims for 2027, Anthropic recognizes a signal.

It is one control in the crawling route of AI training data licensing and acquisition, and it works only for developers that read it.

Google says it aims to support the shared control in 2027, Anthropic says it has recognized a machine-readable no-AI-training signal since 2026-10-30, and no Microsoft or OpenAI statement is on record on their own pages.

This page describes each IETF document by its own text and datatracker status, each developer by its own dated statement, and recommends no signal.

Every statement carries its check date, and how we source, date and check every figure sets out the method.

What a shared AI training preference signal is meant to do

A shared signal replaces one rule per crawler with one statement of preference that any developer adopting the signal reads the same way.

The token draft ties the token to the working group's "AI Training" definition and states that the token "is not a directive for crawling behavior", so it records a preference and does not block anything.

robots.txt itself is a protocol under RFC 9309 and is not legally binding by itself. Today each developer documents its own token, such as Google-Extended, GPTBot and ClaudeBot, and the hub on AI crawlers and how to allow, block or charge them lists them.

The IETF AI Preferences working group: charter and limits

The IETF AI Preferences working group (aipref, Web and Internet Transport area) is chartered to "standardize building blocks that allow for the expression of preferences about how content is collected and processed".

The datatracker shows the group as Active on 2026-11-10 with charter-ietf-aipref-01 approved, and its deliverables are a standards-track vocabulary, standards-track attachment documents and a method for reconciling multiple expressions of preference.

The charter places four things out of scope:

  • technical enforcement of preferences
  • application-layer authentication or authorization of clients or crawlers
  • registries of preferences
  • auditing and other transparency measures for AI training

The working group therefore defines how a preference is stated and attached, and nothing in it makes a crawler obey.

The AIPREF documents and their status on 2026-11-10

Two documents belong to the working group, the vocabulary and the attachment draft, and neither had reached the IESG on 2026-11-10, while the __usage_ai-training token sits in a separate individual draft. The table compares the eight documents on the datatracker's working group documents page as of 2026-11-10, with the datatracker's own status terms.

Document Title Revision and date Kind and status on 2026-11-10 Expires Source
draft-ietf-aipref-vocab A Vocabulary For Expressing AI Usage Preferences -08, 2026-09-13 WG document, intended Proposed Standard; no IESG state; Note to Readers: no WG consensus 2027-03-18 datatracker.ietf.org
draft-ietf-aipref-attach Associating AI Usage Preferences with Content in HTTP -05, 2026-08-18 WG document, intended Proposed Standard; would update RFC 9309; no IESG state 2027-02-20 datatracker.ietf.org
draft-kelley-aipref-training-preference-signal Machine-Readable Training Preference Signal -00, 2026-10-30 individual draft, no stream, not IETF-endorsed; header "Intended status: Standards Track"; defines __usage_ai-training 2027-05-03 datatracker.ietf.org
draft-altanai-aipref-realtime-protocol-bindings AI Preferences for Real-Time Protocol Bindings -01, 2026-08-04 related individual draft not recorded datatracker.ietf.org
draft-hood-aipref-earmark Earmark: Embedded Attribution and Rights Marks for AI Usage Preferences -00, 2026-08-12 related individual draft not recorded datatracker.ietf.org
draft-reilly-aipref-compliance Verifiable Compliance Records for AI Usage Preferences -00, 2026-08-01 related individual draft not recorded datatracker.ietf.org
draft-wallace-aipref-grant-binding A Verifiable-Credential Binding for AI Usage Preferences: Expressing Grants that Lift AIPREF Preferences -03, 2026-10-27 related individual draft not recorded datatracker.ietf.org
draft-zehta-aipref-parameters AIPREF Vocabulary Parameters -00, 2026-08-06 related individual draft not recorded datatracker.ietf.org

An individual draft carries no IETF endorsement, and an Internet-Draft is not a standard. The six related drafts are none of them working group documents, and the datatracker lists each as "I-D Exists".

The vocabulary draft defines three usage categories with short labels: AI Training (train-ai), AI Use (ai-use) and Search (search). Its definitions read as follows:

  • train-ai: "Using an asset to modify the learned parameters of a generative AI model", with training of exclusively non-generative models excluded (section 4.1).
  • ai-use: "Using an asset as input to a generative AI model, where the asset is not directly provided by the user", and it excludes AI Training (section 4.2).
  • search: selecting assets and directing users to their location, where allowing it includes training models whose models and outputs are used exclusively for the search application (section 4.3).

The draft's Note to Readers states that its contents "DO NOT REFLECT CONSENSUS of the Working Group either in whole or part". The suffix "ai-training" belongs to the token draft only and is not a vocabulary label, and the token draft quotes the same AI Training definition.

Two ways to put a training preference in robots.txt: Content-Usage and the policy token

The working group's attachment draft adds a Content-Usage rule and an HTTP Content-Usage header, while the individual draft keys an ordinary robots.txt group to the __usage_ai-training policy token. The two mechanisms differ as follows:

  • Attachment draft (-05): updates RFC 9309 if approved, and its examples are Content-Usage: train-ai=n in a robots.txt group and as an HTTP response header.
  • Token draft (-00): "introduces no new directive names and no new grammar" and uses a User-Agent line naming the token.

The token draft's own example is User-Agent: __usage_ai-training followed by Disallow: / and Allow: /press-kit/, a preference against AI training except for the press kit.

That is the draft's example and not working syntax for any developer except as the developers' statements below say. Group syntax, cache times and fetchers are covered in robots.txt rules for AI crawlers, their limits and examples.

How the draft says the __usage_ai-training token is read

Under the draft, a developer reads the token like a product token, and only a group that names it expresses a preference. The draft sets these rules:

  • The match is exact apart from case.
  • Wildcard groups (User-Agent: *) "MUST NOT be interpreted as matching the policy token", and with no group naming the token "no preference has been expressed".
  • Crawler rules for a developer's own product token are read "exactly as they would in the absence of this document".
  • "Specific expressions SHOULD take precedence over general ones", developers MAY document proprietary tokens as more specific, and "Private agreements MAY override preferences expressed via this mechanism".

These are a draft's rules, binding on nobody, and a site's User-Agent: * rule therefore says nothing about training under the draft.

Who states support for the shared token, in their own words

Google says it aims to support the shared control in 2027, Anthropic says it has recognized a machine-readable no-AI-training signal since 2026-10-30, and no statement by Microsoft or OpenAI is on record on their own pages as of 2026-11-10. Google also states that, for a site that opts out through the shared token without a Google-Extended rule, it plans to read that as a Google-Extended disallow, which Google-Extended and opting out of Gemini training covers in depth.

The table lists developers alphabetically with each statement as a dated claim as of each row's Checked date. "None on record" means no statement was found on the named page, not that the developer refuses or ignores the signal.

Developer What it states about the shared signal (claim) Token named on its page Its own training control Checked Source
Amazon none on record n/a Amazonbot disallow; noarchive 2026-11-10 developer.amazon.com/amazonbot
Anthropic "As of October 30, 2026, Anthropic has begun recognizing a machine-readable signal"; rollout "over time and globally"; a ClaudeBot rule takes precedence no ClaudeBot rule 2026-11-10 privacy.claude.com
Apple none on record n/a Applebot-Extended token 2026-11-10 support.apple.com/en-us/119829
Google "We aim to support this new control along with other AI providers in 2027" (blog); "early 2027" (documentation); shared-token opt-out without a Google-Extended rule read as a Google-Extended disallow blog no; documentation "user-agent: __usage_ai-training" Google-Extended 2026-11-10 developers.google.com
Meta none on record n/a Meta-ExternalAgent disallow 2026-11-10 developers.facebook.com
Microsoft none on record on Bing Webmaster pages n/a not recorded 2026-11-10 blogs.bing.com/webmaster
OpenAI none on its crawler page no GPTBot disallow 2026-11-10 developers.openai.com/api/docs/bots

The token draft's acknowledgments thank participants affiliated with Microsoft, Google, OpenAI, Amazon Web Services and others, and state that "Affiliations are listed for identification only, since IETF participants contribute as individuals". That is no company's statement of support.

Where Anthropic's description and the datatracker record differ

Anthropic describes its signal as following a draft based on IETF work in the aipref working group, while the datatracker lists the only draft defining __usage_ai-training as an individual draft with no stream.

Anthropic's page does not name the token, so this page does not write that Anthropic named it, and the datatracker gives the author's affiliation as Anthropic.

A draft adopted by the working group would carry a draft-ietf-aipref name, which this one does not. The reader can keep both statements side by side, and Anthropic's crawling, licensing and program terms are on Anthropic as a data buyer and its program terms.

Until the datatracker shows an adoption, this page writes the draft as individual.

What the shared token leaves unchanged: crawler rules, collected data and the law

Adding the token changes nothing that a site's existing crawler rules already say, removes nothing already collected and has no legal status of its own. A ClaudeBot rule takes precedence at Anthropic, and Google's stated fallback applies only where a site has no Google-Extended rule.

Other reservation signals such as TDMRep, page meta tags and asset assertions work differently, and machine-readable TDM reservations such as TDMRep and meta tags lists them.

Common Crawl stated on 2025-11-04 that its WARC files are immutable and that affected URLs are removed or filtered from later crawls, and it published its Opt-Out Ledger on 2025-09-17. Three limits apply:

  • No court or authority ruling treating the token as a reservation under Article 4(3) is on record.
  • robots.txt, a protocol under RFC 9309, is not legally binding by itself.
  • No measure of how many sites use the token or Content-Usage is on record.

In the EU, use of works for text and data mining is excepted unless expressly reserved by rightholders "in an appropriate manner, such as machine-readable means in the case of content made publicly available online" (Article 4(3), Directive (EU) 2019/790), and general-purpose AI providers must identify such reservations "including through state-of-the-art technologies" (Article 53(1)(c), Regulation (EU) 2024/1689, applicable from 2025-08-02).

The EU text and data mining exception and the opt-out covers Article 4(3) and the courts.

Adding the shared token: benefits, risks, downsides and alternatives

Adding a __usage_ai-training group costs one robots.txt group, but on 2026-11-10 only Anthropic states it recognizes a general signal and Google plans to read the token in 2027. The trade-offs are these.

Benefits:

  • One group addresses every developer that adopts the signal, since the token draft defines no parallel vocabulary.
  • Google states a planned fallback to its Google-Extended disallow.
  • Anthropic states recognition from 2026-10-30.

Risks:

  • The draft may change or expire on 2027-05-03.
  • The vocabulary draft does not reflect working group consensus.
  • A User-Agent: * rule does not count as the preference under the draft.

Downsides:

  • The signal is prospective, as Common Crawl's practice of keeping collected files shows.
  • A preference brings no payment.
  • Operator tokens are still needed for developers with no statement on record.

Alternatives:

  • Operator-specific tokens such as Google-Extended, GPTBot and Applebot-Extended.
  • The Content-Usage rule once the working group's attachment draft is adopted.
  • Other reservation signals, such as TDMRep and page meta tags.
  • Licensing the content to AI developers, which is a different route.
  • Doing nothing and not licensing at all, which is covered by whether to sell or license data to AI companies at all.

The choice depends on which developers a site wants to reach and on whether the site wants a preference, a block or a license.

Crawler pages, other signals and the law around the shared token

The shared token is one control among several; the pages below cover the crawlers that may read it, the other signals and the law.

This page is information, not legal advice.

Frequently asked questions

Is __usage_ai-training an IETF standard?

No: on 2026-11-10 it is defined only in draft-kelley-aipref-training-preference-signal-00, an individual Internet-Draft published 2026-10-30, expiring 2027-05-03, with no stream and no IETF endorsement.

Does a User-agent: * rule express a no-training preference under the draft?

No: the draft states that wildcard groups "MUST NOT be interpreted as matching the policy token", so only a group naming __usage_ai-training expresses the preference.

When will Google read the shared AI training token?

Google says it aims to support the shared control "in 2027" (Search Central blog, 2026-10-30); its crawler documentation says "early 2027", so Google states no support before then.

Does Microsoft support the shared AI training opt-out?

No Microsoft statement of support is on record as of 2026-11-10; the token draft thanks Microsoft-affiliated participants, but it lists affiliations "for identification only".

What is the difference between train-ai and __usage_ai-training?

train-ai is the working group's vocabulary label used in Content-Usage rules, while __usage_ai-training is the individual draft's robots.txt policy token; both use the same AI Training definition.