
It is one control in the crawling route of AI training data licensing and acquisition, and each signal works only for the developers that read it.
This page describes each signal by its issuer's or operator's own text, lists them alphabetically and recommends none.
TDMRep is a W3C Community Group Final Report of 2024-05-10 and not a W3C Standard, the C2PA specification removed its training assertion in January 2024, and no AI developer on record states that it reads TDMRep, ai.txt or the CAWG assertion.
Amazon and Apple give page meta tags different meanings, and Anthropic recognizes a general no-AI-training signal from 2026-10-30.
Every operator statement carries its check date, and how we source, date and check every figure sets out the method.
What a machine-readable TDM reservation declares
A reservation declares that the holder does not permit text and data mining of the content, and machine-readable means that software, not a person, can detect it.
The EU text and data mining exception for commercial use applies unless use of the works has "not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online" (Article 4(3), Directive (EU) 2019/790, in force 2019-06-06).
Reservations come in two forms. The natural-language form sits in terms of service, and Longpre et al. report that 45 % of C4 was restricted that way, against 5 % or more of C4 tokens fully restricted by robots.txt (2023 to 2024, study published 2024-07-20). robots.txt is a protocol under RFC 9309 and is not legally binding by itself.
The hub on AI crawlers and how to allow, block or charge them covers the crawlers that read or ignore these signals.
Reservation signals side by side: where each sits and what it declares
The signals differ in where they sit (a site file, a page tag, an asset or a robots.txt line) and in whether they refuse mining, attach license terms or only state a preference. The table compares nine signals as of each row's Checked date, with the issuer's or operator's own text.
- "Not recorded" means no statement on record on the check date, not proof that nobody reads the signal.
- "Status" is the issuer's own label.
- A robots.txt token reserves only for the operator that documents it.
| Signal | Where it sits | What it declares (own text) | Status | Checked | Source |
|---|---|---|---|---|---|
__usage_ai-training |
robots.txt policy token, as Google describes the shared token | a no-AI-training preference | individual Internet-Draft of 2026-10-30, expires 2027-05-03, not IETF-endorsed | 2026-10-30 | datatracker.ietf.org |
| ai.txt | not recorded | not recorded | no specification on record | 2026-11-10 | none on record |
| C2PA Training and Data Mining assertion | asset | removed | removed in C2PA 2.0, January 2024 | 2026-11-09 | spec.c2pa.org |
| CAWG Training and Data Mining Assertion | asset, label cawg.training-mining | asset-level permission for training and mining | version 1.1, ratified by DIF 2025-05-16 | 2026-11-09 | cawg.io |
| nosnippet (Apple) | page meta tag | excludes the page from use as additional context in AI output; not a training opt-out | operator documentation | 2026-09-04 | support.apple.com/en-us/119829 |
| noarchive (Amazon) | page meta tag | "do not use the page for model training" | operator documentation | 2026-10-26 | developer.amazon.com/amazonbot |
| robots.txt disallow for a training token (Amazonbot, Applebot-Extended, ClaudeBot, Google-Extended, GPTBot, Meta-ExternalAgent) | robots.txt, one group per token | each operator's own statement on training use | RFC 9309 protocol, not legally binding by itself | 2026-09-04 to 2026-11-09 | operator pages; rfc-editor.org |
| RSL License line | robots.txt or an RSL file | points to license terms; the paid license permits ai-train, the attribution license permits ai-train and ai-input | open standard; licenses by contract | 2026-11-09 | rslcollective.org/license |
| TDMRep (TDM Reservation Protocol) | not described here | a text and data mining reservation in machine-readable form | W3C Community Group Final Report 2024-05-10, not a W3C Standard | 2026-11-09 | w3.org/community/reports/tdmrep |
TDMRep: the W3C Community Group protocol and its status
TDMRep, the TDM Reservation Protocol, is a W3C Community Group Final Report of 2024-05-10 for declaring a text and data mining reservation in machine-readable form, and it is not a W3C Standard.
A Community Group report is published by a W3C community group, not adopted by W3C members, so calling it a "recommendation" or a W3C standard is wrong.
This page does not describe the protocol's mechanics, because the primary report is the place to read them. No AI developer statement on reading TDMRep is recorded as of 2026-11-09, and no adoption figure is stated here.
Training tokens in a disallow rule as a reservation signal
A disallow rule naming a training crawler or a training control token works as a reservation for that one operator, because the operator states what the rule means for training. Amazon says Amazonbot "may be used to train Amazon AI models" and honors allow and disallow (2026-10-26).
Apple says Applebot-Extended does not crawl and that disallowing it opts content out of Apple foundation model training (2026-09-04). Google says Google-Extended covers Gemini training and grounding without any effect on Search inclusion, and OpenAI says disallowing GPTBot signals content "should not be used in training generative AI foundation models" (both 2026-11-09).
Meta says Meta-ExternalAgent is blocked by a disallow rule, and Anthropic says a ClaudeBot rule takes precedence over its general signal (both 2026-11-09). For token syntax, cache times and the fetchers a rule cannot reach, see robots.txt rules for AI crawlers, their limits and examples.
ai.txt: what is on record and what is not
ai.txt is a name site owners use for a file that states AI-use preferences, but no specification text, publisher or AI developer statement about it is on record here as of 2026-11-10. No operator in the readers table below states that it reads ai.txt.
A holder who wants a documented effect therefore uses a signal an operator documents, as the signals table above lists them.
Page-level meta tags: noarchive, nosnippet and what each operator says they mean
Only two AI developers on record give a page-level meta tag an AI-use meaning, and they give different tags different meanings. The two statements are:
- Amazon: noarchive means "do not use the page for model training" (checked 2026-10-26).
- Apple: nosnippet excludes the page from use as additional context in AI output (published 2026-09-04); Apple's training opt-out is the Applebot-Extended token, not nosnippet.
No generic AI meta value is on record as recognized by any issuer or operator, so none is described here as a signal.
Declaring license terms instead of a refusal: the RSL License line
An RSL License line does not reserve content from mining; it points crawlers to license terms, and the line blocks nothing by itself.
The RSL Collective states that a robots.txt or RSL file can carry a License: line pointing to its royalty license or its attribution license (checked 2026-11-09). The royalty license permits ai-train, the attribution license permits ai-train and ai-input, and rates are not published.
The RSL Collective is an intermediary and not a buyer, and payment depends on crawler operators choosing to license. Metering and charging are covered under pay-per-crawl and crawler licensing through RSL.
Asset-level assertions: from the C2PA training assertion to cawg.training-mining
Signals can travel with an image or file instead of a website: the CAWG Training and Data Mining Assertion records an asset-level permission for training and mining, after the C2PA specification removed its own assertion. The two changes are:
- C2PA: the Training and Data Mining assertion was removed in version 2.0 (January 2024).
- CAWG: version 1.1 of the Training and Data Mining Assertion was ratified by DIF on 2025-05-16, and carries the label cawg.training-mining.
No AI developer statement on reading either assertion is recorded. For documentation standards that record permissions beside datasets, see training data provenance and its published standards.
A general no-AI-training preference token, still an individual draft
A single token meant to express a no-AI-training preference to every developer, __usage_ai-training, exists only as an individual Internet-Draft published 2026-10-30. The draft expires 2027-05-03 and the IETF has not endorsed it, so it is not a standard.
Google states in a blog post of 2026-10-30 that it aims to support the control "in 2027", and its crawler documentation says "early 2027". Google also states that a shared-token opt-out without a Google-Extended rule is to be treated as a Google-Extended disallow.
Which AI developers state that they read a reservation
Developers on record state that they honor their own robots.txt tokens, two give meta tags a meaning, one recognizes a general no-AI-training signal and one plans to, while none on record states that it reads TDMRep, ai.txt or the CAWG assertion.
In the EU, providers of general-purpose AI models must identify and comply with Article 4(3) reservations "including through state-of-the-art technologies" (Article 53(1)(c), Regulation (EU) 2024/1689, applies from 2025-08-02), as set out under the copyright policy under EU AI Act Article 53.
The voluntary GPAI Code of Practice (copyright chapter, published 2025-07-10) names robots.txt under RFC 9309 in Measure 1.3 and asks signatories to comply with other appropriate machine-readable protocols for Article 4(3) reservations. The table compares seven developers' statements as of each row's Checked date; "none on record" never means that a developer ignores a signal.
| Developer | Signal it states it reads | Its statement (claim) | Signals with no statement on record | Checked | Source |
|---|---|---|---|---|---|
| Amazon | Amazonbot disallow; noarchive | Amazonbot honors allow and disallow; noarchive means no model training | TDMRep, ai.txt, CAWG | 2026-10-26 | developer.amazon.com/amazonbot |
| Anthropic | ClaudeBot rule; a general machine-readable no-AI-training signal from 2026-10-30, token not named | a ClaudeBot rule takes precedence over the general signal | TDMRep, ai.txt, CAWG | 2026-11-09 | privacy.claude.com |
| Apple | Applebot-Extended token; nosnippet | Applebot-Extended opts out of Apple foundation model training; nosnippet only excludes AI-output context | TDMRep, ai.txt, CAWG | 2026-09-04 | support.apple.com/en-us/119829 |
| Google-Extended token; shared token planned | Google-Extended covers Gemini training and grounding; shared token support "in 2027" (blog) or "early 2027" (documentation) | TDMRep, ai.txt, CAWG | 2026-10-30 to 2026-11-09 | developers.google.com | |
| Meta | Meta-ExternalAgent disallow | blocked by a disallow rule | TDMRep, ai.txt, CAWG | 2026-11-09 | developers.facebook.com |
| OpenAI | GPTBot disallow | content "should not be used in training generative AI foundation models" | TDMRep, ai.txt, CAWG | 2026-11-09 | developers.openai.com/api/docs/bots |
| SpaceXAI LLC (formerly xAI LLC) | "instructions that crawled content should not be used for training", form not named | stated in its Grok 4.7 training-content summary; states it is not a GPAI Code copyright signatory | named signals not recorded | 2026-09-21 | media.x.ai Grok 4.7 summary |
Where the form of a reservation has legal weight
In the EU a reservation of online content takes it out of the commercial text and data mining exception only when made in an appropriate manner, such as machine-readable means (Article 4(3), Directive (EU) 2019/790).
National wording differs: Germany's § 44b(3) UrhG makes a reservation for online works effective only in machine-readable form, while Sweden's 15 a § says "på lämpligt sätt" (in an appropriate manner) and has no machine-readable wording (both checked 2026-11-09).
Whether a natural-language reservation counts is before the German courts in Kneschke v. LAION (OLG Hamburg 5 U 104/24 and BGH I ZR 281/25, pronouncement scheduled for 2026-12-17), and both proceedings are covered on EU text and data mining exception and the opt-out.
Declaring a TDM reservation: benefits, risks, downsides and alternatives
Declaring a reservation costs little and can carry legal weight in the EU, but each signal reaches only the developers that state they read it, and none removes content already collected.
Benefits
- Cost: a robots.txt line or a page tag takes little effort to add.
- Documentation: operators document their own tokens, so the effect of a rule is stated by the party that reads it.
- EU function: a machine-readable reservation serves the purpose Article 4(3) names, and AI Act providers must identify such reservations.
Risks
- Silence: no developer on record states that it reads TDMRep or ai.txt.
- Meaning: the same tag means different things at different operators, as noarchive and nosnippet show.
- Drafts: the shared token is an individual draft that can change or lapse.
Downsides
- Prospective only: Common Crawl states that its WARC files are immutable and that affected URLs are filtered from later crawls (statement of 2025-11-04; Opt-Out Ledger published 2025-09-17).
- No payment: a refusal brings no income.
- Upkeep: several signals must be kept in step across operators.
Alternatives
- A robots.txt rule per operator token.
- An RSL License line pointing to license terms.
- An asset-level CAWG assertion.
- Licensing the content, as set out in the offer below.
- Doing nothing and not licensing at all, as weighed in whether to sell or license data to AI companies at all.
Crawler pages, licensing routes and the law beyond reservation signals
Reservation signals are one control among several; the pages below cover the crawlers, the draft shared token, licensing routes and the law.
- AI crawlers and how to allow, block or charge them: every AI crawler and control method in one place.
- The shared AI training opt-out: the draft shared token and who plans to read it.
- Google-Extended: opting out of Gemini training.
- Amazonbot and the noarchive training opt-out: Amazon's crawler and its meta tag.
- Applebot-Extended: Apple's training opt-out.
- Opting out of AI training through platform settings: content hosted where no site file is the holder's to edit.
- EU rules for AI training data: the reservation wording in all 27 member states.
- News publishers licensing content to AI companies: publishers that reserve content and license it.
- Collective licensing for AI training: collective licenses for reserved content.
- Is web scraping for AI training legal: copyright, contract and computer-misuse law on scraping.
This page is information, not legal advice.
Frequently asked questions
Is TDMRep a W3C standard?
No: TDMRep is a W3C Community Group Final Report of 2024-05-10, and the report itself states it is not a W3C Standard. It is published by a W3C community group and was not adopted by W3C members.
Does the noarchive tag stop AI training?
For Amazon, yes by its own statement: Amazon says noarchive means "do not use the page for model training" (checked 2026-10-26). No other developer on record gives noarchive that meaning.
Is the C2PA training and data mining assertion still in the specification?
No: the C2PA specification removed its Training and Data Mining assertion in version 2.0 (January 2024). The CAWG Training and Data Mining Assertion 1.1, label cawg.training-mining, now covers asset-level permissions.
Do AI developers read ai.txt?
No AI developer statement about reading ai.txt is on record as of 2026-11-10. The documented signals are robots.txt tokens, Amazon's noarchive tag and the general signal Anthropic recognizes.