robots.txt for AI Crawlers: Rules, Limits and Examples

robots.txt is the plain-text file at the root of a host that tells named crawlers, including AI training crawlers, which URLs they should not fetch. It is the control a site owner reaches for first in the crawling route of AI training data licensing and acquisition, and the protocol, specified in RFC 9309 (2022), is not legally binding by itself.

This page lists 20 tokens alphabetically with what each operator says a full disallow changes (checked 2026-11-09), shows example rules, and sets out cache times, server-error handling, fallbacks and legal weight. Amazon states a cache of up to 30 days, a 5xx response means complete disallow under RFC 9309, four operators say their user-initiated fetchers may not apply the rules, and a rule works forward only. In the EU a disallow can serve as a machine-readable reservation. No rule set is recommended.

Every operator statement below names its operator and a check date, and how we source, date and check every figure sets out the method.

What robots.txt does and does not do for AI crawlers

robots.txt asks crawlers that follow RFC 9309 not to fetch named paths, and it works only for bots that choose to read it and are named correctly. A rule belongs to a User-agent line, so each operator's product token needs its own group, and a bot that is not named falls under whatever general group the file holds.

Two names in the file never crawl at all. Google-Extended and Applebot-Extended are control tokens: Google and Apple read them to decide whether content collected by their other crawlers may train Gemini or Apple foundation models. The hub page on AI crawlers and how to allow, block or charge them holds the full bot list and every other method of control.

• Controls: future fetching by a named bot.
• Signals: training use, where an operator ties a token to training, as OpenAI does for GPTBot.
• Is not: access control, or a legal instrument by itself.

The AI user-agent tokens a robots.txt rule can name

A rule works only when it names the token the operator documents, and 20 tokens are recorded here with what a full disallow changes, as each operator states it. The table is as of 2026-11-09 unless the Checked cell says otherwise.

• Kind uses four terms: training crawler, search crawler, user-initiated fetcher and control token.
• "Not recorded" means no statement on the check date, not proof of absence.
• Bytespider is attributed to ByteDance, and the operating entity is not named.

robots.txt examples: blocking training while keeping search, and other patterns

The common pattern disallows each training crawler and training control token by name while leaving search crawlers untouched. Cloudflare states that 17 % of its sites restrict AI training while fewer than 1 % block search crawlers (2026-09-15), which shows how often site owners treat the two uses separately.

The table below compares six patterns of whole-site rules, each token in its own User-agent group, as of 2026-11-09. Each row shows what the operator's own statement says the lines do and what they leave alone, which is how a rule to block AI bots such as these is read in practice.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

The License line is an RSL Collective feature: a publisher adds it to robots.txt or an RSL file to point to a paid license (royalty terms) or an attribution license. RSL Collective is an intermediary, not a buyer, and the line does nothing unless a crawler operator chooses to license.

None of these examples stops three things:

• User-initiated fetchers, whose operators say rules may not apply.
• Unnamed or spoofed bots, which a file cannot tell apart from the real ones.
• Content already collected, which a rule cannot recall.

How operators read the file: cache time, server errors, fallbacks and precedence

Two sites with the same rules can see different results, because operators cache the file for different periods and resolve missing or conflicting rules differently. RFC 9309 says a crawler should not use a cached copy for more than 24 hours unless the file is unreachable, Amazon states a cache of up to 30 days (checked 2026-10-26), and OpenAI says OAI-SearchBot changes take about 24 hours. Crawlers must parse at least 500 KiB, so a file kept below that size is read in full.

The table below compares ten situations by who states the handling, with each row's check date.

Fallbacks and precedence decide what happens when the file is silent or conflicting. Apple states that Applebot follows the Googlebot rules when robots.txt mentions Googlebot but not Applebot. Anthropic states that it has recognized a general machine-readable no-AI-training signal since 2026-10-30 and that a rule naming ClaudeBot takes precedence over it. Google plans to treat an opt-out through the shared token as a Google-Extended disallow when Google-Extended is not named. A server error is the harshest case: a crawler that gets a 5xx for the file must assume complete disallow.

Do AI crawlers respect robots.txt? Operator statements and measured bypasses

Operators of the main training crawlers state that those crawlers follow robots.txt, but four operators say their user-initiated fetchers may not, and network operators have measured bypasses. Anthropic and Common Crawl state that their crawlers follow the file, OpenAI states that a GPTBot disallow signals content should not be used for training, and the four fetcher statements read as follows (all checked 2026-11-09 unless dated):

• OpenAI on ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply".
• Meta on Meta-ExternalFetcher: it "may bypass robots.txt rules".
• Amazon on Amzn-User (2026-10-26): it "may not follow all robots.txt directives".
• Perplexity on Perplexity-User: "this fetcher generally ignores robots.txt rules".

Measurements differ from statements. Cloudflare observed ChatGPT-User stopping when disallowed (2025-08-04), and the same day Cloudflare reported that Perplexity's declared crawler, labeled Perplexity-User/1.0, made 20 to 25 million requests per day while an undeclared crawler made 3 to 6 million, and de-listed Perplexity as a verified bot. TollBit states that its network recorded 1.9 billion or more robots.txt bypasses and 22 billion or more AI bot scrapes in the first half of 2026.

Spoofing is the other gap: Common Crawl writes that it is "aware of crawlers falsely identifying themselves as CCBot". Operators publish checks for this, among them openai.com/gptbot.json, claude.com/crawling/bots.json, index.commoncrawl.org/ccbot.json and Apple reverse DNS (*.applebot.apple.com). The full method for checking who visits a site stays with the section hub.

Where a robots.txt rule carries legal weight

In the EU a robots.txt disallow can serve as the machine-readable reservation that takes online content out of the commercial text and data mining exception, and no US statute or ruling is recorded here that makes robots.txt binding. Article 4(3) of Directive (EU) 2019/790 (in force 2019-06-06) allows the exception only where use has not been reserved "in an appropriate manner, such as machine-readable means in the case of content made publicly available online". The EU text and data mining exception and its opt-out have their own page.

Two EU instruments point at the same file. Regulation (EU) 2024/1689, Article 53(1)(c), applies from 2025-08-02 and requires providers of general-purpose AI models to keep a copyright policy that identifies and complies with Article 4(3) reservations "including through state-of-the-art technologies". The voluntary GPAI Code of Practice, Copyright chapter (published 2025-07-10), says in Measure 1.3 that signatories employ crawlers that read and follow robots.txt as specified in RFC 9309.

Courts have not settled the form of a reservation. Kneschke v. LAION, docket BGH I ZR 281/25 at the Federal Court of Justice, concerns a natural-language reservation, and as of 2026-11-09 the hearing took place on 2026-09-03 with pronouncement scheduled for 2026-12-17, so no decision exists.

Japan treats the file differently. Article 30-4 of the Copyright Act (Act No. 48 of 1970, current text in force 2019-01-01) permits exploitation not for enjoyment, including information analysis, except where it would "unreasonably prejudice the interests of the copyright owner". The non-binding General Understanding on AI and Copyright (English overview, May 2024) places circumventing robots.txt within that proviso. Article 30-4 and its proviso sit within the rules for licensing data for AI training in Japan.

What a Disallow rule cannot undo

A Disallow rule works forward only: it stops future fetches by compliant bots and does not remove pages already in archives, datasets or trained models. Common Crawl states (2025-11-04) that its WARC files are immutable and that affected URLs are removed or filtered from subsequent crawls and public indexes, and it published its full Opt-Out Ledger on 2025-09-17. No other operator's deletion practice is recorded here.

The scale of restriction is measurable. Longpre et al. audited about 14,000 web domains and found that 5 % or more of C4 tokens, and 28 % or more of the most critical sources, were fully restricted by robots.txt in 2023 to 2024, while 45 % of C4 was restricted by terms of service. Those figures describe sites that restricted access, not content that left the datasets.

Relying on robots.txt alone: benefits, risks, downsides and alternatives

robots.txt is free, public and read by the documented training crawlers, but on its own it neither enforces access nor earns anything. The lists below set out what a site gives and gets by relying on it.

Benefits:

• No cost, and a plain text file that any site owner can edit where the host allows it.
• OpenAI, Anthropic, Apple, Meta, Amazon and Common Crawl document how their training crawlers treat a disallow.
• In the EU, the file is a machine-readable means of reservation.

Risks:

• User-initiated fetchers may not apply the rules.
• Spoofed bots can pose as named crawlers.
• Long caches, up to 30 days at Amazon, delay the effect of a change.

Downsides:

• Control tokens and search crawlers must be named one by one.
• A disallow brings no payment.
• The rule is prospective only.

Alternatives:

1. Page-level tags: Amazon states that noarchive means "do not use the page for model training", and Apple states that nosnippet excludes a page from use as AI-output context (checked 2026-09-04).
2. The shared no-AI-training signal: __usage_ai-training is a token of an individual Internet-Draft published 2026-10-30 that expires 2027-05-03 and is not endorsed by the IETF.
3. TDMRep, a W3C Community Group Final Report of 2024-05-10 that is not a W3C Standard.
4. Network-level controls or metered access through intermediaries: Cloudflare blocks AI crawlers by default for new domains since 2025-07-01 and announced separate Search, AI training and AI agents controls on 2026-09-15, with Pay per crawl in closed beta.
5. Licensing content, or leaving it unlicensed, which turns on the question of whether to sell or license data to AI companies at all.

Doing nothing and not licensing at all is also a valid choice, and a site can combine several of these.

Other opt-out signals, bot pages and the law beyond robots.txt

robots.txt is one of several controls, and the pages below cover the others, the individual bots and the law around crawling.

• AI crawlers and how to allow, block or charge them: every AI crawler and control method in one place.
• Machine-readable TDM reservations: TDMRep and other machine-readable TDM reservations.
• The shared AI training opt-out: the draft shared AI training opt-out.
• Pay-per-crawl and crawler licensing: charging crawlers instead of blocking them, including RSL License lines.
• GPTBot and OpenAI's other crawlers: GPTBot and OpenAI's other crawlers in detail.
• Google-Extended: opting out of Gemini training.
• Opting out of AI training through platform settings: opt-outs for content hosted on platforms, where robots.txt is not the owner's to edit.
• Is web scraping for AI training legal: copyright, contract and computer-misuse law on scraping.
• EU AI Act Article 53 copyright policy: the copyright policy general-purpose AI providers must keep under Article 53.

This page is information, not legal advice.

Frequently asked questions

Does blocking GPTBot keep a site out of ChatGPT search?

No: OpenAI runs ChatGPT search through OAI-SearchBot, a separate token, so a GPTBot disallow leaves search crawling untouched (checked 2026-11-09). A site that wants out of both names both tokens.

How long does a robots.txt change take to reach AI crawlers?

RFC 9309 says a crawler should not use a cached copy for more than 24 hours, OpenAI says OAI-SearchBot changes take about 24 hours, and Amazon caches the file for up to 30 days. Other operators are not recorded here.

What happens when robots.txt returns a server error?

Under RFC 9309 a crawler that gets a 5xx server error for robots.txt must assume complete disallow. A site that is down for the file therefore blocks compliant crawlers.

Does Applebot follow rules written for Googlebot?

Yes, when robots.txt mentions Googlebot but not Applebot, Apple states Applebot follows the Googlebot rules (checked 2026-09-04). Naming Applebot in its own group removes that fallback.

Can AI crawlers ignore robots.txt without legal consequences?

robots.txt is not legally binding by itself, but in the EU a machine-readable reservation takes online content out of the commercial text and data mining exception, so ignoring it can matter under copyright law. No US rule is recorded here.