AI Crawlers: Which Bots Collect Training Data and How to Allow, Block or Charge Them

AI crawlers are automated bots that fetch web pages for AI developers, either to collect training data, to build an AI search index or to answer a user's request. Crawling is the route of AI training data licensing and acquisition in which no contract is signed, so the site owner's levers are technical signals, network controls and, where an intermediary supports it, a price.

The list below covers 20 bots and control tokens, 19 documented by their operators and Bytespider without operator documentation. Cloudflare states that nearly 80 % of AI bot crawling on its network from 2025-07-01 to 2025-07-28 was for training. A robots.txt rule stops future collection by bots that follow it, but OpenAI, Meta, Amazon and Perplexity state that their user-initiated fetchers may not apply it, and no block removes content already collected. In the EU, Art. 4(3) of the DSM Directive gives a machine-readable reservation legal effect. Charging runs through intermediaries: Cloudflare Pay per crawl (closed beta), TollBit and the RSL Collective. No bot is ranked and no option recommended.

Every operator statement and statistic names its stating party and date, as set out in how we source, date and check every figure.

What is an AI crawler?

An AI crawler is a bot that collects web content for an AI system, and its operator's stated purpose decides whether that content may train a model. Operators describe four kinds of bot and token:

1. Training crawlers collect content that may train models, such as OpenAI's GPTBot, Anthropic's ClaudeBot and Meta-ExternalAgent.
2. Search crawlers build the index behind AI search answers, such as OAI-SearchBot and PerplexityBot, which Perplexity states "is not used to crawl content for AI foundation models".
3. User-initiated fetchers load a page when a user asks an assistant to, such as ChatGPT-User, Claude-User and Perplexity-User.
4. Control tokens do not crawl at all: Google-Extended and Applebot-Extended are robots.txt names that decide whether content collected by Google's and Apple's crawlers may train Gemini or Apple foundation models.

The kinds overlap. Cloudflare states that mixed-use crawlers made up 36.6 % of verified crawler traffic on its network (press release 2026-09-15), and Meta describes Meta-ExternalAgent as crawling for "training foundation AI models or improving products by indexing content directly" (checked 2026-11-09). A bot's name says less than its operator's documentation.

The crawler pages at a glance

The crawler section has one page per method of control and one per major bot, listed here with what each covers. Its 13 pages, as of 2026-11-10, start with the four control methods and continue with nine single bots in alphabetical order.

AI crawler list: operators, purposes and robots.txt behavior

The list gives every AI crawler and control token whose operator documents it, plus Bytespider, with the operator's own statement on training use and robots.txt, dated. Its 20 rows run alphabetically and are as of 2026-11-09 unless the Checked cell gives an earlier date.

How to read the list:

• Kind uses the four kinds defined above; CCBot and OAI-AdsBot carry the purpose their operators describe instead.
• Training use quotes or paraphrases the operator; it is not DataPacts' judgment of what a bot does.
• "Not recorded" means no operator statement on the check date, not proof that none exists.
• Bytespider is attributed to ByteDance, but the operating entity's identity is not confirmed.

Training crawlers, search crawlers and user-initiated fetchers: why the kind decides what a block does

Blocking a training crawler stops future collection for training by that operator, but it does not stop the same operator's search crawler or a fetch a user triggers. A disallow rule naming GPTBot does not name OAI-SearchBot, so ChatGPT search crawling continues (checked 2026-11-09). Control tokens work the other way round: Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal", and Apple states that pages opted out through Applebot-Extended stay eligible for Search.

User-initiated fetchers sit outside robots.txt by their operators' own account. OpenAI writes of ChatGPT-User that "because these actions are initiated by a user, robots.txt rules may not apply"; Meta states that Meta-ExternalFetcher "may bypass robots.txt rules"; Amazon states that Amzn-User "may not follow all robots.txt directives"; and Perplexity states that Perplexity-User "generally ignores robots.txt rules". Amazon and Perplexity state that these fetchers are not used for training. Cloudflare reported on 2025-08-04 that it saw ChatGPT-User stop when a site disallowed it; that is Cloudflare's measurement, not a commitment by OpenAI.

A block withholds content from crawling, not from the companies that run the crawlers. Google states that its Reddit agreement of 2024-02-22 grants access to "display, train on, and otherwise use" Reddit content and does not change its use of publicly crawlable content. How the labs license archives and platforms is compared under frontier AI labs as data buyers.

Ways to allow or block AI crawlers: rules, page tags, signals and network controls

A site controls AI crawlers at four layers: robots.txt rules per user agent, page-level tags, general opt-out signals and network-level blocking. The robots.txt protocol, RFC 9309, says a crawler SHOULD NOT use a cached copy for more than 24 hours, must parse at least 500 KiB and must assume complete disallow when the file returns a server error (5xx); the RFC describes a protocol and does not make robots.txt legally binding by itself. Amazon states that it caches robots.txt for up to 30 days (checked 2026-10-26), and OpenAI states that robots.txt changes reach OAI-SearchBot in about 24 hours.

The table compares the four layers and TDM reservation protocols as of 2026-11-09, with who states they honor each.

Page tags and general signals depend on each operator's adoption. The __usage_ai-training token comes from an individual Internet-Draft published 2026-10-30, which expires 2027-05-03 and is not endorsed by the IETF. Google says it aims to support a shared AIPREF-based training control with other AI providers in 2027 (its crawler documentation says early 2027), and it plans to treat an opt-out through that control without a Google-Extended rule as disallowing Google-Extended. Anthropic states that it has recognized a machine-readable no-AI-training signal since 2026-10-30, without naming a token, and that a robots.txt rule naming ClaudeBot takes precedence. At the network layer, Cloudflare has blocked AI crawlers by default for new domains since 2025-07-01.

Spoofed and undeclared bots: checking which AI crawlers visit a site

A user-agent string proves nothing on its own, because any bot can send one, so operators publish IP lists or reverse-DNS names that confirm their crawlers. Common Crawl states that "we are aware of crawlers falsely identifying themselves as CCBot" (checked 2026-11-09). A site can run four checks on its server logs:

1. User agent: the string each request sends, such as OpenAI's example Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot.
2. Published IP lists: openai.com/gptbot.json, claude.com/crawling/bots.json and index.commoncrawl.org/ccbot.json.
3. Reverse DNS: *.applebot.apple.com for Applebot and *.crawl.commoncrawl.org for CCBot (IPv4 only).
4. Behavior: CCBot and ClaudeBot honor Crawl-delay, while Applebot and Amazon's crawlers do not, so ignoring it proves nothing for those bots.

Cloudflare reported on 2025-08-04 that Perplexity's declared crawler, listed as Perplexity-User, made 20 to 25 million requests a day and its undeclared crawler 3 to 6 million, and Cloudflare de-listed Perplexity as a verified bot. TollBit states that its network recorded more than 1.9 billion robots.txt bypasses in the first half of 2026. For Bytespider, no operator documentation or verification method was found on 2026-11-09, so a site has no operator list to check its requests against.

How much AI crawlers collect and how many sites restrict them

Most published figures on AI crawling come from network operators measuring their own customers, so each is a claim with a period, not a census of the web. Cloudflare, the source of most rows, states that it manages and protects traffic for 20 % of the web (press release 2025-07-01), and its restriction shares are one supply signal in the AI training data market.

The table lists each figure for its own period with its stating party; no rows are combined.

Two rows need care. Cloudflare's 2025 Year in Review states user-action crawling growth as "over 15x" in its headline and "up over 21x" in its body, and both are shown because the report does not reconcile them. The two Anthropic ratios cover different weeks. Beyond robots.txt, Longpre et al. found that 45 % of the C4 corpus was restricted by terms of service in 2023 to 2024.

Allow, block or charge: benefits, risks, downsides and alternatives for a site's content

Allowing AI crawlers keeps content available to AI products at no charge, blocking them withholds it from future training, and charging turns access into a metered license where an intermediary supports it. Charging needs a party that meters requests, and Cloudflare, TollBit and the RSL Collective each offer a route without acquiring the content themselves. Some publishers combine metering with AI data licensing deals: Reach plc's half-year report of 2026-07-22 names TollBit pay-per-use beside deals with large tech firms such as AWS.

The table compares the three charging routes on record as of 2026-11-09; each is an intermediary, not a buyer.

Benefits of allowing

• Search visibility: search crawlers such as OAI-SearchBot and Claude-SearchBot keep the site in AI search features.
• No upkeep: no list of user agents to maintain as operators add bots.
• One crawl instead of two: OpenAI states it may serve search and training from one crawl when both bots are allowed.

Risks of allowing

• Training without payment: GPTBot, ClaudeBot, Meta-ExternalAgent and Amazonbot collect content their operators say may train models.
• Little traffic in return: the crawl-to-refer ratios in the statistics table.
• Lasting copies: Common Crawl keeps its WARC archive files immutable.

Downsides of blocking or charging

• Forward only: a block removes nothing already collected; Common Crawl filters affected URLs only from later crawls.
• Partial reach: user-initiated fetchers may not apply robots.txt, spoofed bots ignore it, and Amazon may act on a cached file for up to 30 days.
• Untested prices: Cloudflare Pay per crawl is a closed beta without stated fees or payout threshold, and the RSL Collective does not publish rates.

Alternatives

• Doing nothing: keeping current settings and reviewing them when operators change their documentation.
• Selective rules: disallowing training crawlers and control tokens while allowing search crawlers, the pattern behind Cloudflare's 17 % restriction figure.
• Licensing under contract, directly with a developer or through a collective license.
• Not licensing at all: blocking where possible and declining every offer, a choice weighed under whether to sell or license data to AI companies.

No option is recommended here; the choice turns on reliance on AI search, the wish for a price, and whether the site's network supports metering.

Where a crawler opt-out has legal effect

In the EU a machine-readable opt-out has legal effect, because Art. 4(3) of the DSM Directive lets rightholders reserve their content from commercial text and data mining. Directive (EU) 2019/790, in force since 2019-06-06 with transposition due by 2021-06-07, requires the reservation for online content to be made "in an appropriate manner, such as machine-readable means", while the research exception in Art. 3 has no opt-out.

The AI Act ties crawler rules to that reservation. Art. 53(1)(c) of Regulation (EU) 2024/1689, which applies from 2025-08-02, requires providers of general-purpose AI models to keep a copyright policy that identifies and complies with Art. 4(3) reservations "including through state-of-the-art technologies"; national rules on the reservation are covered under the EU text and data mining exception. The voluntary GPAI Code of Practice, published 2025-07-10, asks signatories in Measure 1.3 to use crawlers that read and follow robots.txt as specified in RFC 9309 and other appropriate machine-readable protocols.

On 2026-11-03 Advocate General Szpunar delivered his opinion in Case C-250/25 Like Company (press release 155/26), proposing that the training questions are inadmissible and saying, obiter, that training is as a rule permitted under the TDM exception. An opinion does not bind the Court of Justice and is not a ruling. Outside the EU, RFC 9309 creates no legal duty by itself, and the effect of a disallow rule depends on copyright, contract and computer-misuse law.

Crawling among the other routes to training data, and the law around it

Crawling is one of several routes by which AI developers obtain training data; the pages below cover the others and the law around them.

• How AI companies get training data: every route, from crawling to synthetic data.
• Copyright and AI training across jurisdictions: what an opt-out changes under each copyright model.
• AI copyright lawsuits over training data: court cases over crawled training data.
• Is web scraping for AI training legal: contract and computer-misuse law on scraping.
• Common Crawl: what the CCBot archive contains.
• Content licensing platforms for publishers: platforms that meter crawler access.

This page is information, not legal advice.

Frequently asked questions

Is ChatGPT a web crawler?

ChatGPT fetches pages through ChatGPT-User when a user asks it to, and OpenAI says robots.txt rules may not apply to those fetches. OpenAI's training crawler is GPTBot, and its search crawler is OAI-SearchBot (checked 2026-11-09).

Is Cloudflare blocking AI crawlers?

Cloudflare has blocked AI crawlers by default for new domains since 2025-07-01. On 2026-09-15 it announced separate Search, AI training and AI agents controls, with recommended rather than fixed defaults for new sites, to replace Block AI Bots. Its program terms are covered under Cloudflare pay per crawl and AI Crawl Control.

Does blocking Google-Extended affect Google Search?

No: Google states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls Gemini training and grounding only (checked 2026-11-09).

Does blocking an AI crawler remove content it already collected?

Blocking works forward. Common Crawl keeps its archive files unchanged and filters affected URLs from later crawls and indexes, and no other operator listed here states a removal practice.

How can a site check which AI crawlers visit it?

Server logs show the user agents that request pages, and matching their IP addresses against the operator's published list separates real crawlers from impostors. Apple and Common Crawl also publish reverse-DNS hostnames.