Anthropic runs two other robots, Claude-User for fetches when a Claude user asks a question and Claude-SearchBot for search result quality, and crawler controls are one part of AI training data licensing and acquisition.
Every statement here is Anthropic's, from its article on web crawling last updated 2026-11-04 and re-read on 2026-11-16, and this page recommends no rule. A site can opt out per robot in robots.txt, and since 2026-10-30 Anthropic also recognizes a general no-AI-training signal, under which a rule that names ClaudeBot takes priority.
Every figure and statement carries its stating party and check date, and how we source, date and check every figure sets out the method.
What ClaudeBot collects, and how Claude-User and Claude-SearchBot differ
Anthropic states that ClaudeBot collects web content that could potentially contribute to training its generative AI models, and it describes Claude-User and Claude-SearchBot as separate robots for user questions and for search quality.
Anthropic's overview says it uses "a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users' direction" (checked 2026-11-16).
The list below gives each robot's stated use and what Anthropic says disabling it does.
- ClaudeBot: "collecting web content that could potentially contribute to their training"; a restriction "signals that the site's future materials should be excluded from our AI model training datasets".
- Claude-User: "When individuals ask questions to Claude, it may access websites using a Claude-User agent"; disabling it "may reduce your site's visibility for user-directed web search".
- Claude-SearchBot: "navigates the web to improve search result quality for users"; disabling it "may reduce your site's visibility and accuracy in user search results".
Anthropic calls ClaudeBot "our training crawler" and ties only ClaudeBot to training, and its article names no model beyond "generative AI models", so none is named here.
Claude-Agent is a separate token for a browser hosted by Anthropic that completes tasks at a user's direction, and Anthropic documents it on a different support page, outside the three robots. The section hub on AI crawlers and how to allow, block or charge them places Anthropic's robots among those of other operators.
Anthropic's three robots as Anthropic documents them
Anthropic's article lists exactly three robots, each with its use and what happens when a site disables it. The table below compares them in the order Anthropic lists them, as of 2026-11-16, and ranks none.
| Robot (robots.txt token) | Use (Anthropic's words) | What disabling it does (Anthropic's words) | Training (Anthropic's statement) | IP list | Checked | Source |
|---|---|---|---|---|---|---|
| ClaudeBot | "collecting web content that could potentially contribute to their training" | "signals that the site's future materials should be excluded from our AI model training datasets" | "our training crawler" | claude.com/crawling/bots.json | 2026-11-09, re-checked 2026-11-16 | privacy.claude.com |
| Claude-User | "When individuals ask questions to Claude, it may access websites using a Claude-User agent" | "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search" | not tied to training on the page | same file | 2026-11-16 | privacy.claude.com |
| Claude-SearchBot | "navigates the web to improve search result quality for users" | "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results" | not tied to training on the page | same file | 2026-11-16 | privacy.claude.com |
Reading key:
- Anthropic publishes the robots.txt token for each robot but no full user-agent string.
- One IP file covers the robots, and it does not split addresses per robot.
- The Crawl-delay example on the page names ClaudeBot only.
robots.txt lines for ClaudeBot, Claude-User and Claude-SearchBot
Each Anthropic robot is addressed by its own User-agent group, and Anthropic's example for blocking a robot from a whole site is User-agent: ClaudeBot followed by Disallow: / in the robots.txt file of the top-level directory.
Anthropic adds: "Please do this for every subdomain that you wish to opt out from" (checked 2026-11-16). It also writes that its robots respect "do not crawl" signals by honoring industry standard directives in robots.txt and that it will not attempt to bypass CAPTCHAs for the sites it crawls.
The table below compares five whole-site goals as of 2026-11-16, each token in its own User-agent group, with what Anthropic's statements say the lines do and what keeps running.
| Goal | Lines | What it does (Anthropic's statement) | What keeps running | Source |
|---|---|---|---|---|
| Stop training collection, stay in Claude search | User-agent: ClaudeBot then Disallow: / |
signals future materials should be excluded from training datasets | Claude-SearchBot indexing; Claude-User fetches | privacy.claude.com |
| Slow ClaudeBot instead of blocking it | User-agent: ClaudeBot then Crawl-delay: 1 |
non-standard extension Anthropic says it supports | ClaudeBot crawls at the slower rate | privacy.claude.com |
| Leave Claude search too | the first row plus User-agent: Claude-SearchBot then Disallow: / |
content not indexed for search optimization | Claude-User fetches | privacy.claude.com |
| Stop fetches on users' questions | User-agent: Claude-User then Disallow: / |
content not retrieved in response to a user query | the other two per their groups | privacy.claude.com |
| Opt out on a subdomain | the same groups in the subdomain's own robots.txt | Anthropic: "do this for every subdomain" | the main host per its own file | privacy.claude.com |
User-agent: ClaudeBot
Disallow: /
A User-agent: * group is the general wildcard, and Anthropic's article does not say how its robots read it. Syntax, wildcard groups and other operators' tokens are covered on the page about robots.txt for AI crawlers.
On pacing, Anthropic states: "we support the non-standard Crawl-delay extension to robots.txt", with Crawl-delay: 1 under User-agent: ClaudeBot as its example, and its principles mention "respecting Crawl-delay where appropriate" (checked 2026-11-09 and 2026-11-16). The article gives no Crawl-delay statement for Claude-User or Claude-SearchBot, so none is made here.
Anthropic's no-AI-training signal and why a ClaudeBot rule takes priority
Since October 30, 2026, Anthropic says it recognizes a machine-readable signal through which a site can state "no AI training", and a robots.txt rule that names ClaudeBot takes priority over that general setting for the same path.
Anthropic's wording is "As of October 30, 2026, Anthropic has begun recognizing a machine-readable signal that lets website owners express a preference about whether their content may be used to train AI models", rolled out "across all of our services over time and globally" (checked 2026-11-16).
The table below compares the two cases Anthropic describes, per path, as of 2026-11-16.
| Situation for a path | What Anthropic follows | Checked | Source |
|---|---|---|---|
| robots.txt names ClaudeBot | the ClaudeBot rule, "even if your general setting says something different" | 2026-11-16 | privacy.claude.com |
| robots.txt does not mention ClaudeBot | "the general setting described above" | 2026-11-16 | privacy.claude.com |
Anthropic gives its reason in one sentence: "ClaudeBot is our training crawler, and we treat rules that name it directly as more specific than the general signal."
It adds that "the way we read and apply these settings may change too", because standards in this space are still evolving. The article links an IETF draft, draft-kelley-aipref-training-preference-signal, and names no token string, so this page names none on Anthropic's behalf.
The draft's status, its token and how Anthropic's description differs from the datatracker record are set out on the page about the shared AI training opt-out.
Checking that a visitor really is ClaudeBot: the IP list and the missing user-agent string
A request comes from Anthropic, by Anthropic's own test, when its source IP address is on the list Anthropic publishes at claude.com/crawling/bots.json; the robot's name in the user-agent header alone proves nothing, because Anthropic publishes the tokens but no full user-agent string.
Anthropic writes: "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic" (checked 2026-11-16).
A check takes three steps:
- Match the robot's token (ClaudeBot, Claude-User or Claude-SearchBot) in the User-Agent header, remembering that any client can send it.
- Match the source IP address against claude.com/crawling/bots.json.
- For questions, or if the crawlers seem not to work as described, write to claudebot@anthropic.com from an email that includes the domain in question, as Anthropic asks.
The file carries a creation time of 2026-10-20 and 14 IPv4 prefixes at the 2026-11-16 reading, with no split per robot.
Anthropic advises against the shortcut of blocking addresses: "Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."
Claude-Agent traffic checks against a different file, claude.com/.well-known/sail, and Anthropic says to "match on the Claude-Agent token rather than the full string".
What a ClaudeBot disallow does and does not change
A ClaudeBot disallow signals, in Anthropic's words, that the site's future materials should be excluded from Anthropic's AI model training datasets; it does not stop Claude-SearchBot or Claude-User unless each gets its own group.
Anthropic's sentence is "When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets" (checked 2026-11-16).
A ClaudeBot disallow leaves four things running:
- Claude-SearchBot indexing for search, a separate robot with its own group.
- Claude-User fetches on a user's question, likewise separate.
- Content collected before the rule, since the word is "future" and Anthropic's article says nothing about earlier collection.
- A general no-AI-training signal for the paths the ClaudeBot rule covers, because the ClaudeBot rule is followed instead.
Anthropic's article names no exception for Claude-User, and it states what it states: no comparison with other operators follows. robots.txt is not legally binding by itself under RFC 9309 (2022), and licenses a site has signed with anyone are untouched by the rule.
Anthropic's EU commitments and the Reddit lawsuit around its web collection
In the EU, a ClaudeBot disallow can act as the machine-readable reservation of DSM Directive Article 4(3), and Anthropic is a listed signatory of the GPAI Code of Practice, which commits signatories to crawlers that follow robots.txt as specified in RFC 9309.
Article 4(3) of Directive (EU) 2019/790 (in force 2019-06-06) allows text and data mining only where use has not been reserved "in an appropriate manner, such as machine-readable means in the case of content made publicly available online", as set out on the page about the EU text and data mining exception and its opt-out.
Anthropic is among the 22 signatories on the European Commission's Code page, last updated 2026-10-07 (checked 2026-11-09). The Code is voluntary and was published on 2025-07-10, and its Copyright chapter, Measure 1.3, says signatories employ crawlers that read and follow robots.txt as specified in RFC 9309.
Regulation (EU) 2024/1689, Article 53(1)(c), applies from 2025-08-02 and requires general-purpose AI providers to keep a policy that identifies and complies with Article 4(3) reservations.
In the US, no statute or ruling that makes robots.txt binding is recorded here. Reddit, Inc. v.
Anthropic PBC alleges breach of contract, unjust enrichment, trespass to chattels, tortious interference with contract and unfair competition over automated access to Reddit content to train Claude; the N.D. Cal. remanded it on 2026-03-30 (3:25-cv-05643-TLT, Dkt. 60), and as of 2026-11-09 it is pending in San Francisco County Superior Court (CGC-25-625892), with the claims narrowed in September 2026 to breach of contract, interference with contract and unfair competition, as Anthropic reported in its S-1/A.
The record held here does not say whether the complaint names ClaudeBot, so none is claimed, and the page on Reddit v. Anthropic claims, rulings and status tracks the case.
What ClaudeBot's crawling sends back: Cloudflare's crawl-to-refer measurements
Cloudflare's measurements show Anthropic's crawling returning very few referrals: it reported a crawl-to-refer ratio of 70,900:1 for 19 to 26 June 2025 and nearly 50,000:1 for 1 to 7 August 2025. The table below compares four Cloudflare measurements as of 2026-11-16, each with its own window and none ranked.
| Measure | Value (Cloudflare's claim) | Window | Source |
|---|---|---|---|
| ClaudeBot share of websites accessed | 11.17 % | year to July 2024 (post 2024-07-03) | blog.cloudflare.com |
| Anthropic crawl-to-refer ratio | 70,900:1 | 2025-06-19 to 2025-06-26 | blog.cloudflare.com |
| Anthropic crawl-to-refer ratio | nearly 50,000:1 | 2025-08-01 to 2025-08-07 | blog.cloudflare.com |
| Anthropic crawl-to-refer ratio, 2025 | as much as 500,000:1 at peak; roughly 25,000:1 to 100,000:1 after May | 2025-01-01 to 2025-12-02 | blog.cloudflare.com (Radar 2025 Year in Review) |
Cloudflare states that the June 2025 ratio may be overstated because the Claude app sends no Referer. The windows differ in length and date, so no trend is drawn from the four rows.
Blocking ClaudeBot: benefits, risks, downsides and alternatives
Disallowing ClaudeBot costs nothing and Anthropic states what it means, but it earns nothing, reaches only future training collection, and leaves Claude-SearchBot and Claude-User to their own rules. Licensing content to Anthropic is a separate decision from crawler rules, and the profile of Anthropic as a data buyer covers that side.
Benefits:
- The rule is free, a plain text file.
- Anthropic states in its own words what a ClaudeBot restriction means.
- Search visibility stays if Claude-SearchBot stays allowed.
- In the EU, the disallow can act as a machine-readable reservation under DSM Directive Article 4(3).
Risks:
- Spoofed tokens pass as Anthropic's robots without an IP check.
- Blocking IP addresses may stop Anthropic reading robots.txt, by its own warning.
- A rule on the main host alone leaves subdomains open, since each needs its own file.
Downsides:
- The signal is prospective only: "future materials".
- A disallow brings no payment, and Anthropic's article mentions none.
- A ClaudeBot rule overrides a later general no-AI-training setting for the same path, in both directions.
Alternatives:
- Allow everything.
- Disallow ClaudeBot and keep Claude-SearchBot allowed.
- Rely on the general no-AI-training signal without a ClaudeBot rule, where Anthropic follows the general setting.
- Charge crawlers through intermediaries, license content directly, or decide on whether to sell or license your data to AI companies at all, where not licensing is a valid outcome.
Blocking, allowing and licensing can be combined, and none is recommended here.
Other AI crawlers, opt-out signals and the law around Anthropic's crawling
ClaudeBot is one crawler among many, and the pages below cover the others, the other ways to signal an opt-out and the law around crawling.
- AI crawlers and how to allow, block or charge them: every AI crawler and control method in one place.
- GPTBot and OpenAI's other crawlers: OpenAI's GPTBot and its search and user bots.
- Google-Extended: opting out of Gemini training.
- Amazonbot: Amazon's crawler and its page-level training opt-out.
- CCBot: Common Crawl's crawler, whose archive other developers train on.
- PerplexityBot and Perplexity-User: Perplexity's search crawler and user fetcher.
- Meta-ExternalAgent: Meta's AI training crawler.
- Bytespider: ByteDance's undocumented crawler, by contrast with Anthropic's documented ones.
- Applebot-Extended: Apple's training opt-out token.
- Machine-readable TDM reservations and meta tags: reservation signals beyond robots.txt.
- Pay-per-crawl and crawler licensing: charging crawlers instead of blocking them.
- Opting out of AI training through platform settings: opt-outs for content hosted on platforms, where robots.txt is not the owner's to edit.
- EU AI Act Article 53 copyright policy: the copyright policy GPAI providers such as Anthropic must keep under Article 53.
- Is web scraping for AI training legal: copyright, contract and computer-misuse law on scraping.
This page is information, not legal advice.
Frequently asked questions
Does blocking ClaudeBot remove a site from Claude's search answers?
Not by Anthropic's account: Claude-SearchBot and Claude-User are separate robots with their own robots.txt groups, and Anthropic ties reduced search visibility to disabling those two, not ClaudeBot. The statement was checked on 2026-11-16.
What is the ClaudeBot user agent?
Anthropic gives ClaudeBot as the name to use in robots.txt, but its article publishes no full user-agent string. Anthropic's own check is whether the source IP address is on claude.com/crawling/bots.json (checked 2026-11-16).
Does ClaudeBot respect Crawl-delay?
Yes, by Anthropic's statement: it supports the non-standard Crawl-delay extension and gives Crawl-delay: 1 under User-agent: ClaudeBot as its example. The statement was checked on 2026-11-09 and 2026-11-16.
Does a ClaudeBot block remove content Anthropic already collected?
Anthropic does not say so: it states that restricting ClaudeBot signals that the site's future materials should be excluded from its AI model training datasets, and says nothing about earlier collection. The statement was checked on 2026-11-16.
Is Claude-Agent one of Anthropic's crawlers?
No, in Anthropic's description: Claude-Agent marks a browser hosted by Anthropic that completes tasks at a user's direction, uses its own IP file, and is managed separately from ClaudeBot, Claude-User and Claude-SearchBot. The description was checked on 2026-11-16.