DataPacts

CCBot: Common Crawl's Crawler

Checked

At a glance

CCBot is the web crawler of the Common Crawl Foundation, a US nonprofit that publishes the pages it collects as free archives, which AI developers and dataset builders reuse to train models.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

Crawler controls are one part of AI training data licensing and acquisition, because a site's rule for CCBot reaches every later user of those archives.

Every crawler statement here is Common Crawl's own, taken from its CCBot page and FAQ and checked on 2026-11-16, and the page recommends no rule. A site blocks CCBot with two robots.txt lines, and Common Crawl publishes five IP ranges (dated 2026-08-11) for checking that a visitor is the real crawler.

Every figure and statement carries its stating party and check date, and how we source, date and check every figure sets out the method.

What CCBot collects and where the pages go

CCBot fetches public web pages for Common Crawl's archive, and Common Crawl publishes that archive at no cost while others filter it into training datasets.

Common Crawl describes itself as a 501(c)(3) nonprofit, founded in 2007, that provides "a copy of the Internet" for research and analysis (FAQ and About page, checked 2026-11-16). Its About page says the organization replaced a custom crawler in 2013 with a system based on Apache Nutch, designated CCBot.

CCBot does not feed one developer's models, unlike a developer's own crawler such as GPTBot. Its output goes to whoever downloads the archive, and Common Crawl states that at least 64 % of 47 large language models published from 2019 to 2023 were trained on filtered versions of its data (About page, checked 2026-11-16).

The hub on AI crawlers and how to allow, block or charge them places CCBot among the other operators' bots.

CCBot's user agent and identity as Common Crawl documents them

CCBot identifies itself with the user agent string CCBot/2.0 (https://commoncrawl.org/faq/), and its robots.txt exclusion token is CCBot.

Common Crawl adds that it "may increment the version number in the future", so a rule that matches the token survives a version change where a match on the full string does not. The FAQ also lists the string of an older bot, which is not reproduced here.

The table below lists the attributes Common Crawl states for CCBot, as of 2026-11-16, with the page each comes from.

Attribute Value (Common Crawl's statement) Checked Source
Operator Common Crawl Foundation, 501(c)(3) nonprofit as it states 2026-11-16 commoncrawl.org/faq
User agent string CCBot/2.0 (https://commoncrawl.org/faq/) 2026-11-16 commoncrawl.org/ccbot
robots.txt token CCBot 2026-11-16 commoncrawl.org/faq
Version "We may increment the version number in the future" 2026-11-16 commoncrawl.org/faq
Software Nutch-based, uses Apache Hadoop; replaced a custom crawler in 2013 2026-11-16 commoncrawl.org/faq; commoncrawl.org/about
Protocols HTTP/1.1 and HTTP/2 (HTTP/2 only over TLS); IPv4 and IPv6 2026-11-16 commoncrawl.org/faq
JavaScript and cookies not executed, not used 2026-11-16 commoncrawl.org/faq
Crawl-delay honored 2026-11-16 commoncrawl.org/faq
IP list index.commoncrawl.org/ccbot.json; reverse DNS *.crawl.commoncrawl.org (IPv4 only) 2026-11-16 commoncrawl.org/ccbot
  • The string is Common Crawl's own wording, so a log line is compared with it as printed.
  • The token CCBot is what a robots.txt User-agent line matches.

robots.txt lines Common Crawl gives for CCBot

Common Crawl states that the two lines User-agent: CCBot and Disallow: / make its crawler stop crawling a website, and that a Crawl-delay line under the same user agent slows it down instead. Its words are "Add these lines to your robots.txt file and our crawler will stop crawling your website".

For a slower crawl, its example Crawl-delay: 2 keeps CCBot from requesting pages "more than once every 2 seconds", and by default the crawler waits a few seconds between requests to the same site.

The table below compares four whole-site rules and what each leaves running, as of 2026-11-16, with Common Crawl's FAQ as the source for the first three rows and RFC 9309 for the last.

Goal Lines What it does (Common Crawl's statement) What keeps running Source
Stop CCBot User-agent: CCBot and Disallow: / "our crawler will stop crawling your website" pages in earlier crawls; other crawlers; spoofers commoncrawl.org/faq
Slow CCBot User-agent: CCBot and Crawl-delay: 2 no more than one request every 2 seconds crawling at the slower rate commoncrawl.org/faq
Point CCBot to pages Sitemap: https://example.com/sitemap.xml "utilizes any Sitemap announced in the robots.txt file" sampling still applies commoncrawl.org/faq
Leave CCBot allowed no CCBot group the file's general * group applies (RFC 9309, not a Common Crawl statement) everything rfc-editor.org/rfc/rfc9309

These are whole-site rules only. Common Crawl's pages do not document path-level rules, and a robots.txt file is a request, never legally binding by itself. Syntax, precedence and the tokens of other operators sit with robots.txt for AI crawlers.

How CCBot chooses and fetches pages

CCBot reads robots.txt before it fetches a page, crawls a sample of each site rather than the whole site, and backs off when a server answers with HTTP 429 or a 5xx error, by Common Crawl's account (FAQ and About page, checked 2026-11-16). The list below gives the fetch rules it states.

  • Order: robots.txt first, then HTTP GET requests for pages the file allows.
  • Discovery: links from other sites, so a page can be fetched without the site having announced it.
  • Sampling: "a randomly selected subset" of a site, not the entire site.
  • Priority: Harmonic Centrality, computed from Common Crawl's Web Graph, alongside PageRank.
  • Redirects: up to four consecutive, or five when fetching robots.txt "in line with RFC 9309".
  • Protocols: HTTP/1.1 and HTTP/2, HTTP/2 only over TLS, over IPv4 and IPv6.
  • Back-off: an adaptive algorithm slows requests on HTTP 429 or 5xx answers.
  • Efficiency: conditional GET requests, and gzip, Brotli and ZStandard encodings.

Common Crawl states that JavaScript is not executed and cookies are not used, so content that only a script renders is not fetched. It also states that CCBot honors the nofollow attribute on links embedded on a site, and that any future change would apply only to link discovery.

Checking that a visitor really is CCBot: IP ranges and reverse DNS

A request is from CCBot only if its source address falls in one of the ranges Common Crawl publishes at index.commoncrawl.org/ccbot.json and, for IPv4, its reverse DNS name ends in crawl.commoncrawl.org and resolves back to the same address.

Common Crawl writes that "we are aware of crawlers falsely identifying themselves as CCBot" and recommends verifying user agent strings.

The table below lists the five ranges Common Crawl published as of 2026-08-11, checked 2026-11-16 against its FAQ and the JSON file.

Range Family Reverse DNS As of Source
3.41.188.32/29 IPv4 *.crawl.commoncrawl.org 2026-08-11 ccbot.json; FAQ
18.97.9.168/29 IPv4 *.crawl.commoncrawl.org 2026-08-11 ccbot.json; FAQ
18.97.14.80/29 IPv4 *.crawl.commoncrawl.org 2026-08-11 ccbot.json; FAQ
18.97.14.88/30 IPv4 *.crawl.commoncrawl.org 2026-08-11 ccbot.json; FAQ
2600:1f28:365:8000::/56 IPv6 none ("not yet supported") 2026-08-11 ccbot.json; FAQ

Common Crawl publishes the current ranges at index.commoncrawl.org/ccbot.json; check that file before relying on this table.

  1. Match the user agent token CCBot in the request.
  2. Match the source address against the published ranges.
  3. For IPv4, run the reverse lookup and then the forward lookup. Common Crawl's own example resolves 18.97.14.84 to 18-97-14-84.crawl.commoncrawl.org, which resolves back to 18.97.14.84.

The JSON file, created 2026-08-11T13:40:00Z, notes that "FCrDNS is also recommended" for IPv4. Over IPv6 there is no reverse DNS, so the list is the only check there. A spoofer ignores robots.txt, so blocking fake CCBot traffic takes a firewall rule rather than a robots.txt line.

What a CCBot disallow does and does not change

A CCBot disallow stops future crawling of the site by Common Crawl's statement, but it leaves pages in earlier crawls untouched, it says nothing to other crawlers, and Common Crawl gives no time for when its crawler sees the change.

Its FAQ says "We will periodically continue to check if the robots.txt file has been updated", with no interval, while RFC 9309 says a cached robots.txt copy should not be used for more than 24 hours.

A CCBot disallow leaves four things in place:

  • Pages in earlier crawl files: Common Crawl states its WARC files are immutable and that affected URLs are removed or filtered from subsequent crawls and public indexes (statement of 2025-11-04).
  • Datasets already built from earlier crawls.
  • Other crawlers, because a CCBot line binds only CCBot.
  • Spoofers, which do not read the rule.

Common Crawl also keeps an opt-out list of legal requests it receives, published in full on 2025-09-17 and renamed from Opt-Out Registry to Opt-Out Ledger "to reflect the immutable nature of it".

Requests go to info@commoncrawl.org, and Common Crawl encourages publishers to use standard methods like robots.txt files to control crawling. What the Common Crawl archive contains and how removal works there is covered on the dataset page.

Page-level meta tags such as noai or noarchive and the X-Robots-Tag header are not documented on Common Crawl's CCBot page or FAQ, so nothing is stated about them here.

Who uses the pages CCBot collects, by Common Crawl's account

Common Crawl states that at least 64 % of 47 large language models published from 2019 to 2023 were trained on filtered versions of its data, citing a 2024 Mozilla Foundation study, and its About page names GPT-3, T5, LLaMA, BLOOM, Pythia and Falcon as examples. Common Crawl adds that use is typically after filtering and processing by third parties.

The table below lists the six models in the order Common Crawl lists them, as of 2026-11-16, with the paper figures where the paper gives them.

Model Developer, year (Common Crawl's list) Common Crawl part, as Common Crawl states Paper figure on record Source
GPT-3 OpenAI, 2020 "More than 80% of training tokens derived from filtered Common Crawl data" 410 billion tokens, 60 % weight in the training mix (Table 2.2) commoncrawl.org/about; arxiv.org/abs/2005.14165
T5 (C4) Google, 2019 C4 "constructed by applying quality filters to a single Common Crawl snapshot" not stated here commoncrawl.org/about
LLaMA Meta, 2023 Common Crawl data processed through the CCNet pipeline CommonCrawl 67.0 % sampling proportion (Table 1) commoncrawl.org/about; arxiv.org/abs/2302.13971
BLOOM BigScience, 2022 the OSCAR corpus, derived from Common Crawl not stated here commoncrawl.org/about
Pythia EleutherAI Pile-CC, a filtered Common Crawl subset within The Pile not stated here commoncrawl.org/about
Falcon Technology Innovation Institute RefinedWeb, processed from Common Crawl not stated here commoncrawl.org/about

The GPT-3 row shows two different measures, a share of training tokens and a weight in the training mix, and they are kept apart. Dataset builders filter the crawl before training, and the derived sets are compared on the hub for major AI training datasets and their sources.

A CCBot disallow as an EU machine-readable reservation

In the EU, a robots.txt line that disallows CCBot is a likely machine-readable means for a rightsholder to reserve text and data mining rights under Article 4(3) of the DSM Directive, though no final ruling on record confirms this or decides its effect against Common Crawl.

Directive (EU) 2019/790, in force since 2019-06-06, allows the reservation "in an appropriate manner, such as machine-readable means in the case of content made publicly available online".

A reservation addressed to CCBot names one crawler. Whether it also binds a later user of earlier crawl files is not decided in any ruling recorded here.

In the US, no statute or ruling recorded here makes robots.txt binding, and the EU text and data mining exception and its opt-out are covered on their own page.

Blocking CCBot: benefits, risks, downsides and alternatives

A CCBot block needs only two lines and Common Crawl states that its crawler honors the rule, but it works only forward, brings no payment and does not reach copies already in earlier crawls. Licensing content is a separate decision from crawler rules.

Benefits:

  • The block is two robots.txt lines, and CCBot obeys them by Common Crawl's statement.
  • Future pages leave a pipeline that feeds many datasets, not one developer.
  • The disallow can serve as an EU reservation in machine-readable form, though no final ruling confirms it.

Risks:

  • Spoofers ignore the rule.
  • Common Crawl states no time for the change to take effect.
  • Earlier crawl files stay unchanged.

Downsides:

  • A block also leaves the research and non-AI uses Common Crawl names, such as language translation software, trend prediction and disease tracking.
  • A block brings no payment for pages already collected.
  • Common Crawl's About page cites a 2025 COLM paper that reports a close-to-zero "Data Compliance Gap" when models respect robots.txt opt-outs, so by Common Crawl's summary a block may change little in model quality.

Alternatives:

Keeping the robots.txt rules without licensing is a complete outcome, and none of these options is recommended here.

Other AI crawlers, opt-out routes and the law around CCBot

CCBot is one crawler among many; the pages below cover the others, the other ways to signal an opt-out, removal from datasets and the law around crawling.

This page is information, not legal advice.

Frequently asked questions

What is CCBot?

CCBot is the web crawler of the Common Crawl Foundation, a US nonprofit that publishes the pages it collects as free archives. AI developers and dataset builders filter those archives into training data (checked 2026-11-16).

Does CCBot respect robots.txt?

Common Crawl states that CCBot checks robots.txt before fetching a page and stops crawling a site whose robots.txt carries User-agent: CCBot and Disallow: /. It also honors Crawl-delay (checked 2026-11-16).

What is the CCBot user agent string?

Common Crawl gives CCBot/2.0 (https://commoncrawl.org/faq/) as the string and CCBot as the robots.txt token. It says the version number may change in the future (checked 2026-11-16).

Does blocking CCBot remove my pages from Common Crawl?

No: a block stops future crawling. Common Crawl states its WARC files are immutable and affected URLs are filtered only from subsequent crawls and public indexes (statement of 2025-11-04).

How do I slow CCBot down without blocking it?

Add a Crawl-delay line under User-agent: CCBot. Common Crawl's own example, Crawl-delay: 2, limits its crawler to no more than one request every 2 seconds, and the crawler backs off on HTTP 429 or 5xx answers (checked 2026-11-16).