
Its latest crawl on record, CC-MAIN-2026-43, holds 2.16 billion pages (371.19 TiB uncompressed), and Common Crawl states that at least 64 % of 47 large language models published from 2019 to 2023 were trained on its data, citing a 2024 Mozilla Foundation study.
The archive is the unpaid counterpart to licensed data in AI training data licensing: the files are open to anyone, its crawler obeys robots.txt by Common Crawl's own account, and removal works only for future crawls.
Its Terms of Use, last updated on March 7, 2024, cap Common Crawl's liability at USD 100 (checked November 10, 2026). The sections below cover the organization, the archive, opt-out, terms, derived datasets, funding and legal position.
The Common Crawl Foundation: Who Runs the Archive
The archive is run by the Common Crawl Foundation, which describes itself as a 501(c)(3) nonprofit, gives a Beverly Hills address in its terms of use and dates its founding to 2007. The table lists the identity attributes that Common Crawl publishes about itself, as of November 10, 2026.
| Attribute | Value | Source |
|---|---|---|
| Legal name | Common Crawl Foundation | commoncrawl.org/terms-of-use |
| Website | commoncrawl.org | commoncrawl.org |
| Address | 9663 Santa Monica Blvd., #425, Beverly Hills, CA 90210 | commoncrawl.org/terms-of-use |
| Status | 501(c)(3) nonprofit, as Common Crawl states; registry number not checked | commoncrawl.org/faq; commoncrawl.org/about |
| Founded | 2007; founder named on the About page: Gil Elbaz | commoncrawl.org/about |
| Executive Director | Rich Skrenta since 2023, as the About page states | commoncrawl.org/about |
| Staff | about 18, as of 2026, as Common Crawl states | commoncrawl.org/about |
| Collecting since | 2008, "petabytes of data" | commoncrawl.org/overview |
| Crawler | CCBot | commoncrawl.org/ccbot |
The titles and the founding story are written here as Common Crawl's pages give them, not as independently verified records.
The identity matters for a holder in one practical way: the terms of use name the Foundation, with the Beverly Hills address, as the party behind the service, so it is the entity that publishes the archive, runs CCBot and answers for the terms discussed below.
The archive is the base layer for many of the sets compared on the hub for major AI training datasets, because C4, FineWeb, Dolma, RedPajama, LAION-5B, DataComp and Pile-CC all start from Common Crawl snapshots.
What the Archive Contains: Crawl Size and Cadence
Common Crawl publishes its archive as numbered crawls, and the latest on record, CC-MAIN-2026-43, holds 2.16 billion web pages collected from October 5 to October 18, 2026. The table gives the figures for that crawl from commoncrawl.org, as of November 10, 2026.
| Attribute | CC-MAIN-2026-43 | Source |
|---|---|---|
| Status | latest crawl, re-checked 2026-11-10 | commoncrawl.org/latest-crawl |
| Crawl period | 2026-10-05 to 2026-10-18 | commoncrawl.org/blog/october-2026-crawl-archive-now-available |
| Announced | 2026-10-21 | same |
| Web pages | 2.16 billion | same |
| Uncompressed size | 371.19 TiB | same |
| URLs not in any prior crawl | 830.1 million | same |
The announcement of October 21, 2026 is the source for every number in the table, and it describes one crawl, not the whole archive.
Common Crawl describes the whole archive only as "petabytes of data, regularly collected since 2008" (commoncrawl.org/overview, checked November 9, 2026). This page states no total archive size, no crawl count and no release interval, because commoncrawl.org gives none in the pages used here.
The 830.1 million new URLs are the part of the crawl that no earlier crawl held, which means the other pages of the 2.16 billion were seen before. A newer crawl supersedes these figures, so the latest crawl ID is worth re-checking on the Common Crawl site before any figure is reused.
Older crawl announcements stay online on the same blog, and they describe crawls that are no longer the latest. A figure from an undated forum post or an older announcement is therefore not a size of the archive today.
How Pages Enter the Archive and How a Site Keeps Them Out
Pages enter the archive through Common Crawl's own crawler, CCBot, which Common Crawl states obeys robots.txt, so a site keeps future pages out with a Disallow rule but cannot pull back pages already archived.
CCBot identifies itself as "CCBot/2.0 (https://commoncrawl.org/faq/)", honors Crawl-delay, waits a few seconds between requests by default and executes no JavaScript and sets no cookies (Common Crawl, checked November 9, 2026). The bot has its own page, CCBot, so this page stays with the archive.
Removal works forward only. Common Crawl states that its WARC files are immutable and that affected URLs are removed or filtered from subsequent crawls and public indexes (Common Crawl, November 4, 2025). It published the full list of its Opt-Out Ledger on September 17, 2025.
A site that blocks CCBot today therefore stops being collected in later crawls, while the copies in earlier crawls stay in the files. A block also binds only the crawler it names: it says nothing to other operators, and it does not change a dataset that someone already built from an older snapshot.
Impostors complicate the picture. Common Crawl writes that "we are aware of crawlers falsely identifying themselves as CCBot" and publishes a list of its addresses at index.commoncrawl.org/ccbot.json, plus reverse DNS names ending in crawl.commoncrawl.org (IPv4 only). A request that claims the CCBot name but fails both checks is not Common Crawl's crawler.
The Disallow route is the standard robots.txt mechanism: a User-agent line naming CCBot followed by a Disallow line. The syntax, its limits and the other training crawlers are set out in robots.txt rules, limits and examples for AI crawlers, which also explains why a robots.txt file is a request and not an enforcement tool.
Common Crawl's Terms of Use: What They Settle and What They Leave to the User
Common Crawl's Terms of Use, last updated on March 7, 2024, govern access to its service and cap Common Crawl's total liability at USD 100 (checked November 10, 2026). The table gives the terms this page relies on.
| Term | Value | Source |
|---|---|---|
| Last updated | 2024-03-07, re-checked 2026-11-10 | commoncrawl.org/terms-of-use |
| Party | Common Crawl Foundation, Beverly Hills address | same |
| Liability cap | USD 100 total | same |
| Carried into derived sets | C4 (Ai2 copy): ODC-BY plus the Common Crawl terms of use | huggingface.co/datasets/allenai/c4 |
The terms bind whoever uses the service, and they do not replace the rights of the site owners whose pages sit in the files.
Derived sets can carry the terms along: the Ai2 copy of C4 states its license as ODC-BY plus the Common Crawl terms of use, so a user of that set takes on both.
How the licenses of the derived sets differ is the subject of dataset licenses for AI training. The terms do not decide whether training on crawled pages is lawful.
That question stays with the user and with the copyright, contract and scraping law of the place of use, and the pages in the archive remain subject to their owners' rights. This page does not summarize other clauses of the terms.
Who Has Trained on Common Crawl: Model Papers and Dataset Cards
Common Crawl states, citing a 2024 Mozilla Foundation study, that at least 64 % of 47 LLMs published from 2019 to 2023 were trained on its data, and dataset cards show how much of the open training sets comes from it.
The claim is Common Crawl's own summary of a third-party study; this page repeats it as such and has not re-read the study.
Developers rarely train on raw crawls. They filter, deduplicate and re-package snapshots into named sets, and the cards and papers of those sets state the Common Crawl part in different units. The table lists seven datasets, alphabetically, as of November 10, 2026, with each maintainer's own figure for its Common Crawl part.
The figures use different units (tokens, documents, dumps, image-text pairs) and are not added or compared. Each is the maintainer's own statement from the dataset card, paper or release post named in the source column, and the DataPacts page for each dataset, where one exists, covers its license and status in full.
| Dataset | Maintainer | Common Crawl part, as the source states | Release or check | Source |
|---|---|---|---|---|
| C4 | Google researchers (T5 paper); copy hosted by Ai2 | web text extracted from the April 2019 crawl; ODC-BY plus Common Crawl terms | 2019; card checked 2026-11-09 | arxiv.org/abs/1910.10683; huggingface.co/datasets/allenai/c4 |
| DataComp CommonPool | DataComp project | 12.8 billion candidate image-text pairs | 2023-04-27 | arxiv.org/abs/2304.14108 |
| Dolma 3 pool | Ai2 | Common Crawl 8.14 trillion tokens, 9.67 billion documents, of over 9 trillion tokens (Ai2 states) | card checked 2026-11-09 | huggingface.co/datasets/allenai/dolma3_pool |
| FineWeb | Hugging Face | 110 dumps, CC-MAIN-2013-20 to CC-MAIN-2025-26 | card checked 2026-11-09 | huggingface.co/datasets/HuggingFaceFW/fineweb |
| LAION-5B and Re-LAION-5B | LAION e.V. | 5.85 billion image-text pairs, metadata CC BY 4.0, taken down 2023-12-19; Re-LAION-5B 5,526,641,167 pairs, Apache-2.0 | 2022-03-31; 2024-08-30 | laion.ai/blog/laion-5b; laion.ai/blog/relaion-5b |
| RedPajama | Together | Data-1T: 878 billion Commoncrawl tokens; V2: 84 snapshots, over 100 billion documents | cards checked 2026-11-09 | huggingface.co/datasets/togethercomputer |
| The Pile | EleutherAI | Pile-CC, one of 22 subsets of 825 GiB | 2020-12-31 | arxiv.org/abs/2101.00027 |
For a holder, the lineage has one consequence: a page blocked today can still sit in sets built from older crawls. FineWeb alone draws on 110 dumps back to CC-MAIN-2013-20, and the files of each crawl stay unchanged, so a block placed in 2026 does not reach a set filtered in 2019 or 2024.
How Common Crawl Is Funded and Its Ties to AI Developers
Common Crawl publishes no funder list in the record used here; it states that donations from AI companies cover only "a small fraction of our overall operating needs" (Common Crawl, November 4, 2025).
That statement is Common Crawl's own account, published in a post of the same date as its removal-practice statement, and not an audited figure. It says donations from AI companies exist and are a minor part of operating needs, and it names no donor and gives no amount.
The nonprofit status and the staff of about 18 (as of 2026, Common Crawl states) are the other facts on record. On July 27, 2026 Common Crawl posted that it joined Project Tapestry, an initiative of the AI Alliance.
This page states no named donor, no budget and no hosting arrangement, because none is on record in the sources used. Membership in an AI Alliance project is a statement about participation, and it is not a statement about funding or about how the archive is used.
Common Crawl's Legal Position: Reservations, Rulings and a Pending Dataset Case
Common Crawl's legal position depends on where its archive and the datasets built from it are used, and the main open questions in the record concern machine-readable opt-outs and datasets derived from the crawl.
In the EU, Article 4(3) of Directive (EU) 2019/790 (in force June 6, 2019) lets a rightsholder reserve rights for online content, and the reservation must be machine-readable; Article 7(1) does not protect the Article 4 exception against contract override, which means a contract can restrict what the exception would otherwise allow.
The wider EU text and data mining exception and the opt-out page covers how reservations work.
In the US, the two training rulings on record concern books, not web archives: Bartz v. Anthropic (N.D. Cal. 3:24-cv-05417, Dkt. 231, June 23, 2025), which held training on books fair use but did not excuse pirated library copies, and Kadrey v.
Meta (N.D. Cal. 3:23-cv-03417-VC, Dkt. 598, June 25, 2025), which granted summary judgment to Meta on the record before it. The Kadrey ruling states that it "does not stand for the proposition that Meta's use of copyrighted materials to train its language models is lawful".
LAION-5B, built on Common Crawl, is the dataset at the center of the German case on dataset creation.
The dated routing line: in Kneschke v. LAION, LG Hamburg 310 O 227/23 decided on September 27, 2024, OLG Hamburg 5 U 104/24 on December 10, 2025, and the Federal Court of Justice (BGH) docket I ZR 281/25 was heard on September 3, 2026, with pronouncement scheduled for December 17, 2026, as of November 9, 2026.
No case in the DataPacts case tracker names Common Crawl as a party (checked November 10, 2026).
This section is information, not legal advice.
Content in the Archive: Benefits, Risks, Downsides and Alternatives for a Holder
For a site owner or data holder, being in Common Crawl brings reach without payment, and leaving it stops only future collection.
Benefits
- Reach: pages in the archive can flow into every dataset built on it, from C4 to LAION-5B.
- No cost and no contract: the holder signs nothing and pays nothing for inclusion.
Risks
- Copies persist in immutable files and in older derived sets.
- Impostor crawlers can use the CCBot name, so a user-agent string alone proves nothing.
- Common Crawl's liability under its terms is capped at USD 100 (terms of March 7, 2024).
Downsides
- Content that is free in the archive is harder to license for a fee, since a buyer can obtain it at no cost.
- Restricting is spreading but works forward: Longpre et al. (July 20, 2024) found about 5 % or more of all C4 tokens and 28 % or more of the most critical sources fully restricted by robots.txt in 2023 to 2024.
Alternatives
- Block CCBot going forward with a robots.txt rule.
- Record an opt-out in Common Crawl's Opt-Out Ledger.
- License the content directly instead of leaving it to be crawled.
- Not licensing at all and leaving the content as it is, as covered in alternatives to selling data to AI labs, including not selling.
The choice depends on the type of content and on whether any buyer would license it, and no option here is presented as the right one.
Crawler Controls, Removal Records and the Law Beyond This Page
This page covers Common Crawl as a source; crawler controls, removal requests, other sources and the wider law each have their own page.
- AI crawlers and how to allow, block or charge them: every training crawler.
- Opting out of AI training on platforms: platform settings.
- Removing content from AI training datasets: takedowns and opt-out ledgers.
- Training data provenance: where data came from.
- How AI companies get training data: the nine routes.
- Licensed vs scraped training data: the licensed alternative.
- Copyright and AI training across jurisdictions: fair use and TDM exceptions.
- AI copyright lawsuits over training data: the case tracker.
Frequently Asked Questions
What does Common Crawl do?
Common Crawl crawls the web with CCBot and publishes free archives that others use to build AI training sets. The Common Crawl Foundation, a US nonprofit, has collected pages since 2008. C4, FineWeb, Dolma, RedPajama, LAION-5B, DataComp and Pile-CC draw on its snapshots.
How big is Common Crawl?
The latest crawl on record, CC-MAIN-2026-43, holds 2.16 billion pages and 371.19 TiB uncompressed (announced October 21, 2026). Common Crawl describes the whole archive as "petabytes of data" and states no total crawl count on the pages used here.
Who funds Common Crawl?
No funder list is on record; Common Crawl states that AI company donations are "a small fraction of our overall operating needs" (November 4, 2025). It describes itself as a 501(c)(3) nonprofit with about 18 staff as of 2026.
Is it legal to train AI on Common Crawl data?
It depends on the jurisdiction and the use, and no ruling on record decides it for Common Crawl itself. The terms govern access to the service and cap liability, and the EU and US positions are in the legal section above. The copyright, contract and computer-misuse questions sit on the page is web scraping for AI training legal.
Agents and advisers can use the introducer form.