DataPacts

Common Crawl: What It Contains, License, Legal Status

Checked

At a glance

Common Crawl is a free archive of web pages collected since 2008 by the Common Crawl Foundation, a US nonprofit, and it is one of the most widely used sources of AI training data.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

CCBot collects public pages into numbered Common Crawl crawls that never change; C4, FineWeb, LAION-5B and four other datasets are built from them. A robots.txt Disallow reaches later crawls only.

Its latest crawl on record, CC-MAIN-2026-43, holds 2.16 billion pages (371.19 TiB uncompressed), and Common Crawl states that at least 64 % of 47 large language models published from 2019 to 2023 were trained on its data, citing a 2024 Mozilla Foundation study.

The archive is the unpaid counterpart to licensed data in AI training data licensing: the files are open to anyone, its crawler obeys robots.txt by Common Crawl's own account, and removal works only for future crawls.

Its Terms of Use, last updated on March 7, 2024, cap Common Crawl's liability at USD 100 (checked November 10, 2026). The sections below cover the organization, the archive, opt-out, terms, derived datasets, funding and legal position.

The Common Crawl Foundation: Who Runs the Archive

The archive is run by the Common Crawl Foundation, which describes itself as a 501(c)(3) nonprofit, gives a Beverly Hills address in its terms of use and dates its founding to 2007. The table lists the identity attributes that Common Crawl publishes about itself, as of November 10, 2026.

Attribute Value Source
Legal name Common Crawl Foundation commoncrawl.org/terms-of-use
Website commoncrawl.org commoncrawl.org
Address 9663 Santa Monica Blvd., #425, Beverly Hills, CA 90210 commoncrawl.org/terms-of-use
Status 501(c)(3) nonprofit, as Common Crawl states; registry number not checked commoncrawl.org/faq; commoncrawl.org/about
Founded 2007; founder named on the About page: Gil Elbaz commoncrawl.org/about
Executive Director Rich Skrenta since 2023, as the About page states commoncrawl.org/about
Staff about 18, as of 2026, as Common Crawl states commoncrawl.org/about
Collecting since 2008, "petabytes of data" commoncrawl.org/overview
Crawler CCBot commoncrawl.org/ccbot

The titles and the founding story are written here as Common Crawl's pages give them, not as independently verified records.

The identity matters for a holder in one practical way: the terms of use name the Foundation, with the Beverly Hills address, as the party behind the service, so it is the entity that publishes the archive, runs CCBot and answers for the terms discussed below.

The archive is the base layer for many of the sets compared on the hub for major AI training datasets, because C4, FineWeb, Dolma, RedPajama, LAION-5B, DataComp and Pile-CC all start from Common Crawl snapshots.

What the Archive Contains: Crawl Size and Cadence

Common Crawl publishes its archive as numbered crawls, and the latest on record, CC-MAIN-2026-43, holds 2.16 billion web pages collected from October 5 to October 18, 2026. The table gives the figures for that crawl from commoncrawl.org, as of November 10, 2026.

Attribute CC-MAIN-2026-43 Source
Status latest crawl, re-checked 2026-11-10 commoncrawl.org/latest-crawl
Crawl period 2026-10-05 to 2026-10-18 commoncrawl.org/blog/october-2026-crawl-archive-now-available
Announced 2026-10-21 same
Web pages 2.16 billion same
Uncompressed size 371.19 TiB same
URLs not in any prior crawl 830.1 million same

The announcement of October 21, 2026 is the source for every number in the table, and it describes one crawl, not the whole archive.

Common Crawl describes the whole archive only as "petabytes of data, regularly collected since 2008" (commoncrawl.org/overview, checked November 9, 2026). This page states no total archive size, no crawl count and no release interval, because commoncrawl.org gives none in the pages used here.

The 830.1 million new URLs are the part of the crawl that no earlier crawl held, which means the other pages of the 2.16 billion were seen before. A newer crawl supersedes these figures, so the latest crawl ID is worth re-checking on the Common Crawl site before any figure is reused.

Older crawl announcements stay online on the same blog, and they describe crawls that are no longer the latest. A figure from an undated forum post or an older announcement is therefore not a size of the archive today.

How Pages Enter the Archive and How a Site Keeps Them Out

Pages enter the archive through Common Crawl's own crawler, CCBot, which Common Crawl states obeys robots.txt, so a site keeps future pages out with a Disallow rule but cannot pull back pages already archived.

CCBot identifies itself as "CCBot/2.0 (https://commoncrawl.org/faq/)", honors Crawl-delay, waits a few seconds between requests by default and executes no JavaScript and sets no cookies (Common Crawl, checked November 9, 2026). The bot has its own page, CCBot, so this page stays with the archive.

Removal works forward only. Common Crawl states that its WARC files are immutable and that affected URLs are removed or filtered from subsequent crawls and public indexes (Common Crawl, November 4, 2025). It published the full list of its Opt-Out Ledger on September 17, 2025.

A site that blocks CCBot today therefore stops being collected in later crawls, while the copies in earlier crawls stay in the files. A block also binds only the crawler it names: it says nothing to other operators, and it does not change a dataset that someone already built from an older snapshot.

Impostors complicate the picture. Common Crawl writes that "we are aware of crawlers falsely identifying themselves as CCBot" and publishes a list of its addresses at index.commoncrawl.org/ccbot.json, plus reverse DNS names ending in crawl.commoncrawl.org (IPv4 only). A request that claims the CCBot name but fails both checks is not Common Crawl's crawler.

The Disallow route is the standard robots.txt mechanism: a User-agent line naming CCBot followed by a Disallow line. The syntax, its limits and the other training crawlers are set out in robots.txt rules, limits and examples for AI crawlers, which also explains why a robots.txt file is a request and not an enforcement tool.

Common Crawl's Terms of Use: What They Settle and What They Leave to the User

Common Crawl's Terms of Use, last updated on March 7, 2024, govern access to its service and cap Common Crawl's total liability at USD 100 (checked November 10, 2026). The table gives the terms this page relies on.

Term Value Source
Last updated 2024-03-07, re-checked 2026-11-10 commoncrawl.org/terms-of-use
Party Common Crawl Foundation, Beverly Hills address same
Liability cap USD 100 total same
Carried into derived sets C4 (Ai2 copy): ODC-BY plus the Common Crawl terms of use huggingface.co/datasets/allenai/c4

The terms bind whoever uses the service, and they do not replace the rights of the site owners whose pages sit in the files.

Derived sets can carry the terms along: the Ai2 copy of C4 states its license as ODC-BY plus the Common Crawl terms of use, so a user of that set takes on both.

How the licenses of the derived sets differ is the subject of dataset licenses for AI training. The terms do not decide whether training on crawled pages is lawful.

That question stays with the user and with the copyright, contract and scraping law of the place of use, and the pages in the archive remain subject to their owners' rights. This page does not summarize other clauses of the terms.

Who Has Trained on Common Crawl: Model Papers and Dataset Cards

Common Crawl states, citing a 2024 Mozilla Foundation study, that at least 64 % of 47 LLMs published from 2019 to 2023 were trained on its data, and dataset cards show how much of the open training sets comes from it.

The claim is Common Crawl's own summary of a third-party study; this page repeats it as such and has not re-read the study.

Developers rarely train on raw crawls. They filter, deduplicate and re-package snapshots into named sets, and the cards and papers of those sets state the Common Crawl part in different units. The table lists seven datasets, alphabetically, as of November 10, 2026, with each maintainer's own figure for its Common Crawl part.

The figures use different units (tokens, documents, dumps, image-text pairs) and are not added or compared. Each is the maintainer's own statement from the dataset card, paper or release post named in the source column, and the DataPacts page for each dataset, where one exists, covers its license and status in full.

Dataset Maintainer Common Crawl part, as the source states Release or check Source
C4 Google researchers (T5 paper); copy hosted by Ai2 web text extracted from the April 2019 crawl; ODC-BY plus Common Crawl terms 2019; card checked 2026-11-09 arxiv.org/abs/1910.10683; huggingface.co/datasets/allenai/c4
DataComp CommonPool DataComp project 12.8 billion candidate image-text pairs 2023-04-27 arxiv.org/abs/2304.14108
Dolma 3 pool Ai2 Common Crawl 8.14 trillion tokens, 9.67 billion documents, of over 9 trillion tokens (Ai2 states) card checked 2026-11-09 huggingface.co/datasets/allenai/dolma3_pool
FineWeb Hugging Face 110 dumps, CC-MAIN-2013-20 to CC-MAIN-2025-26 card checked 2026-11-09 huggingface.co/datasets/HuggingFaceFW/fineweb
LAION-5B and Re-LAION-5B LAION e.V. 5.85 billion image-text pairs, metadata CC BY 4.0, taken down 2023-12-19; Re-LAION-5B 5,526,641,167 pairs, Apache-2.0 2022-03-31; 2024-08-30 laion.ai/blog/laion-5b; laion.ai/blog/relaion-5b
RedPajama Together Data-1T: 878 billion Commoncrawl tokens; V2: 84 snapshots, over 100 billion documents cards checked 2026-11-09 huggingface.co/datasets/togethercomputer
The Pile EleutherAI Pile-CC, one of 22 subsets of 825 GiB 2020-12-31 arxiv.org/abs/2101.00027

For a holder, the lineage has one consequence: a page blocked today can still sit in sets built from older crawls. FineWeb alone draws on 110 dumps back to CC-MAIN-2013-20, and the files of each crawl stay unchanged, so a block placed in 2026 does not reach a set filtered in 2019 or 2024.

How Common Crawl Is Funded and Its Ties to AI Developers

Common Crawl publishes no funder list in the record used here; it states that donations from AI companies cover only "a small fraction of our overall operating needs" (Common Crawl, November 4, 2025).

That statement is Common Crawl's own account, published in a post of the same date as its removal-practice statement, and not an audited figure. It says donations from AI companies exist and are a minor part of operating needs, and it names no donor and gives no amount.

The nonprofit status and the staff of about 18 (as of 2026, Common Crawl states) are the other facts on record. On July 27, 2026 Common Crawl posted that it joined Project Tapestry, an initiative of the AI Alliance.

This page states no named donor, no budget and no hosting arrangement, because none is on record in the sources used. Membership in an AI Alliance project is a statement about participation, and it is not a statement about funding or about how the archive is used.

Common Crawl's legal position depends on where its archive and the datasets built from it are used, and the main open questions in the record concern machine-readable opt-outs and datasets derived from the crawl.

In the EU, Article 4(3) of Directive (EU) 2019/790 (in force June 6, 2019) lets a rightsholder reserve rights for online content, and the reservation must be machine-readable; Article 7(1) does not protect the Article 4 exception against contract override, which means a contract can restrict what the exception would otherwise allow.

The wider EU text and data mining exception and the opt-out page covers how reservations work.

In the US, the two training rulings on record concern books, not web archives: Bartz v. Anthropic (N.D. Cal. 3:24-cv-05417, Dkt. 231, June 23, 2025), which held training on books fair use but did not excuse pirated library copies, and Kadrey v.

Meta (N.D. Cal. 3:23-cv-03417-VC, Dkt. 598, June 25, 2025), which granted summary judgment to Meta on the record before it. The Kadrey ruling states that it "does not stand for the proposition that Meta's use of copyrighted materials to train its language models is lawful".

LAION-5B, built on Common Crawl, is the dataset at the center of the German case on dataset creation.

The dated routing line: in Kneschke v. LAION, LG Hamburg 310 O 227/23 decided on September 27, 2024, OLG Hamburg 5 U 104/24 on December 10, 2025, and the Federal Court of Justice (BGH) docket I ZR 281/25 was heard on September 3, 2026, with pronouncement scheduled for December 17, 2026, as of November 9, 2026.

No case in the DataPacts case tracker names Common Crawl as a party (checked November 10, 2026).

This section is information, not legal advice.

Content in the Archive: Benefits, Risks, Downsides and Alternatives for a Holder

For a site owner or data holder, being in Common Crawl brings reach without payment, and leaving it stops only future collection.

Benefits

  • Reach: pages in the archive can flow into every dataset built on it, from C4 to LAION-5B.
  • No cost and no contract: the holder signs nothing and pays nothing for inclusion.

Risks

  • Copies persist in immutable files and in older derived sets.
  • Impostor crawlers can use the CCBot name, so a user-agent string alone proves nothing.
  • Common Crawl's liability under its terms is capped at USD 100 (terms of March 7, 2024).

Downsides

  • Content that is free in the archive is harder to license for a fee, since a buyer can obtain it at no cost.
  • Restricting is spreading but works forward: Longpre et al. (July 20, 2024) found about 5 % or more of all C4 tokens and 28 % or more of the most critical sources fully restricted by robots.txt in 2023 to 2024.

Alternatives

The choice depends on the type of content and on whether any buyer would license it, and no option here is presented as the right one.

Crawler Controls, Removal Records and the Law Beyond This Page

This page covers Common Crawl as a source; crawler controls, removal requests, other sources and the wider law each have their own page.

Frequently Asked Questions

What does Common Crawl do?

Common Crawl crawls the web with CCBot and publishes free archives that others use to build AI training sets. The Common Crawl Foundation, a US nonprofit, has collected pages since 2008. C4, FineWeb, Dolma, RedPajama, LAION-5B, DataComp and Pile-CC draw on its snapshots.

How big is Common Crawl?

The latest crawl on record, CC-MAIN-2026-43, holds 2.16 billion pages and 371.19 TiB uncompressed (announced October 21, 2026). Common Crawl describes the whole archive as "petabytes of data" and states no total crawl count on the pages used here.

Who funds Common Crawl?

No funder list is on record; Common Crawl states that AI company donations are "a small fraction of our overall operating needs" (November 4, 2025). It describes itself as a 501(c)(3) nonprofit with about 18 staff as of 2026.

It depends on the jurisdiction and the use, and no ruling on record decides it for Common Crawl itself. The terms govern access to the service and cap liability, and the EU and US positions are in the legal section above. The copyright, contract and computer-misuse questions sit on the page is web scraping for AI training legal.

Agents and advisers can use the introducer form.