Each record keeps the page's URL and crawl timestamp next to the text, so the owner of a website can check whether a page is in it.
The copy most users load is the allenai/c4 release on Hugging Face, which carries the Open Data Commons Attribution License (ODC-BY) plus the crawl's Terms of Use.
This page reads C4 from the side of the people whose pages it holds. Within AI training data licensing, C4 is free web text, the unpaid counterpart to licensed text: it brings its source sites no payment and no contract.
Who Built C4: The T5 Paper and the Copies Hosted Today
C4 was built by nine researchers at Google, who introduced it in the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", first posted on arXiv on 2019-10-23. The table lists the identity facts as of 2026-11-16.
| Attribute | Value | Source |
|---|---|---|
| Built by | nine researchers at Google (affiliation "Google, Mountain View, CA"), Raffel et al. | arxiv.org/abs/1910.10683 |
| Paper | "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", arXiv 1910.10683, v1 2019-10-23, v4 2023-09-19 | arxiv.org |
| First release | as part of TensorFlow Datasets; catalog versions 2.2.0 to 3.1.0 (default) | arxiv.org; tensorflow.org/datasets/catalog/c4 |
| Prepared copy | allenai/c4 on Hugging Face, "the processed version of Google's C4 dataset", five variants | huggingface.co/datasets/allenai/c4 |
| Host of the copy | Hugging Face, Inc.; repository not gated, last modified 2024-01-09 | huggingface.co |
| Stated purpose | "mainly intended to pretrain language models and word representations" (allenai card) | dataset card |
| Registry number | not checked | n/a |
The paper gives its reason for a new set: earlier sets "use a more limited set of filtering heuristics, are not publicly available, and/or are different in scope". Google released C4 as a TensorFlow Datasets builder that each user runs against the crawl, in versions 2.2.0 to 3.1.0, the last being the default.
The prepared copy most users load is allenai/c4, whose card calls it the processed version of Google's C4 dataset and thanks Google "for creating the code that curates and filters the data".
The card is published by The Allen Institute for Artificial Intelligence (Ai2), a Seattle-based non-profit, and the files are hosted by Hugging Face, Inc. Neither Ai2 nor Hugging Face created C4. C4 is one of the public corpora compared on the page about major AI training datasets.
What C4 Holds: The April 2019 Snapshot, Record Fields and Size
C4 holds English text from web pages in Common Crawl's April 2019 archive, one record per page, with the page's URL and crawl timestamp kept next to the text.
The paper started from the crawl's "web extracted text" for April 2019, and each record on the allenai card has three fields: url, text and timestamp, with an example timestamp of 2019-04-25T12:57:54Z.
Size is stated differently by each publisher, and this page keeps the counts as each states them. The paper says "about 750 GB" in 2019. The allenai en variant is 305 GB with 364,868,892 training documents (checked 2026-11-09), while the TensorFlow Datasets catalog, last updated 2022-12-06, lists 364,613,570 training examples for its en config.
The 2021 documentation study counted 365 million documents and 156 billion tokens with the SpaCy tokenizer. Counting rules differ between the three copies, so no count is called wrong and none is added to another.
| Variant | What it is | Size, as the publisher states | Source |
|---|---|---|---|
| en | the cleaned English corpus | allenai: 305 GB, 364,868,892 training documents (checked 2026-11-09); TFDS: 364,613,570 training examples; study: 365 million documents, 156 billion tokens | huggingface.co; tensorflow.org; arxiv.org/abs/2104.08758 |
| en.noclean | the English snapshot with no other filters | allenai: 2.3 TB; study: 1.1 billion documents, 1.4 trillion tokens | huggingface.co; arxiv.org |
| en.noblocklist | en without the bad-words filter | allenai: 380 GB; study: 395 million documents, 198 billion tokens | huggingface.co; arxiv.org |
| realnewslike | en limited to the domains of the RealNews dataset | allenai: 15 GB | huggingface.co |
| multilingual (mC4) | text in 108 languages, identified with CLD3 | allenai: 9.7 TB, 108 subsets | huggingface.co |
| webtextlike | en limited to URLs in OpenWebText; TFDS only, needs a manual file fetch | not stated | tensorflow.org |
The pages came from Common Crawl's archive. How the crawl collects pages and what else it feeds is covered on the page about what Common Crawl contains.
What C4's Filters Keep and Drop
C4 keeps only lines and pages that pass a set of heuristic filters the T5 paper lists, and it drops any page the language detector does not rate as English with a probability of at least 0.99. The paper's section 2.2 gives these rules.
- Lines are kept only if they end in terminal punctuation.
- Pages need at least 3 sentences, and lines at least 5 words.
- Pages containing any word on the "List of Dirty, Naughty, Obscene or Otherwise Bad Words" are removed.
- Lines mentioning Javascript are removed, as are pages with a curly bracket or with the placeholder phrase used in dummy text.
- Citation markers are removed.
- Lines containing "terms of use", "privacy policy", "cookie policy", "uses cookies", "use of cookies" or "use cookies" are removed.
- Repeated three-sentence spans are deduplicated.
The sixth rule matters to a site owner: the filter removed the lines in which a site states its terms, so C4's copy of a page carries the text without the page's own terms notice.
Being kept depended on these heuristics, not on any permission, and the paper's C4 section describes no handling of permissions or robots.txt.
The en.noblocklist variant is the same text without the bad-words filter, and en.noclean skips the other filters as well, which is why it is several times larger than en. The paper presents the filters as choices about text quality, and none of the steps checks a license or a rights reservation.
What the 2021 Study of C4's Sources Found
A 2021 study by researchers at Ai2, the University of Washington, Hugging Face and Queer in AI documented C4's sources and found that the single most represented website is patents.google.com.
The study is Dodge et al., "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus", arXiv 2104.08758, first posted on 2021-04-18 and updated on 2021-09-30 as the EMNLP 2021 camera-ready version. It covers three versions of C4 with different filtering.
The study states that C4 has been used to train models such as T5 and the Switch Transformer. Dodge et al. found patents.google.com first and patents.com in the top 10, with Wikipedia and news sites such as the New York Times, the Los Angeles Times and Al Jazeera well represented.
They report that a significant portion of the text comes from .gov sites, and 33,874,654 tokens from 58,394 unique URLs on .mil sites. They also found machine-translated text, because patents.google.com translates foreign patents into English, and benchmark text inside C4.en at rates from under 2 % to over 50 % depending on the benchmark.
The study also found that the blocklist filter disproportionately removes text from and about minority individuals, such as African American English and LGBTQ+ discussions.
Its authors wrote that personal data and copyrighted text likely exist in C4.en and left quantifying or removing them to future work, and they recommended a place for others to report issues. How datasheets and data cards record such findings is covered on the page about dataset documentation, datasheets and data cards.
Consent Signals on C4's Sources After the 2019 Snapshot
A 2024 audit by Longpre et al. found that, by 2023 to 2024, robots.txt fully restricted about 5 % or more of C4's tokens and 28 % or more of its most critical sources.
The authors, in a paper published on 2024-07-20, also report that 45 % of C4 is restricted by terms of service, across about 14,000 audited web domains.
Those restrictions date from years after the April 2019 snapshot, so they describe what site owners signal now and not what C4 already holds. A robots.txt block reaches only future crawls, because the upstream crawler states that it obeys robots.txt and its past files do not change.
The figures are the authors' claim, read against a snapshot that is more than seven years old on the date of this page, and they are not recounted per domain here.
The same audit is summarized on the hub for major datasets and on the Common Crawl page, so this page keeps only what bears on C4's own record: the signals measured are later than the copy, and they can change what the next crawl captures but not what C4 contains.
C4's Terms: ODC-BY Plus the Crawl's Terms of Use
The allenai/c4 card releases C4 under the Open Data Commons Attribution License (ODC-BY) and adds that users are also bound by Common Crawl's Terms of Use in respect of the content in the dataset.
The card says "We are releasing this dataset under the terms of ODC-BY", and Common Crawl's Terms of Use were last updated on 2024-03-07 (checked 2026-11-09). The TensorFlow Datasets catalog page names no license for the dataset, and the T5 paper states none. The table sets the layers side by side as of 2026-11-16.
| Layer | Terms | What it covers | What it leaves out | Source |
|---|---|---|---|---|
| The dataset (allenai/c4) | ODC-BY 1.0 | rights over the database (s.2.4) | the individual pages inside it, which stay under their own rights | huggingface.co; opendatacommons.org |
| The upstream crawl | Common Crawl Terms of Use, last updated 2024-03-07 | use of the content in the dataset, per the card | detail on the Common Crawl page | commoncrawl.org/terms-of-use |
| The original release (TFDS) | no dataset license stated on the catalog page | n/a | n/a | tensorflow.org/datasets/catalog/c4 |
ODC-BY section 2.4 states that the license governs rights over the database and not its individual contents. The license therefore does not license the pages inside C4, whose text stays under its authors' rights, and it does not settle whether training on those pages is lawful.
This page does not describe C4 as free for commercial use or as open source. How ODC licenses compare with CC and CDLA terms is covered on the page about dataset licenses for AI training.
C4's Legal Standing: No Court Record and No Removal Route
No court ruling or pending case on file names C4, as of 2026-11-16, and no licensing deal on file covers it. The search covered the court records and deals tracked on this site.
The allenai card describes no removal form, contact address or opt-out for site owners, and the T5 paper describes no handling of permissions or robots.txt. The hosted repository was last modified on 2024-01-09.
In the EU, a rightholder's reservation against text and data mining of content made publicly available online must be made in an appropriate manner, such as machine-readable means, under Article 4(3) of Directive (EU) 2019/790.
C4 was fixed in 2019, so a reservation matters for it only if it reached the crawler before the snapshot. The rule is set out on the page about the EU text and data mining exception.
The upstream crawler states that it obeys robots.txt. A reservation or block is therefore a tool for the future, and the owner who wants the pages out of a copy that already exists has no route described on the card.
Common Crawl says its past crawl files stay unchanged while affected URLs are removed or filtered only from later crawls and public indexes (statement of 2025-11-04), so a block placed today does not change the 2019 snapshot or copies already taken.
A Website in C4: Benefits, Risks, Downsides and Alternatives for the Owner
For the owner of a website whose pages sit in C4, the dataset brings no payment and no contract, and nothing done today changes the April 2019 snapshot. The four lists below separate what inclusion does from what it costs and end with the owner's options, including doing nothing.
Benefits
- Each record keeps the source URL, so inclusion can be checked page by page.
- Attribution travels with the database under ODC-BY, as a database-level duty only.
Risks
- Copies taken since 2019 stay with whoever holds them.
- The 2021 study expected personal data and copyrighted text in C4.en.
- No court ruling on file concerns training with C4, as of 2026-11-16.
Downsides
- The filter removed the page's own terms lines from the record.
- The card offers no removal route.
Alternatives
- Reserve rights in machine-readable form and block the upstream crawler for future snapshots.
- Send a notice to the host naming the pages, with no claim here about how the host answers.
- License the text directly to AI developers, as the page on text corpora for AI training describes.
- Not license at all and leave the pages as they are.
The benefits are narrow and the risks depend on court outcomes that have not arrived. Not licensing is a valid outcome, and the choice stays with the owner.
Crawler Controls, Other Corpora and Removal Beyond This Page
This page covers C4 and its variants; the crawler, removal routes, other corpora and the wider law each have their own page. Personal-data rules for training sets differ by jurisdiction.
- FineWeb: a newer filtered set under the same two layers of terms.
- robots.txt for AI crawlers: blocking crawlers for future snapshots.
- CCBot, Common Crawl's crawler: the upstream crawler.
- Machine-readable TDM reservations: reservations a machine can read.
- Privacy law and AI training data: personal data in training sets.
- Copyright and AI training across jurisdictions: fair use and TDM exceptions.
- Is web scraping for AI training legal: copyright, contract and computer-misuse law.
- LAION-5B and Re-LAION-5B: the image-text set from the same archive.
- Training data provenance: the chain of rights behind a dataset.
- Licensed vs scraped training data: the licensed alternative.
- AI copyright lawsuits over training data: the case tracker.
- Alternatives to selling data, including not selling at all: not licensing at all.
- Removing content from AI training datasets: removal routes across datasets.
- Dolma: Ai2's later corpus.
- RedPajama: another open text corpus.
- OpenWebText: the URL list behind the webtextlike config.
Frequently Asked Questions About C4
What is the C4 dataset?
C4 is English web text that Google researchers filtered from Common Crawl's April 2019 archive for the T5 paper. The paper describes it as about 750 GB, and it is now mostly used through the allenai/c4 copy hosted by Hugging Face.
How big is C4, and what is mC4?
The allenai en variant is 305 GB with 364,868,892 training documents (checked 2026-11-09). mC4 is the multilingual variant on the same card, 9.7 TB in 108 language subsets.
What license does C4 use?
The allenai/c4 card states ODC-BY plus Common Crawl's Terms of Use. The license covers the database, not the pages inside it, which stay under their own rights.
Is my website in C4?
Each record keeps its source URL, so a page can be checked against the files. The card describes no removal route, and blocking the crawler affects only future snapshots. This page is information, not legal advice.
Agents and advisers can use the introducer form.