
Common Crawl states on its About page, citing a 2024 Mozilla Foundation study, that at least 64 % of 47 large language models published from 2019 to 2023 were trained on its archives (checked 2026-11-09).
Book text came largely through Books3, a collection now removed from its host, and nine lawsuits in the case tracker name a public dataset or a books dataset.
This page reads those datasets from the side of the people and companies whose content sits inside them, as one part of AI training data licensing: what each corpus contains, who publishes it, which terms it carries, which court records name it and what inclusion means for the holder.
Most publishers license the compilation, not the works inside it. The page does not explain how to build, clean or format a dataset for training.
Every size, term and case status below names its source and the date it was checked, following the rules in how we source, date and check every figure.
What is LLM training data, and which public corpora supply it?
LLM training data falls into two groups: data a developer collects or licenses itself, and public datasets that third parties publish for anyone to copy. The first group stays mostly private; the second is documented in dataset cards, papers and takedown files that a holder can check.
Public corpora fall into six types, each with an example from the table below:
- Web crawl archives and filtered sets: Common Crawl, and C4 or FineWeb built from it.
- Book and mixed-source corpora: The Pile, Books3 and Common Corpus.
- Code: BigCode's The Stack.
- Image-text link sets: LAION-5B and DataComp CommonPool.
- Speech: Mozilla Common Voice.
- Egocentric video and robot trajectories: Ego4D and Open X-Embodiment.
The publishers include the Common Crawl Foundation, a 501(c)(3) nonprofit founded in 2007; Ai2 (The Allen Institute for Artificial Intelligence), Together Computer, Inc. and Pleias SAS; and collaborations such as BigCode, which names no separate legal entity.
Most open text sets are hosted by Hugging Face, Inc., a Delaware corporation according to its terms of service.
Model providers sometimes name the public datasets they used. The public summary of training content for Grok 4.7, published by SpaceXAI LLC (formerly xAI LLC) on 2026-09-21 on the EU AI Act template, names Common Corpus (Pleias), OpenAlex metadata and Crossref metadata.
The major datasets at a glance
The table covers 17 datasets, from the web crawl to book, image, code, speech and robotics corpora, plus the page on their licenses.
Each row gives the publisher, contents, size as stated, terms and legal record as of 2026-11-09 unless the As of column gives the publisher's own date, with the publisher's card, paper or page as the source.
| Dataset | Publisher | What it contains | Size as stated | Terms | Legal or removal record | As of | Source |
|---|---|---|---|---|---|---|---|
| Common Crawl | Common Crawl Foundation (501(c)(3) per its FAQ) | web crawl archives, regularly collected since 2008 (its claim) | CC-MAIN-2026-43: 2.16 billion pages, 371.19 TiB uncompressed | Terms of Use, updated 2024-03-07; liability cap USD 100 | Opt-Out Ledger published 2025-09-17; WARC files immutable | 2026-11-09 | commoncrawl.org |
| OpenWebText | Skylion007 project page | web page text | 38 GB from 8,013,769 documents | packaging CC0 | none on record | 2026-11-09 | skylion007.github.io |
| C4 (Colossal Clean Crawled Corpus) | Google T5 paper; allenai copy on Hugging Face | filtered English web text from an April 2019 snapshot | about 750 GB; en copy 305 GB, 364,868,892 documents | ODC-BY plus Common Crawl terms of use | robots.txt fully restricted 5 %+ of tokens, 2023 to 2024 (Longpre et al.) | 2026-11-09 | arxiv.org; huggingface.co |
| RedPajama | Together Computer, Inc. | 1T: mixed sources incl. Common Crawl; V2: web documents | 1T: 878 billion Common Crawl tokens; V2: over 100 billion documents | not stated | book configuration "defunct" over Books3; named in the In re Salesforce complaint | 2026-11-09 | huggingface.co |
| Dolma | Ai2 (The Allen Institute for Artificial Intelligence) | Common Crawl web text and other sources | v1.7: 4.5 TB gzip; Dolma 3 pool: over 9 trillion tokens (Ai2's claim) | ODC-BY since 2024-04-15 | none on record | 2026-11-09 | huggingface.co |
| FineWeb | Hugging Face, Inc. | filtered web text from 110 Common Crawl dumps | token total not stated | not stated | Torstar notice listing 361 domains, 2025-01-22 | 2026-11-09 | huggingface.co |
| DataComp, CommonPool and DCLM | mlfoundations research collaboration (legal entity not stated) | image-text candidate pool; DCLM web text | CommonPool: 12.8 billion pairs; DCLM-Pool: 240 trillion tokens | DCLM-Baseline: CC BY 4.0, research purposes only | none on record | 2026-11-09 | arxiv.org; huggingface.co |
| The Pile | EleutherAI (legal entity not stated) | 22 subsets incl. Books3 and Pile-CC | 825 GiB in the paper body; title says 800GB | not stated | named in the In re Salesforce complaint | 2020-12-31 | arxiv.org |
| Books3 | derived from the Bibliotik tracker (Pile paper) | full text of books | about 200,000 books (RettighedsAlliancen's figure); 100.96 GiB in The Pile | no license | removal from host announced 2023-08-14; named in Bartz, Huckabee and Nazemian | 2026-11-09 | rettighedsalliancen.dk; arxiv.org |
| Common Corpus | Pleias SAS | openly licensed and public domain text | 2,267,302,720,836 tokens (Pleias's claim); paper: about two trillion | not stated | named in the Grok 4.7 EU training summary | 2026-11-09 | huggingface.co |
| Institutional Books (Harvard Library and IDI) | Institutional Data Initiative, Harvard Law School Library | scanned library volumes | 983,004 volumes, 242 billion tokens | commercial AI use only after contacting IDI; noncommercial Early-Access terms | none on record | 2026-11-09 | arxiv.org; huggingface.co |
| Common Pile | EleutherAI and collaborators | openly licensed text from 30 sources | 8 TB | not stated | none on record | 2025-06-05 | arxiv.org |
| LAION-5B and Re-LAION-5B | LAION e.V. (Amtsgericht Hamburg VR 25085) | image links with captions, no images | 5.85 billion pairs; Re-LAION-5B: 5,526,641,167 | metadata CC BY 4.0; Re-LAION-5B Apache-2.0 | taken down 2023-12-19; Kneschke v. LAION at the BGH | 2026-11-09 | laion.ai |
| The Stack | BigCode (collaboration, no legal entity named) | source code from public repositories | v2: 67.5 TB full, 32.1 TB deduplicated; 658 languages (card) vs 619 (paper) | not stated | v2.2.0 removed opt-outs received before 2026-07-29 | 2026-11-09 | huggingface.co |
| Mozilla Common Voice | Mozilla Data Collective (operating entity not named on its site) | read and spontaneous speech | v27.0: 295 languages, a little over 42,593 hours (its claim) | CC0 (community playbook); "a permissive licence" (v27 post) | none on record | 2026-09-21 | community.mozilladatacollective.com |
| Ego4D | Ego4D consortium | first-person video | 3,670 hours, 923 participants (its claim) | not stated | none on record | 2022-02-17 | ego4d-data.org |
| Open X-Embodiment | research consortium (21 institutions and 34 labs on the same page) | robot trajectories | 1M+ trajectories, 22 embodiments | not stated | none on record | 2026-11-09 | robotics-transformer-x.github.io |
| Dataset Licenses for AI Training | topic page | CC, ODC, CDLA and custom terms on datasets | n/a | ODC-BY 1.0 covers the database only (s. 2.4) | n/a | 2026-11-09 | opendatacommons.org |
Corpora built from the web crawl: Common Crawl and the sets filtered from it
Most text corpora start from Common Crawl, a nonprofit archive of the public web, and differ in which pages they keep and how they filter them.
The Common Crawl Foundation has collected crawls since 2008; its latest, CC-MAIN-2026-43, holds 2.16 billion pages according to its announcement of 2026-10-21. C4, FineWeb, Dolma, RedPajama, LAION-5B, DataComp and The Pile's Pile-CC subset all draw on its archives.
Each filtered set keeps a different slice of the crawl. C4, described in Google's T5 paper of 2019, filters an April 2019 snapshot to about 750 GB of English text, and the allenai copy on Hugging Face holds 305 GB in 364,868,892 training documents.
FineWeb, from Hugging Face, is built from 110 Common Crawl dumps, from CC-MAIN-2013-20 to CC-MAIN-2025-26. Together Computer counts 878 billion Common Crawl tokens in RedPajama-Data-1T, and over 100 billion documents from 84 snapshots in RedPajama-V2, of which 20 billion are unique, in five languages.
Ai2 released Dolma v1.7 on 2024-04-15 at 4.5 TB gzip-compressed, and states that its later Dolma 3 pool holds over 9 trillion tokens. The DataComp project counts 240 trillion tokens in DCLM-Pool, and its DCLM-Baseline set holds 4 trillion tokens in 3 billion documents.
OpenAI's GPT-2 paper of 2019 describes WebText, a set selected from outbound Reddit links with at least 3 karma: slightly over 8 million documents and 40 GB.
OpenWebText, published on the Skylion007 project page, holds 38 GB from 8,013,769 documents. The crawl itself, its crawler and its full terms belong to the Common Crawl page linked from the table.
Book and mixed-source corpora: what The Pile and Books3 held and what replaced them
Book text entered open training data mainly through Books3, a collection derived from a private torrent tracker, and The Pile, the mixed corpus that packaged it. EleutherAI's Pile paper, dated 2020-12-31, describes 22 subsets and gives the size as 825 GiB in its body, while its title says 800GB.
The paper lists Books3 at 100.96 GiB of raw text with a 12.07 % weight in the mix and 1.5 epochs, and names the Bibliotik tracker as its origin.
Books3 no longer sits on its original host. The Danish rights group RettighedsAlliancen announced on 2023-08-14 that it had the collection removed in mid-July 2023, and it puts the collection at about 200,000 books, its own figure.
Together Computer marks the book configuration of RedPajama-Data-1T as "defunct due to reported copyright infringement for Books3". Book text licensed title by title, with its fees and splits, is covered under books as AI training data.
Openly licensed and library corpora are the replacements on record. EleutherAI and collaborators released Common Pile v0.1 on 2025-06-05, 8 TB of openly licensed text from 30 sources.
Pleias SAS states that Common Corpus, first released in November 2024, holds 2,267,302,720,836 tokens of openly licensed and public domain text, while its paper (arXiv 2506.01732) says about two trillion.
The Institutional Data Initiative at Harvard Law School Library describes Institutional Books 1.0 in a paper of 2025-06-10: 983,004 of 1,075,899 scanned volumes, 242 billion tokens.
Image, code, speech and robotics corpora
Outside text, the public corpora that matter most for data holders are LAION's image-text link sets, BigCode's The Stack, Mozilla's Common Voice and two robotics collections.
LAION-5B, announced by LAION e.V. of Hamburg on 2022-03-31, lists 5.85 billion image links with captions, 2.3 billion of them in English, and holds no images: those stay on their own sites under their own copyright.
The Re-LAION-5B page gives the original count as 5.8 billion, and LAION took LAION-5B down on 2023-12-19 before releasing Re-LAION-5B with 5,526,641,167 pairs on 2024-08-30.
DataComp, a research collaboration publishing as mlfoundations, offers CommonPool, a candidate pool of 12.8 billion image-text pairs described in its paper of 2023-04-27.
The Stack is BigCode's collection of source code from public repositories. BigCode states that v1 held 6.4 TB in 358 programming languages; the v2 card gives 67.5 TB in full and 32.1 TB deduplicated, and counts 658 languages where the v2 paper (arXiv 2402.19173) says 619 programming languages.
Code licensed directly, outside public repositories, is covered under licensing source code to AI companies.
Mozilla Common Voice collects read and spontaneous speech. Its Scripted Speech release v27.0 of 2026-09-21 covers 295 languages and, by the Mozilla Data Collective's own count, a little over 42,593 hours.
Two collections supply physical-world data. The Ego4D consortium states 3,670 hours of first-person video from 923 participants in 74 locations across 9 countries (2022-02-17). Open X-Embodiment states more than 1 million robot trajectories from 22 embodiments, and its project page names 21 institutions in one place and 34 labs in another.
Which terms travel with each corpus
Each public corpus carries its publisher's terms, and those terms often govern only the compilation, not the pages, books or images inside it. The Open Data Commons Attribution License (ODC-BY 1.0), used by C4 and Dolma, states in section 2.4 that it covers rights in the database, not its individual contents.
The Data Provenance Initiative's audit of more than 1,800 text datasets, published 2023-10-25, found licenses omitted on hosting sites in more than 70 % of cases. How rights pass, or fail to pass, along a dataset's history is traced under training data provenance.
Common Crawl's Terms of Use, last updated 2024-03-07, are terms of use, not an open-source license. The allenai copy of C4 carries ODC-BY plus Common Crawl's terms of use.
Dolma moved to ODC-BY on 2024-04-15, and Ai2 limits Dolma 3 Mix to research and educational use under the same license. DCLM-Baseline is released under CC BY 4.0 for research purposes only, and OpenWebText's packaging is CC0.
Book, image and speech sets show the widest spread. Institutional Books requires anyone who wants commercial use, including AI model training, to contact IDI first, and offers noncommercial Early-Access terms otherwise. LAION-5B's metadata is CC BY 4.0 and Re-LAION-5B is Apache-2.0.
Common Voice is CC0 according to Mozilla's community playbook, while the v27 release post calls it "a permissive licence".
The terms of The Pile, FineWeb, The Stack, Common Corpus, RedPajama, Ego4D and Open X-Embodiment are not stated here, and no license on a compilation is read on this page as a license to the works inside it.
Court records that name a training dataset
Nine lawsuits in the case tracker name a public training dataset or a books dataset in a complaint, ruling or filing, and this page gives one line per case.
Books3 is the dataset named most often, in Bartz v. Anthropic, Huckabee v. Bloomberg, Nazemian v. NVIDIA and In re Salesforce, while LAION-5B appears in Kneschke v.
LAION, Andersen v. Stability AI and the UK judgment in Getty Images v. Stability AI. What a complaint says is an allegation until a court rules on it. Rulings, settlements and next dates for each case sit in the tracker of AI copyright lawsuits over training data.
The table gives each case's court, docket, dataset named and status as of 2026-11-09, US cases first, with the docket entry or court page as the source.
| Case | Court and docket | Dataset named | Status as of 2026-11-09 | Source |
|---|---|---|---|---|
| Andersen v. Stability AI | N.D. Cal. 3:23-cv-00201-WHO | LAION-based training of image models (complaint) | DMCA claims dismissed 2024-08-12 (Dkt. 223), copyright claims survive; discovery | order, Dkt. 223 |
| Kadrey v. Meta | N.D. Cal. 3:23-cv-03417-VC | books from shadow libraries; Books3 testimony in dispute (Dkt. 879) | summary judgment for Meta on training, on the record, 2025-06-25 (Dkt. 598); related Entrepreneur trial set for 2027-05-03 | Dkt. 598; Dkt. 879 |
| Huckabee v. Bloomberg | S.D.N.Y. 1:23-cv-09152-MMG | Books3 for BloombergGPT (complaint) | motion to dismiss denied 2025-11-24; later entries not checked | opinion and order of 2025-11-24 |
| In re Mosaic LLM Litigation | N.D. Cal. 3:24-cv-01451 | books datasets for the MPT and DBRX models (complaint) | cross-motions for summary judgment filed 2026-06-30; ruling pending | govinfo |
| Nazemian v. NVIDIA | N.D. Cal. 4:24-cv-01454-JST | Books3 for NeMo Megatron (complaint) | discovery; fact discovery closes 2027-02-12 (Dkt. 407) | Dkt. 407 |
| Bartz v. Anthropic | N.D. Cal. 3:24-cv-05417-AMO | Books3, LibGen and PiLiMi copies | settled: USD 1.5 billion fund, final approval and judgment 2026-07-20 (Dkt. 680) | final approval order, Dkt. 680 |
| In re Salesforce (Tanzer v. Salesforce) | N.D. Cal. 3:25-cv-08862-CRB | The Pile, RedPajama and Books3 (complaint) | pending | complaint of 2025-10-15 |
| Getty Images v. Stability AI (UK) | High Court [2025] EWHC 2863 (Ch); Court of Appeal CA-2026-000007 | LAION-5B, 5.85 billion pairs, cited in the judgment | secondary infringement dismissed 2025-11-04; permission to appeal 2025-12-16; appeal part heard 2026-11-11, judgment reserved 2026-11-12 (case tracker, checked 2026-11-16) | caselaw.nationalarchives.gov.uk; casetracker.justice.gov.uk |
| Kneschke v. LAION | BGH I ZR 281/25, from LG Hamburg 310 O 227/23 and OLG Hamburg 5 U 104/24 | LAION-5B | LG Hamburg judgment 2024-09-27; BGH hearing 2026-09-03, pronouncement scheduled for 2026-12-17 | bundesgerichtshof.de |
No court has ruled that a dataset publisher's license makes training on the works inside the dataset lawful. The German Federal Court of Justice (BGH) has set its pronouncement in Kneschke v. LAION for 2026-12-17, so no BGH ruling exists yet.
What inclusion in a public corpus means for the holder of the content
Once content is copied into a public corpus, the holder can stop future copies and request removal from hosted versions, but cannot recall copies already downloaded.
Common Crawl shows the limit: in a statement of 2025-11-04 it says that its WARC archive files are immutable and that affected URLs are removed or filtered from later crawls and public indexes, so its opt-out works forward, not backward.
Removal from a hosted version depends on each publisher. The Stack v2.2.0 removed opt-out requests received before 2026-07-29. LAION states that 2,236 links matched partner hash lists, an upper bound, and its research-safe variant of Re-LAION-5B removes 3.044 % of samples.
Torstar's cease-and-desist letter to Hugging Face over FineWeb, dated 2025-01-22, listed 361 domains and asked for a response by 2025-01-31, and Hugging Face's public folder of takedown notices held 29 files for 2025 and 30 for 2026 up to 2026-10-12 (counted 2026-11-09).
Settings on the platforms that hold a user's content are a separate route, covered under opting out of AI training.
Consent signals have also shifted after the copying. Longpre et al., in the Consent in Crisis audit published 2024-07-20, found that robots.txt fully restricted more than 5 % of C4's tokens and more than 28 % of its most critical sources in 2023 to 2024, and that terms of service restricted 45 % of C4, across about 14,000 audited web domains. Those restrictions came years after the April 2019 snapshot behind C4.
Money has reached holders of content in public corpora through litigation and licensing, not through the corpora. In Bartz v. Anthropic the court found on 2025-06-23 (Order on Fair Use, Dkt. 231) that training was fair use but that pirated library copies were not excused.
The USD 1.5 billion non-reversionary settlement, approved with final judgment on 2026-07-20 (Dkt. 680), covers 482,460 works, of which 440,490 had been claimed as of 2026-04-16.
Disclosure runs the other way for licensed data. Under Article 53(1)(d) of the EU AI Act (Regulation (EU) 2024/1689), which applies from 2025-08-02, the Commission template C(2025) 8311 shows licensed datasets only as Yes or No plus modality, while summaries such as the Grok 4.7 one name public datasets.
A holder can therefore trace a public corpus more easily than a private license. Takedowns, opt-out lists and removal forms for each corpus are set out in removing content from AI training datasets.
Benefits, risks, downsides and alternatives for a holder whose content sits in a public dataset, including not licensing
A holder that finds its content in a public corpus can do nothing, object, license the same material on its own terms, or combine these, and each choice has a cost.
Benefits
- Removal from hosted versions and future crawls: The Stack and Common Crawl both act on opt-out requests for later versions.
- A documented record: the Bartz settlement counted works by a list, so a work identified in a dataset can support a later claim.
- Terms the holder sets: a curated copy licensed directly carries a scope and price of the holder's choosing.
Risks
- Removal does not reach copies already made or the sets built from them.
- A takedown notice must name each domain or work, as Torstar's notice to Hugging Face did.
- Courts have ruled for the developer on training, in Bartz and in Kadrey (Dkt. 598, 2025-06-25), where Meta won summary judgment on the record.
How fair use, the EU opt-out and other exceptions treat such copying differs by country, as compared in copyright and AI training across jurisdictions.
Downsides
- Monitoring several corpora and hosts is ongoing work, because each publisher issues its own versions, cards and notices.
- An opt-out can remove content from uses the holder wanted, such as research or open-license collections.
Alternatives
- Do nothing and leave the content where it is.
- Record an opt-out with Common Crawl so that later crawls skip the content.
- Publish the material under an open license with stated terms, as the sources of Common Pile do.
- License a curated version directly to a developer, or decide not to license at all.
Each of these routes, including not licensing, is weighed against a sale in alternatives to selling data to AI labs, including not selling at all.
Where public datasets meet the other routes, provenance and the law
Public datasets are one route to training data among several; the pages below cover the other routes, the market and the law that decides what a dataset may contain.
- How AI companies get training data, from crawling and licensing to users, contractors and synthetic data.
- AI crawlers that feed the crawl archives, and how to allow, block or charge them.
- The EU text and data mining exception behind the Kneschke case.
- Fair use and AI training in US law.
- The AI training data market, for figures on market size.
- Synthetic data for AI training, including data generated by LLMs.
- How to sell a dataset that a company holds itself.
- The AI training data glossary, for terms such as corpus, crawl archive and dataset card.
- Pirated books and shadow libraries in AI training.
- Public domain and openly licensed material in AI training.
This page is information, not legal advice.
Frequently asked questions
How big is LLM training data?
There is no single size, because publishers count in different units: pages, bytes, documents or tokens. OpenWebText is measured in gigabytes, The Stack v2 in terabytes, and Ai2 counts its Dolma 3 pool in tokens. The sizes in the table above are each publisher's own statement with its date, and some publishers contradict their own papers.
Is open-source LLM training data free to use commercially?
Not always: several open corpora restrict commercial use, and most licenses cover only the compilation. DCLM-Baseline is licensed for research purposes only, Dolma 3 Mix for research and educational use, and Institutional Books asks commercial users to contact IDI first. ODC-BY covers the database, under section 2.4, not the works inside it.
What does LLM training data look like?
A record in a public corpus is usually a web page, a text document or an image link with its caption. Common Crawl stores its raw crawls as WARC files, C4 holds filtered English documents, and LAION-5B holds image links with captions rather than the images themselves.
Can I find out whether my content is in a training dataset?
Partly: some publishers keep public records, but no single search covers every corpus. Common Crawl publishes its Opt-Out Ledger, The Stack's card gives the cutoff of its latest opt-out removal, and Hugging Face publishes the takedown notices it receives. Licensed private datasets appear in EU summaries only as Yes or No.
Why are some well-known datasets missing from this page?
A dataset is listed only when its publisher's own card, paper or page supports each entry in its row. RefinedWeb is not listed for that reason, since no primary record of its contents, size or license is cited here. The training mixes of individual models are not listed either, because this page covers datasets rather than models.