It is not the Pile of 2020, a different, mixed corpus that EleutherAI's own post calls the predecessor of the Common Pile.
This page reads the Common Pile from the side of the people whose work it holds. Within AI training data licensing, it is openly licensed text, the unpaid counterpart to licensed text: inclusion brings an author no payment and no contract. Two 7B Comma v0.1 models were trained on it.
Who Built the Common Pile v0.1: EleutherAI, Its Collaborators and the 2025 Paper
The Common Pile v0.1 was built by EleutherAI with researchers from the University of Toronto, the Vector Institute, Hugging Face and other institutions, and is described in an arXiv paper of June 5, 2025. The table lists the identity facts as of 2026-11-17.
| Attribute | Value | Source |
|---|---|---|
| Built by | EleutherAI with collaborators; legal entity not stated | blog.eleuther.ai/common-pile/ |
| Collaborators named | University of Toronto and Vector Institute, Hugging Face, Allen Institute for AI, Teraflop AI, Cornell, MIT, CMU, Lila Sciences, poolside, University of Maryland, Lawrence Livermore National Laboratory | blog.eleuther.ai |
| Paper | "The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text", arXiv 2506.05209, v1 2025-06-05, 27 authors | arxiv.org/abs/2506.05209 |
| Release | 2025-06-05 | blog.eleuther.ai; arxiv.org |
| Host | Hugging Face, Inc., organization "common-pile", 63 dataset repositories, none gated | huggingface.co |
| Code | GitHub r-three/common-pile, MIT License (code) | github.com |
| Funding named | Mozilla Foundation, Sutter Hill Ventures; NSERC; Lawrence Livermore National Laboratory under a US Department of Energy contract | arxiv.org |
The paper lists 27 authors, and the first three are marked as equal contributors from the University of Toronto and the Vector Institute. The blog post says the collection took about two years of curation.
EleutherAI is written here as a brand because its site names no legal entity. Hugging Face, Inc. hosts the files and did not create the corpus. The blog calls the Common Pile the successor to The Pile, EleutherAI's 2020 corpus.
The 30 Sources of the Common Pile and the Rights Basis Stated for Each
The Common Pile draws on 30 sources in ten domains, from research papers and US patents to public domain books, wikis, open source code and transcribed YouTube videos. The table follows the paper's Appendix B grouping, with raw and filtered sizes from its Table 4, as of 2026-11-17; sizes are UTF-8 gigabytes as printed.
| Domain | Source | Rights basis as stated | Raw GB | Filtered GB | Source |
|---|---|---|---|---|---|
| Scholarly | peS2o | CC BY, CC BY-SA, CC0 or public domain | 188.2 | 182.6 | arXiv 2506.05209 |
| Scholarly | PubMed Central | journal-designated CC BY, CC BY-SA or CC0 | 158.9 | 147.1 | same |
| Scholarly | ArXiv Papers | CC BY, CC BY-SA or CC0 | 21 | 19 | same |
| Scholarly | ArXiv Abstracts | CC0 metadata under arXiv policy | 2.4 | 2.4 | same |
| Forums | StackExchange | CC BY-SA | 103.7 | 89.7 | same |
| Forums | GitHub Archive | repository license on the Blue Oak Council list | 54.7 | 40.4 | same |
| Forums | Ubuntu IRC | public domain | 6.3 | 5.3 | same |
| Government | USGPO (GovInfo) | US government works | 74.5 | 36.1 | same |
| Government | USPTO | patents as US government works | 1,003.4 | 661.1 | same |
| Government | Caselaw Access Project and Court Listener | public domain documents only | 78 | 77 | same |
| Government | UK Hansard | Open Parliament License | 10 | 9.6 | same |
| Government | Regulations.gov | US federal agency documents | 6.1 | 5.1 | same |
| Task data | Data Provenance Initiative | audited open license, public domain sources | 7 | 3 | same |
| Books | Biodiversity Heritage Library | license metadata, public domain | 96 | 35 | same |
| Books | Pre-1929 Books (Internet Archive, HathiTrust) | US publication before 1929 | 73.8 | 46.3 | same |
| Books | Library of Congress | "Selected Digitized Books", public domain | 47.8 | 35.6 | same |
| Books | Project Gutenberg | public domain per metadata | 26.2 | 20.1 | same |
| OER | Directory of Open Access Books | CC BY or CC BY-SA | 12.5 | 12 | same |
| OER | PressBooks | public domain, CC BY or CC BY-SA | 1.5 | 0.6 | same |
| OER | OERCommons | public domain, CC BY or CC BY-SA | 0.1 | 0.05 | same |
| OER | LibreTexts | public domain, CC BY, CC BY-SA or GFDL | 5.3 | 3.6 | same |
| Wikis | Wikimedia (eight English wikis) | CC BY-SA | 90.5 | 57.4 | same |
| Wikis | Wikiteam (about 330,000 wikis) | CC BY, CC BY-SA or public domain | 437.5 | 13.7 | same |
| Code | Stack V2 (openly licensed subset) | licenses detected by the Stack V2 creators, all open | 4,774.7 | 259.9 | same |
| Code | Python Enhancement Proposals | public domain | 0.01 | 0.01 | same |
| Audio | Creative Commons YouTube | CC BY | 21.5 | 18.6 | same |
| Web | CCCC (Creative Commons Common Crawl) | CC BY, CC BY-SA or CC0 marker; 537 domains checked | 260 | 58 | same |
| Web | Foodista | CC BY | 0.09 | 0.08 | same |
| Web | News (Open Newswire sites) | CC BY or CC BY-SA | 0.4 | 0.3 | same |
| Web | Public Domain Review | CC BY-SA | 0.007 | 0.007 | same |
| Total | 30 sources | 7,557.9 | 1,838.3 | same, Table 4 |
The rights basis is the builders' statement for each source, not a registry's or a court's finding.
US federal government works carry no copyright under 17 U.S.C. 105(a), which is the basis for the patent and GovInfo rows, and the paper says books published in the US before 1929 passed into the public domain on January 1, 2024. The other texts carry CC BY, CC BY-SA, CC0 or Blue Oak Council licenses.
On scale, the paper says the web set came from 52 Common Crawl snapshots, with the top 1,000 domains checked by hand and 537 kept.
The YouTube set covers over 2,000 channels and over 1.1 million videos, transcribed with Whisper. How copyright terms differ by country is covered on the page about public domain and openly licensed material in AI training.
How Big the Common Pile Is: 8 TB in the Title, 7.6 TB in the Text, 1.8 TB After Filtering
The Common Pile is about 7.6 terabytes of raw text, which the paper's title rounds to 8TB, and 1.8 terabytes remain after the builders' filtering and deduplication. Table 4 of the paper totals 7,557.9 GB raw in 692,854,953 documents and 1,838.3 GB filtered in 233,817,169 documents.
Size is uneven by source: the openly licensed part of Stack V2 is 4,774.7 GB raw and 259.9 GB filtered, and USPTO patents are 1,003.4 GB raw, which the paper calls its second-largest source.
The paper estimates in its Appendix I that approximately half of the Common Pile, around 3.8TB, was created since 2020. No token count is published for the raw corpus.
The paper also compares the size with other open corpora in its section 2.2: the Open License Corpus at 0.85 TB from 12 sources, KL3M at 3 TB, and Common Corpus at 7.4 TB against the Common Pile's 7.6 TB.
It adds that Common Corpus does not retain full per-document licensing information across all sources and incorporates OpenAlex, which it calls known to provide inaccurate licensing information. Pleias's dataset card states a license field per document, and both are their authors' statements. The page on Common Corpus gives the other side.
The Open License Rule Behind the Common Pile and What It Excluded
The Common Pile counts a text as openly licensed when its license meets the Open Knowledge Foundation's Open Definition 2.1, which permits anyone to use, modify and share the work for any purpose. The table sets out the paper's rules as of 2026-11-17, in the order of its section 2.
| Rule | What the paper states | Source |
|---|---|---|
| Open license test | Open Knowledge Foundation Open Definition 2.1; examples CC BY, CC BY-SA, Blue Oak Council licenses such as MIT | arXiv 2506.05209, s. 2 |
| Excluded licenses | CC NC and ND, not open under that definition | same |
| License laundering | only sources where the license came from the copyright holder; OpenAlex, YouTube Commons, Hacker News on Kaggle excluded | s. 2.1 |
| Collection licenses | ODC-By on a corpus does not extend to its documents | s. 2.1 |
| Synthetic text | LLM-generated datasets avoided | s. 2.1 |
| Caveats | licensing collected late 2024; owners may change licenses; unclear quoted material possible | s. 2.1 |
NC and ND licenses fail the definition and were left out. The builders included a source only where the licensing was provided by the copyright holder, so they excluded OpenAlex, YouTube Commons and the Hacker News dataset on Kaggle over what the paper calls license laundering, and they avoided text generated by an LLM.
A license on a whole collection does not license the documents inside it. In the paper's words, ODC-By, by definition, does not extend to individual documents within the corpus, and it names Dolma, FineWeb and TxT360 as examples.
The blog adds that no widely used license mentions AI explicitly. Collection licenses such as ODC-By are compared on the page about dataset licenses for AI training.
License Labels in the Common Pile and What Its Builders Do Not Guarantee
Each document in the Common Pile carries its own license label, and the builders state that these labels may be wrong in some cases. The Stack V2 card puts the per-document license in the metadata field.
The Hugging Face repository record, read on 2026-11-17, shows a license tag on 1 of the 63 dataset repositories, and the Comma dataset card has no license metadata.
The Library of Congress card says its overall license is "intended to be CC-BY-4.0, but this is not a guarantee for individual documents", while the paper describes that collection as public domain books. Both statements are shown here as written.
The paper lists its own caveats. It says the builders collected and curated the Common Pile in late 2024, so licensing may not be completely aligned with more recent updates by owners.
It says public domain documents may contain material of unclear status, and it says attributing model predictions to training data remains an active area of research.
The Comma model cards go further: they say "we cannot make a guarantee" that the models were trained exclusively on openly licensed text, citing license laundering and inaccurate metadata.
The collection code on GitHub is MIT-licensed, which covers the code and not the text. This page does not call the Common Pile license-safe, copyright-free or legally clean, because its builders do not.
Comma v0.1: The Two 7B Models Trained on Common Pile Text
Comma v0.1-1T and Comma v0.1-2T are two 7-billion-parameter language models that the Common Pile team trained on a filtered mix of its text, for 1 trillion and 2 trillion tokens. The table compares them as of 2026-11-17.
| Attribute | Comma v0.1-1T | Comma v0.1-2T | Source |
|---|---|---|---|
| Parameters | 7,002,656,768 (BF16) | 7,002,656,768 (BF16) | Hub API |
| Training tokens | 965 billion plus 35 billion cool-down | 1.93 trillion plus 75.5 billion cool-down | model cards |
| Hardware | 64 Nvidia H100 | 512 AMD MI300A | model cards |
| License tag | apache-2.0 | apache-2.0 | model cards |
| Repository created | 2025-05-15 | 2025-06-05 | Hub API |
| Languages | English and 15 programming languages | same | model cards |
| Builders' caveat | "we cannot make a guarantee" of exclusively openly licensed training text | same | model cards |
The Comma dataset is the filtered, reweighted mix and not the Common Pile itself. Its card gives 463.6 billion tokens in the main stage and 1,034.4 billion effective tokens after repeats, with the 2T run repeating every source twice as often.
The tokenizer has a 64,000 vocabulary. The cool-down for the 1T model appears as three figures: 35 billion tokens on the model card, 37.7 billion in the paper's Appendix L, and 39.5 billion effective tokens on the dataset card. This page keeps all three and reconciles none.
The results are the authors' claims. The paper says the 1T model outperforms budget-matched baselines on over half of the benchmarks tested, and that the 2T model is competitive with OLMo, Llama 2 and DeepSeekLLM.
It also says Qwen3 8B, trained on 36 trillion tokens, shows superior performance across most benchmarks. The models are base models without alignment.
What Filtering Changed Before Training: Language, Toxicity, Personal Data and Duplicates
Before training Comma, the team filtered the raw Common Pile for language, quality, toxicity and personal data and removed near-duplicates, so the models saw less than a quarter of the raw bytes, comparing 1,838.3 GB with 7,557.9 GB. The paper describes these steps:
- a fastText filter that keeps English text;
- a quality classifier for web text and a filter for OCR errors;
- two toxicity classifiers trained on Jigsaw data;
- regular-expression redaction of email addresses, phone numbers and IP addresses, replaced with placeholders such as
<EMAIL_ADDRESS>; - removal of duplicates where two documents share more than 90 % of their 20-grams.
Code was kept only in 15 programming languages and filtered for educational quality. Redaction by pattern is a different task from the one a holder faces before a license, described on the page about PII removal and de-identification before licensing.
Legal Standing of the Common Pile: Court Records, Notices and Correction Routes
No court record on file names the Common Pile, and none of the 59 text notices in Hugging Face's public takedown repository names it, as of November 17, 2026. The repository also holds 18 PDF notices, which were not read.
The one court record on file that named EleutherAI concerns the 2020 Pile's Books3, not the Common Pile. No EU training-content summary on file names the Common Pile either.
The only route the cards describe is a GitHub discussion for incorrect licensing, and they describe no opt-out or removal form. Routes across datasets are compared on the page about removing content from AI training datasets.
An Openly Licensed Work in the Common Pile: Benefits, Risks, Downsides and Alternatives for the Author
For an author whose CC BY paper, CC BY-SA wiki edit, YouTube talk or MIT-licensed code sits in the Common Pile, inclusion follows from the license the author chose, and it brings no payment and no contract.
Benefits
- Per-document license labels travel with the text.
- Source metadata and a public method make attribution easier to check.
Risks
- Labels can be wrong, and the builders do not guarantee them.
- YouTube talks enter as machine transcripts made with Whisper.
- The cards describe no removal route.
Downsides
- Inclusion brings no payment.
- A license chosen earlier already permits this use where its terms allow.
Alternatives
- Future works under NC or ND licenses fall outside the paper's rule; this is information, not a legal opinion.
- Licensing text directly to AI developers, as covered under text corpora for AI training.
- Reporting a wrong label through the builders' GitHub discussion, with no claim on the answer.
- Not licensing at all.
Not licensing is a valid outcome, and nothing here suggests that inclusion in a corpus creates a claim to payment.
Other Open Corpora, Copyright Terms and License Families Beyond This Page
This page covers the Common Pile v0.1; other corpora, copyright rules, cases and the option of not licensing each have their own page.
- Corpora: major AI training datasets, Dolma, FineWeb, what Common Crawl contains, Institutional Books and, mentioned only, The Stack.
- Data types: scientific data and papers as AI training data and open-source license contamination.
- Law and sources: copyright and AI training across jurisdictions and licensed vs scraped training data.
- Cases and options: Huckabee v. Bloomberg, AI copyright lawsuits over training data and alternatives to selling data, including not selling at all.
This page is information, not legal advice.
Frequently Asked Questions About the Common Pile
What is the Common Pile?
The Common Pile v0.1 is EleutherAI and collaborators' dataset of public domain and openly licensed text. It was released on 2025-06-05 from 30 sources and is not the 2020 Pile.
How big is the Common Pile?
The paper's title says 8TB, and its Table 4 totals 7,557.9 GB raw and 1,838.3 GB filtered. All three figures are shown here as published.
What license does the Common Pile use?
The Common Pile has no single dataset license, because each document keeps its own. The builders say each license meets the Open Definition 2.1 and do not guarantee that the labels are correct.
Can the Common Pile be used commercially?
The builders publish no commercial terms for the dataset. Licenses meeting the Open Definition 2.1 allow use for any purpose, while conditions such as attribution and share-alike still apply. This page is information, not legal advice.