DataPacts

Common Pile: What It Contains, License, Legal Status

Checked

At a glance

The Common Pile v0.1 is a text dataset for training language models, released on June 5, 2025 by EleutherAI and collaborators, which its paper's abstract describes as an eight terabyte collection of openly licensed text, with public domain text included, from 30 sources.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

It is not the Pile of 2020, a different, mixed corpus that EleutherAI's own post calls the predecessor of the Common Pile.

This page reads the Common Pile from the side of the people whose work it holds. Within AI training data licensing, it is openly licensed text, the unpaid counterpart to licensed text: inclusion brings an author no payment and no contract. Two 7B Comma v0.1 models were trained on it.

Who Built the Common Pile v0.1: EleutherAI, Its Collaborators and the 2025 Paper

The Common Pile v0.1 was built by EleutherAI with researchers from the University of Toronto, the Vector Institute, Hugging Face and other institutions, and is described in an arXiv paper of June 5, 2025. The table lists the identity facts as of 2026-11-17.

Attribute Value Source
Built by EleutherAI with collaborators; legal entity not stated blog.eleuther.ai/common-pile/
Collaborators named University of Toronto and Vector Institute, Hugging Face, Allen Institute for AI, Teraflop AI, Cornell, MIT, CMU, Lila Sciences, poolside, University of Maryland, Lawrence Livermore National Laboratory blog.eleuther.ai
Paper "The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text", arXiv 2506.05209, v1 2025-06-05, 27 authors arxiv.org/abs/2506.05209
Release 2025-06-05 blog.eleuther.ai; arxiv.org
Host Hugging Face, Inc., organization "common-pile", 63 dataset repositories, none gated huggingface.co
Code GitHub r-three/common-pile, MIT License (code) github.com
Funding named Mozilla Foundation, Sutter Hill Ventures; NSERC; Lawrence Livermore National Laboratory under a US Department of Energy contract arxiv.org

The paper lists 27 authors, and the first three are marked as equal contributors from the University of Toronto and the Vector Institute. The blog post says the collection took about two years of curation.

EleutherAI is written here as a brand because its site names no legal entity. Hugging Face, Inc. hosts the files and did not create the corpus. The blog calls the Common Pile the successor to The Pile, EleutherAI's 2020 corpus.

The 30 Sources of the Common Pile and the Rights Basis Stated for Each

The Common Pile draws on 30 sources in ten domains, from research papers and US patents to public domain books, wikis, open source code and transcribed YouTube videos. The table follows the paper's Appendix B grouping, with raw and filtered sizes from its Table 4, as of 2026-11-17; sizes are UTF-8 gigabytes as printed.

Domain Source Rights basis as stated Raw GB Filtered GB Source
Scholarly peS2o CC BY, CC BY-SA, CC0 or public domain 188.2 182.6 arXiv 2506.05209
Scholarly PubMed Central journal-designated CC BY, CC BY-SA or CC0 158.9 147.1 same
Scholarly ArXiv Papers CC BY, CC BY-SA or CC0 21 19 same
Scholarly ArXiv Abstracts CC0 metadata under arXiv policy 2.4 2.4 same
Forums StackExchange CC BY-SA 103.7 89.7 same
Forums GitHub Archive repository license on the Blue Oak Council list 54.7 40.4 same
Forums Ubuntu IRC public domain 6.3 5.3 same
Government USGPO (GovInfo) US government works 74.5 36.1 same
Government USPTO patents as US government works 1,003.4 661.1 same
Government Caselaw Access Project and Court Listener public domain documents only 78 77 same
Government UK Hansard Open Parliament License 10 9.6 same
Government Regulations.gov US federal agency documents 6.1 5.1 same
Task data Data Provenance Initiative audited open license, public domain sources 7 3 same
Books Biodiversity Heritage Library license metadata, public domain 96 35 same
Books Pre-1929 Books (Internet Archive, HathiTrust) US publication before 1929 73.8 46.3 same
Books Library of Congress "Selected Digitized Books", public domain 47.8 35.6 same
Books Project Gutenberg public domain per metadata 26.2 20.1 same
OER Directory of Open Access Books CC BY or CC BY-SA 12.5 12 same
OER PressBooks public domain, CC BY or CC BY-SA 1.5 0.6 same
OER OERCommons public domain, CC BY or CC BY-SA 0.1 0.05 same
OER LibreTexts public domain, CC BY, CC BY-SA or GFDL 5.3 3.6 same
Wikis Wikimedia (eight English wikis) CC BY-SA 90.5 57.4 same
Wikis Wikiteam (about 330,000 wikis) CC BY, CC BY-SA or public domain 437.5 13.7 same
Code Stack V2 (openly licensed subset) licenses detected by the Stack V2 creators, all open 4,774.7 259.9 same
Code Python Enhancement Proposals public domain 0.01 0.01 same
Audio Creative Commons YouTube CC BY 21.5 18.6 same
Web CCCC (Creative Commons Common Crawl) CC BY, CC BY-SA or CC0 marker; 537 domains checked 260 58 same
Web Foodista CC BY 0.09 0.08 same
Web News (Open Newswire sites) CC BY or CC BY-SA 0.4 0.3 same
Web Public Domain Review CC BY-SA 0.007 0.007 same
Total 30 sources 7,557.9 1,838.3 same, Table 4

The rights basis is the builders' statement for each source, not a registry's or a court's finding.

US federal government works carry no copyright under 17 U.S.C. 105(a), which is the basis for the patent and GovInfo rows, and the paper says books published in the US before 1929 passed into the public domain on January 1, 2024. The other texts carry CC BY, CC BY-SA, CC0 or Blue Oak Council licenses.

On scale, the paper says the web set came from 52 Common Crawl snapshots, with the top 1,000 domains checked by hand and 537 kept.

The YouTube set covers over 2,000 channels and over 1.1 million videos, transcribed with Whisper. How copyright terms differ by country is covered on the page about public domain and openly licensed material in AI training.

How Big the Common Pile Is: 8 TB in the Title, 7.6 TB in the Text, 1.8 TB After Filtering

The Common Pile is about 7.6 terabytes of raw text, which the paper's title rounds to 8TB, and 1.8 terabytes remain after the builders' filtering and deduplication. Table 4 of the paper totals 7,557.9 GB raw in 692,854,953 documents and 1,838.3 GB filtered in 233,817,169 documents.

Size is uneven by source: the openly licensed part of Stack V2 is 4,774.7 GB raw and 259.9 GB filtered, and USPTO patents are 1,003.4 GB raw, which the paper calls its second-largest source.

The paper estimates in its Appendix I that approximately half of the Common Pile, around 3.8TB, was created since 2020. No token count is published for the raw corpus.

The paper also compares the size with other open corpora in its section 2.2: the Open License Corpus at 0.85 TB from 12 sources, KL3M at 3 TB, and Common Corpus at 7.4 TB against the Common Pile's 7.6 TB.

It adds that Common Corpus does not retain full per-document licensing information across all sources and incorporates OpenAlex, which it calls known to provide inaccurate licensing information. Pleias's dataset card states a license field per document, and both are their authors' statements. The page on Common Corpus gives the other side.

The Open License Rule Behind the Common Pile and What It Excluded

The Common Pile counts a text as openly licensed when its license meets the Open Knowledge Foundation's Open Definition 2.1, which permits anyone to use, modify and share the work for any purpose. The table sets out the paper's rules as of 2026-11-17, in the order of its section 2.

Rule What the paper states Source
Open license test Open Knowledge Foundation Open Definition 2.1; examples CC BY, CC BY-SA, Blue Oak Council licenses such as MIT arXiv 2506.05209, s. 2
Excluded licenses CC NC and ND, not open under that definition same
License laundering only sources where the license came from the copyright holder; OpenAlex, YouTube Commons, Hacker News on Kaggle excluded s. 2.1
Collection licenses ODC-By on a corpus does not extend to its documents s. 2.1
Synthetic text LLM-generated datasets avoided s. 2.1
Caveats licensing collected late 2024; owners may change licenses; unclear quoted material possible s. 2.1

NC and ND licenses fail the definition and were left out. The builders included a source only where the licensing was provided by the copyright holder, so they excluded OpenAlex, YouTube Commons and the Hacker News dataset on Kaggle over what the paper calls license laundering, and they avoided text generated by an LLM.

A license on a whole collection does not license the documents inside it. In the paper's words, ODC-By, by definition, does not extend to individual documents within the corpus, and it names Dolma, FineWeb and TxT360 as examples.

The blog adds that no widely used license mentions AI explicitly. Collection licenses such as ODC-By are compared on the page about dataset licenses for AI training.

License Labels in the Common Pile and What Its Builders Do Not Guarantee

Each document in the Common Pile carries its own license label, and the builders state that these labels may be wrong in some cases. The Stack V2 card puts the per-document license in the metadata field.

The Hugging Face repository record, read on 2026-11-17, shows a license tag on 1 of the 63 dataset repositories, and the Comma dataset card has no license metadata.

The Library of Congress card says its overall license is "intended to be CC-BY-4.0, but this is not a guarantee for individual documents", while the paper describes that collection as public domain books. Both statements are shown here as written.

The paper lists its own caveats. It says the builders collected and curated the Common Pile in late 2024, so licensing may not be completely aligned with more recent updates by owners.

It says public domain documents may contain material of unclear status, and it says attributing model predictions to training data remains an active area of research.

The Comma model cards go further: they say "we cannot make a guarantee" that the models were trained exclusively on openly licensed text, citing license laundering and inaccurate metadata.

The collection code on GitHub is MIT-licensed, which covers the code and not the text. This page does not call the Common Pile license-safe, copyright-free or legally clean, because its builders do not.

Comma v0.1: The Two 7B Models Trained on Common Pile Text

Comma v0.1-1T and Comma v0.1-2T are two 7-billion-parameter language models that the Common Pile team trained on a filtered mix of its text, for 1 trillion and 2 trillion tokens. The table compares them as of 2026-11-17.

Attribute Comma v0.1-1T Comma v0.1-2T Source
Parameters 7,002,656,768 (BF16) 7,002,656,768 (BF16) Hub API
Training tokens 965 billion plus 35 billion cool-down 1.93 trillion plus 75.5 billion cool-down model cards
Hardware 64 Nvidia H100 512 AMD MI300A model cards
License tag apache-2.0 apache-2.0 model cards
Repository created 2025-05-15 2025-06-05 Hub API
Languages English and 15 programming languages same model cards
Builders' caveat "we cannot make a guarantee" of exclusively openly licensed training text same model cards

The Comma dataset is the filtered, reweighted mix and not the Common Pile itself. Its card gives 463.6 billion tokens in the main stage and 1,034.4 billion effective tokens after repeats, with the 2T run repeating every source twice as often.

The tokenizer has a 64,000 vocabulary. The cool-down for the 1T model appears as three figures: 35 billion tokens on the model card, 37.7 billion in the paper's Appendix L, and 39.5 billion effective tokens on the dataset card. This page keeps all three and reconciles none.

The results are the authors' claims. The paper says the 1T model outperforms budget-matched baselines on over half of the benchmarks tested, and that the 2T model is competitive with OLMo, Llama 2 and DeepSeekLLM.

It also says Qwen3 8B, trained on 36 trillion tokens, shows superior performance across most benchmarks. The models are base models without alignment.

What Filtering Changed Before Training: Language, Toxicity, Personal Data and Duplicates

Before training Comma, the team filtered the raw Common Pile for language, quality, toxicity and personal data and removed near-duplicates, so the models saw less than a quarter of the raw bytes, comparing 1,838.3 GB with 7,557.9 GB. The paper describes these steps:

  • a fastText filter that keeps English text;
  • a quality classifier for web text and a filter for OCR errors;
  • two toxicity classifiers trained on Jigsaw data;
  • regular-expression redaction of email addresses, phone numbers and IP addresses, replaced with placeholders such as <EMAIL_ADDRESS>;
  • removal of duplicates where two documents share more than 90 % of their 20-grams.

Code was kept only in 15 programming languages and filtered for educational quality. Redaction by pattern is a different task from the one a holder faces before a license, described on the page about PII removal and de-identification before licensing.

No court record on file names the Common Pile, and none of the 59 text notices in Hugging Face's public takedown repository names it, as of November 17, 2026. The repository also holds 18 PDF notices, which were not read.

The one court record on file that named EleutherAI concerns the 2020 Pile's Books3, not the Common Pile. No EU training-content summary on file names the Common Pile either.

The only route the cards describe is a GitHub discussion for incorrect licensing, and they describe no opt-out or removal form. Routes across datasets are compared on the page about removing content from AI training datasets.

An Openly Licensed Work in the Common Pile: Benefits, Risks, Downsides and Alternatives for the Author

For an author whose CC BY paper, CC BY-SA wiki edit, YouTube talk or MIT-licensed code sits in the Common Pile, inclusion follows from the license the author chose, and it brings no payment and no contract.

Benefits

  • Per-document license labels travel with the text.
  • Source metadata and a public method make attribution easier to check.

Risks

  • Labels can be wrong, and the builders do not guarantee them.
  • YouTube talks enter as machine transcripts made with Whisper.
  • The cards describe no removal route.

Downsides

  • Inclusion brings no payment.
  • A license chosen earlier already permits this use where its terms allow.

Alternatives

  • Future works under NC or ND licenses fall outside the paper's rule; this is information, not a legal opinion.
  • Licensing text directly to AI developers, as covered under text corpora for AI training.
  • Reporting a wrong label through the builders' GitHub discussion, with no claim on the answer.
  • Not licensing at all.

Not licensing is a valid outcome, and nothing here suggests that inclusion in a corpus creates a claim to payment.

This page covers the Common Pile v0.1; other corpora, copyright rules, cases and the option of not licensing each have their own page.

This page is information, not legal advice.

Frequently Asked Questions About the Common Pile

What is the Common Pile?

The Common Pile v0.1 is EleutherAI and collaborators' dataset of public domain and openly licensed text. It was released on 2025-06-05 from 30 sources and is not the 2020 Pile.

How big is the Common Pile?

The paper's title says 8TB, and its Table 4 totals 7,557.9 GB raw and 1,838.3 GB filtered. All three figures are shown here as published.

What license does the Common Pile use?

The Common Pile has no single dataset license, because each document keeps its own. The builders say each license meets the Open Definition 2.1 and do not guarantee that the labels are correct.

Can the Common Pile be used commercially?

The builders publish no commercial terms for the dataset. Licenses meeting the Open Definition 2.1 allow use for any purpose, while conditions such as attribution and share-alike still apply. This page is information, not legal advice.