Pleias first released it in November 2024 on Hugging Face, and the accompanying paper puts its size at about two trillion tokens.
This page reads Common Corpus from the side of the people whose work it holds. Within AI training data licensing, it is public domain and openly licensed text, the unpaid counterpart to licensed text: inclusion brings an author no payment and no contract.
Size, languages and licenses are given as the card and the paper state them, and the two differ in places.
Who Built Common Corpus: Pleias, Its Partners and the Paper
Common Corpus was created by Pleias, a French AI lab, which says on the dataset card that it built the corpus in association with several partners. The table lists the identity facts as of 2026-11-17.
| Attribute | Value | Source |
|---|---|---|
| Created by | Pleias, "in association with several partners" (card) | huggingface.co/datasets/PleIAs/common_corpus |
| First release | November 2024 (card); repository created 2024-11-12 | huggingface.co |
| Paper | "Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training", arXiv 2506.01732, v1 2025-06-02, v4 2026-10-05 | arxiv.org/abs/2506.01732 |
| Support named | AI Alliance; French Ministry of Culture (ALT-EDIC prefiguration); partnerships with Wikimedia Enterprise and Wikidata/Wikimedia Germany | dataset card; arxiv.org |
| Host | Hugging Face, Inc.; repository not gated, last modified 2026-05-06 | huggingface.co |
| Stated purpose | pre-training of "fully open and auditable LLMs" (paper) | arxiv.org |
| Legal name and registry | not stated | n/a |
The paper has ten authors, with Pleias in Paris as the lead affiliation. Its first version appeared on 2025-06-02 and its fourth on 2026-10-05, and the card calls it an ICLR 2026 oral.
Beyond the partnerships in the table, the paper also names LANGU:IA, DINUM and the Current AI initiative. Hugging Face, Inc. hosts the repository and did not create the dataset. Common Corpus is one of the public corpora compared on the page about major AI training datasets.
What Common Corpus Holds: Six Collections, Two Token Counts and Record Fields
Common Corpus is split into six collections, OpenCulture, OpenGovernment, OpenSource, OpenScience, OpenWeb and OpenSemantic, holding books and newspapers, legal and financial documents, code, scientific articles, web text and Wikidata statements. On the card, OpenCulture draws on public domain books and newspapers, Wikisource and Gutenberg.
OpenGovernment holds Finance Commons (SEC and WTO documents) and Legal Commons (Europarl, the Caselaw Access Project and Chinese case law). OpenSource is GitHub code, OpenScience is OpenAlex and other open science repositories, OpenWeb is Wikipedia, YouTube Commons and Stack Exchange, and OpenSemantic is Wikidata rewritten as natural-language statements in over 300 languages.
The table sets the card's counts beside the paper's Table 3 as of 2026-11-17, in the card's order. The paper calls OpenSource "Open Code".
| Collection | Main sources (card) | Tokens, card | Tokens, paper Table 3 | Source |
|---|---|---|---|---|
| OpenCulture | public domain books and newspapers, Wikisource, Gutenberg | 967,018,390,906 | 885,982,490,090 | card; arXiv 2506.01732 |
| OpenGovernment | Finance Commons (SEC, WTO), Legal Commons (Europarl, Caselaw Access Project, Chinese case law) | 579,150,518,908 | 407,067,554,189 | card; arXiv |
| OpenSource (paper: Open Code) | GitHub code (paper: via The Stack v1 and v2) | 283,227,402,898 | 283,227,402,898 | card; arXiv |
| OpenScience | OpenAlex and other open science repositories | 281,193,563,789 | 281,193,563,789 | card; arXiv |
| OpenWeb | Wikipedia, YouTube Commons, Stack Exchange | 88,517,032,065 | 73,217,485,489 | card; arXiv |
| OpenSemantic | Wikidata, rewritten as natural-language statements | 67,958,671,827 | 67,958,671,827 | card; arXiv |
| Stated total | 2,267,302,720,836 (six rows add up to slightly less) | 1,998,647,168,282 (517,033,648 documents) | card; arXiv |
The card's total of 2,267,302,720,836 tokens is Pleias's claim (checked 2026-11-09, re-read 2026-11-17), and its six collection counts add up to slightly less. The paper's Table 3 gives 1,998,647,168,282 tokens in 517,033,648 documents and 1,088,638,258,040 words, counted with Pleias's own tokenizer.
The card describes a second version that added Wikidata and document-level information including licensing, and a third, ongoing version that adds large Chinese, Japanese, Arabic, Korean and Hindi collections, which is one reason the counts move. This page keeps both counts, adds neither to the other and calls neither wrong.
Each record carries an identifier, collection, open type, license, date, title, creator, language, word count, token count and text.
The card says the source URL is in the metadata for most of the dataset and that more than half of Common Corpus predates the 21st century. How such fields fit into provenance records is covered on the page about dataset documentation and data cards.
Languages in Common Corpus and Why the Counts Differ
Common Corpus is mostly English and French, and its two sources disagree on how many other languages pass 10 billion tokens: the card says 8, the paper says nine.
The card says 8 languages hold more than 10 billion tokens (it names German, Spanish, Italian, Polish, Greek and Latin), 33 languages hold more than 1 billion, and the metadata uses 13 language tags: en, fr, de, zh, it, es, ja, pl, la, nl, ru, ar and ko.
The paper says at least 10 billion tokens for nine languages, identified with fastText. Its Table 5 leads with English at 968,757,721,747 tokens, French at 275,358,437,630 and German at 112,127,458,251, and lists 34 languages above 1 billion tokens (counted from the table on 2026-11-17).
Nearly 85 % of OpenScience documents are in English, and the paper says OpenCulture covers over 13 European languages from institutions based in the US or the EU only. Both counts are shown here as published, and neither is called wrong.
Licenses Inside Common Corpus: Tokens per License and No Single Dataset License
Common Corpus has no single license: each document carries its own license field, and the paper counts most of its tokens as public domain. The repository record shows no dataset license and the card metadata lists none.
The card describes the license field as covering uncopyrighted material (public domain, US federal public domain, CC0 on Wikidata) or various free licenses (Creative Commons, MIT, the French Licence ouverte and others).
Selection rules differ by source in the paper. OpenAlex records are filtered to CC-BY, Public Domain or CC0, and CC-BY-SA.
Code from The Stack v1 and v2 is filtered to permissive licenses, Stack Exchange text is CC-BY-SA and reused from The Pile, and YouTube Commons contributes transcripts of 2,063,066 videos under CC-BY. The table gives the paper's top ten licenses by tokens as of 2026-11-17, in the order published.
| License | Tokens | Source |
|---|---|---|
| Public Domain | 1,138,508,375,958 | arXiv 2506.01732 v4, Table 4 |
| CC-By | 287,749,264,457 | same |
| MIT | 142,694,227,607 | same |
| CC-By-SA | 74,768,060,836 | same |
| Apache-2.0 | 68,750,977,037 | same |
| BSD-3-Clause | 18,483,944,333 | same |
| Open license | 10,432,513,767 | same |
| BSD-2-Clause | 5,497,145,480 | same |
| CC-BY-4.0 | 2,110,966,243 | same |
| CC0-1.0 | 1,877,206,195 | same |
Pleias states on the card that all data "may be used for both commercial and non-commercial purposes" and that users can filter to public domain only or to a subset of free licenses such as attribution only. That is Pleias's statement.
Each license keeps its own conditions, attribution for CC-By and share-alike for CC-By-SA among them, and the license text of each document governs. How CC, ODC and CDLA terms compare is covered on the page about dataset licenses for AI training.
How Pleias Checked Copyright Expiry for Cultural Heritage Texts
Pleias states that all OpenCulture documents are in the public domain, and where it did not rely on the guarantee of the holding institution it applied its own rights-verification criteria. The paper's Appendix E lists them.
- For non-US authors, the author's life plus 70 years, with death dates reconciled against Wikidata.
- For anonymous and collective works such as newspapers, a publication-date cutoff derived from that life-plus-70 rule.
- For US authors, publication plus 95 years, without researching renewals.
- No new rights from digitization, which the paper supports by citing Article 14 of the 2019 Copyright Directive.
These are Pleias's criteria, not a court's or a registry's finding. The paper restricts OpenCulture to institutions based in the US or the EU, and it names the Internet Archive (about 2 million monographs) and Delpher (50,000 Dutch monographs and periodicals) among its sources.
The EU term the first criterion rests on runs for the author's life plus 70 years from 1 January of the following year (Directive 2006/116/EC, Articles 1(1) and 8). Terms by country, and how they apply to training, belong to the page on public domain and openly licensed material in AI training.
What Curation Changed in the Text: OCR Correction, Toxic Content and Personal Data
Pleias did not train on the digitized texts as they were: it segmented them, corrected OCR errors with its own models, filtered toxic content and replaced personal data. The paper and card name these tools.
- Segmentext, which segments the documents.
- OCRoscope and OCRerrcr, which detect and score OCR errors.
- OCRonos, built on Llama 3 8B, which the paper says "can act as a synthetic rewriting tool rather than a strict correction tool" on deteriorated text.
- The Celadon toxicity classifier, with the card adding that texts with high toxicity scores, offensive terms and slurs were removed.
For personal data, the paper uses Microsoft Presidio with custom patterns for phone numbers, email addresses, IBANs, IP addresses and URLs, which raised phone detection to 85 %.
Detected personal data is replaced "with fictitious but realistic values", and the card says small parts of a French administrative crawl were dropped over GDPR risk. The seller's side of this step is covered on the page about PII removal and de-identification before licensing.
The paper says none of the curation methods could facilitate a hundred percent accuracy, and that Common Corpus alone is large enough only for small models at present. A text in the corpus can therefore differ from the original an author published.
Who Has Named Common Corpus as Training Data, and Its Legal Standing
Pleias trained its own models on Common Corpus, and xAI's EU training-content summary for Grok 4.7 names Common Corpus among the public datasets it used.
The paper says PleIAs 350M (trained on a filtered subset of about 1 trillion tokens) and PleIAs 1.2B perform comparably to models trained on closed or non-permissively licensed data, a claim by the paper's authors.
The Grok 4.7 summary, published by SpaceXAI on 2026-09-21, lists Common Corpus (Pleias), OpenAlex metadata and Crossref metadata as public datasets, which is SpaceXAI's claim. The buyer's own terms are on the profile of xAI (SpaceXAI) as a data buyer.
Wikimedia Enterprise listed Pleias among its existing partners on 2026-01-15, in a post about its paid Enterprise API that states no value, and the card names a partnership with Wikimedia Enterprise.
No court case in the case tracker names Common Corpus, as of 2026-11-17. The card and the paper describe no removal form, contact route or opt-out for authors.
The card says the dataset "in its entirety meets the requirements of the Code of Conduct of the AI Act", which is Pleias's claim and not a confirmed finding.
The European Commission's page lists Pleias among 22 signatories of the general-purpose AI Code of Practice (page last updated 2026-10-07, checked 2026-11-09). Article 53(1)(d) of Regulation (EU) 2024/1689 requires providers of general-purpose AI models to publish a sufficiently detailed summary of training content, as covered under EU AI Act Article 53.
An Openly Licensed Work in Common Corpus: Benefits, Risks, Downsides and Alternatives for the Author
For an author whose openly licensed paper, video transcript or code sits in Common Corpus, inclusion follows from the license the author chose, and it brings no payment and no contract. The lists weigh what that means.
Benefits:
- The license field and, for most documents, the source URL travel with the text.
- Use by named model providers documents the work as training data.
Risks:
- The text can differ from the original after OCR correction and personal data replacement.
- The card and the paper describe no removal route.
- Public domain status of cultural heritage texts rests on Pleias's own criteria.
Downsides:
- No payment follows from inclusion.
- A license chosen earlier already permits this use where its terms allow.
Alternatives:
- Choose the license of future works deliberately, which is a decision for the author and not legal advice.
- License text directly to AI developers, as covered under text corpora for AI training.
- Contact the publisher through its own channels for a work believed to be wrongly classed as public domain.
- Not licensing at all, which is a valid outcome.
Other Open Corpora, Copyright Terms and Removal Beyond This Page
This page covers Common Corpus; other open corpora, license families, copyright exceptions and removal routes each have their own page.
- Corpora: C4, FineWeb, The Pile, Dolma and what Common Crawl contains. The paper compares itself with crawl-based sets and finds little overlap with FineWeb.
- Related data types: scientific data and papers as AI training data, selling or licensing source code to AI companies and libraries, archives and museums licensing collections.
- Law: EU text and data mining exception, copyright and AI training across jurisdictions and AI copyright lawsuits over training data.
- Sources and options: licensed vs scraped training data and alternatives to selling data, including not selling at all.
- Mentioned only: Common Pile, Institutional Books, The Stack, Wikimedia Enterprise AI deals and removing content from AI training datasets.
Frequently Asked Questions About Common Corpus
What is Common Corpus?
Common Corpus is Pleias's text dataset of public domain and openly licensed material. Pleias first released it in November 2024, and it is hosted on Hugging Face, Inc.'s platform.
How big is Common Corpus?
The card states 2,267,302,720,836 tokens, and the paper counts 1,998,647,168,282 tokens in 517,033,648 documents. The paper's abstract says about two trillion, and both counts are shown here.
What license does Common Corpus use?
Common Corpus has no single license, because each document carries its own. The paper's count is led by Public Domain, CC-By, MIT, CC-By-SA and Apache-2.0.
Can Common Corpus be used commercially?
Pleias states that all data may be used for commercial and non-commercial purposes. Each document's conditions, such as attribution or share-alike, still apply. This page is information, not legal advice.