It mixes openly licensed and public sources with scraped and copied ones, including a book collection later removed from its host and a scrape of YouTube captions. EleutherAI calls its 2025 openly licensed corpus the Pile's successor.
This page reads the Pile from the side of the people whose text it holds. Within AI training data licensing, it is a free mixed corpus, the unpaid counterpart to licensed text.
Who Built the Pile: EleutherAI's 2020 Paper and 2022 Datasheet
The Pile was built by twelve researchers listed under EleutherAI, who introduced it in the paper "The Pile: An 800GB Dataset of Diverse Text for Language Modeling", submitted to arXiv on 2020-12-31. The table lists the identity facts as of 2026-11-17.
| Attribute | Value | Source |
|---|---|---|
| Built by | 12 authors listed under EleutherAI (Gao, Biderman, Black, Golding, Hoppe, Foster, Phang, He, Thite, Nabeshima, Presser, Leahy) | arxiv.org/abs/2101.00027 |
| Legal entity | not stated on EleutherAI's site | eleuther.ai |
| Paper | "The Pile: An 800GB Dataset of Diverse Text for Language Modeling", arXiv 2101.00027, submitted 2020-12-31 | arxiv.org |
| Datasheet | "Datasheet for the Pile", arXiv 2201.07311, 2022-01-13; release date given as 2021-01-01 | arxiv.org/abs/2201.07311 |
| Hosting named by EleutherAI | the Eye, plus community mirrors (datasheet); "hosted by the Eye" (project site) | arxiv.org; pile.eleuther.ai |
| Hub copy | EleutherAI/pile, license tag "other", not gated, last modified 2023-05-03; deduplicated copy EleutherAI/the_pile_deduplicated, created 2022-12-02 | huggingface.co |
| Updates | "EleutherAI does not plan to update the Pile" (datasheet) | arxiv.org |
The paper names EleutherAI as the only affiliation. This page writes EleutherAI as a brand, because the group's site names no legal entity. The datasheet, written by Biderman, Bicheno and Gao, gives the release date as January 1, 2021, one day after the paper's submission, so both dates stay visible with their source.
The paper also offers the Pile as a benchmark, called Pile BPB, and the Hugging Face card names Leo Gao and Stella Biderman as its primary curators. The Pile is one of the public corpora compared on the page about major AI training datasets.
What the Pile Holds: 22 Components by Size and Weight
The Pile holds 22 components, from Pile-CC, a filtered extract of Common Crawl at 227.12 GiB, to the Enron Emails at 0.88 GiB, and each enters the mix with its own weight and number of passes. The table repeats the paper's Table 1 as of 2026-11-17.
| Component | Raw size | Weight | Epochs | Source |
|---|---|---|---|---|
| Pile-CC | 227.12 GiB | 18.11 % | 1.0 | arXiv 2101.00027, Table 1 |
| PubMed Central | 90.27 GiB | 14.40 % | 2.0 | same |
| Books3 | 100.96 GiB | 12.07 % | 1.5 | same |
| OpenWebText2 | 62.77 GiB | 10.01 % | 2.0 | same |
| ArXiv | 56.21 GiB | 8.96 % | 2.0 | same |
| GitHub | 95.16 GiB | 7.59 % | 1.0 | same |
| FreeLaw | 51.15 GiB | 6.12 % | 1.5 | same |
| Stack Exchange | 32.20 GiB | 5.13 % | 2.0 | same |
| USPTO Backgrounds | 22.90 GiB | 3.65 % | 2.0 | same |
| PubMed Abstracts | 19.26 GiB | 3.07 % | 2.0 | same |
| Gutenberg (PG-19) | 10.88 GiB | 2.17 % | 2.5 | same |
| OpenSubtitles | 12.98 GiB | 1.55 % | 1.5 | same |
| Wikipedia (en) | 6.38 GiB | 1.53 % | 3.0 | same |
| DM Mathematics | 7.75 GiB | 1.24 % | 2.0 | same |
| Ubuntu IRC | 5.52 GiB | 0.88 % | 2.0 | same |
| BookCorpus2 | 6.30 GiB | 0.75 % | 1.5 | same |
| EuroParl | 4.59 GiB | 0.73 % | 2.0 | same |
| HackerNews | 3.90 GiB | 0.62 % | 2.0 | same |
| YouTube Subtitles | 3.73 GiB | 0.60 % | 2.0 | same |
| PhilPapers | 2.38 GiB | 0.38 % | 2.0 | same |
| NIH ExPorter | 1.89 GiB | 0.30 % | 2.0 | same |
| Enron Emails | 0.88 GiB | 0.14 % | 2.0 | same |
| Total | 825.18 GiB raw; 1254.20 GiB effective | n/a | n/a | same |
In the paper, Books3, Gutenberg, OpenSubtitles, Wikipedia, DM Mathematics, EuroParl and Enron Emails came from prior work with minimal preprocessing. The total is 825.18 GiB raw and 1254.20 GiB effective after repetition, while the title says 800GB. Twelve sources were new datasets introduced by the paper, and OpenWebText2 and BookCorpus2 extend earlier sets.
Pile-CC was extracted with jusText from WARC files of 2013 to 2020, of which the paper processed 22 of 3,679 chunks, and filtered with a classifier trained against OpenWebText2. The crawl itself belongs to the page about Common Crawl, linked below.
OpenWebText2 is the paper's extension of OpenWebText, which has its own page. The set it extends is covered on the page about OpenWebText.
How the Pile's Authors Rated Consent for Each Component
The Pile's authors rated each component on three tiers, public, compliant with the host's terms of service, and consented to by the authors, and treated ambiguous or unknown consent as no consent. The paper puts it this way: "We adopted a strict model of consent, where ambiguous or unknown consent is treated as non-consensual."
The ratings are the authors' own findings. They wrote that Books3 and OpenSubtitles were used consistently with the host's terms, but that "the data host is not authorized to post the data online by the parties that own it".
The Enron emails were "not collected with the permission of the authors" and were gathered by the U.S. government in a criminal investigation.
Five components "were not collected and distributed in a ToS compliant fashion", and their authors had no ability to consent. Of those five, the paper names YouTube Subtitles as the only one it created itself.
The datasheet answers "No" to a mechanism for revoking consent and "No" to a data protection impact analysis. It adds that no formal ethical review was done, because EleutherAI has no institutional review board. How datasheets record such answers belongs to the page on dataset documentation, datasheets and data cards.
The Pile's YouTube Subtitles Component: How the Captions Were Gathered
The YouTube Subtitles component holds human-made closed captions from 173,651 YouTube videos, which EleutherAI gathered for the Pile itself, 3.73 GiB of text with a weight of 0.60 %. The method is in the paper's appendix C.19.
The authors built a list of search terms by prompting GPT-3 and filtering the output by hand, centred on educational topics. They took 1,000 video IDs per term, removed duplicates, and fetched all human-generated captions in every available language with the open-source youtube-transcript-api, which the paper calls "a very popular unofficial API".
The datasheet says the captions "were added by the video owners or volunteers, and then scraped directly from YouTube by EleutherAI", for videos from 2005-02-14 to 2020-07-01. The paper counts the component among the five not collected in a ToS-compliant way. This page states no view on YouTube's terms and no court view of them.
Today YouTube offers channel owners a third-party training setting that is off by default, and YouTube states that it "isn't facilitating payments" between AI companies and creators (checked 2026-11-09).
The setting governs future use and does not reach captions copied in 2020. Platform settings and their limits are covered on the page about opting out of AI training through platform settings.
What Left the Pile's Hosts: The Book Component and Later Takedowns
The Pile's book component, Books3, is the part that left its host: the Danish Rights Alliance announced on 2023-08-14 that the host had removed it in mid-July 2023.
The Pile paper describes Books3 as derived from the Bibliotik private tracker, at 100.96 GiB raw and a 12.07 % weight, and the datasheet says it was "made available by The Eye".
The paper itself was never withdrawn. The EleutherAI/pile repository still carries a loader whose data URLs point to the-eye.eu, last modified 2023-05-03. The table gives the dated events as of 2026-11-17.
| Date | Event | Source |
|---|---|---|
| 2020-12-31 | paper submitted; Books3 listed as a component | arxiv.org |
| 2023-05-03 | last change to the EleutherAI/pile Hub repository (loader points to the-eye.eu) | huggingface.co |
| 2023-07 (mid-month) | host removes Books3 after the Danish Rights Alliance's approach; announced 2023-08-14 | rettighedsalliancen.dk |
| 2023-08-25 | notice to Hugging Face on a books-3 re-upload | huggingface.co/datasets/huggingface-legal/takedown-notices |
| 2023-08-30 | third-party pile-uncopyrighted copy created without five components, its uploader's claim | huggingface.co/datasets/monology/pile-uncopyrighted |
| 2024-07-04 | notice to Hugging Face on a Books3 re-upload, asking to prevent future uploads | huggingface.co/datasets/huggingface-legal/takedown-notices |
On Hugging Face, the Danish Rights Alliance sent notices on 2023-08-25, for a books-3 re-upload of at least 109 works of its members, and on 2024-07-04, for a Books3 re-upload of "approximately 200.000 illegal e-books" in its words.
The second notice asked Hugging Face to prevent future uploads. None of the 59 text notices in the public takedown repository names the Pile or EleutherAI, and 18 PDF notices were not read. The book component's origin, counts, removal and dockets belong to the page on Books3.
A third-party copy on the Hub, pile-uncopyrighted, created 2023-08-30, states that it removed "everything from the Books3, BookCorpus2, OpenSubtitles, YTSubtitles, and OWT2 subsets" on its reading of the paper's section 7.1.
That is the uploader's claim, and it is not proof that the rest is free of copyright. Whether the-eye.eu still serves the files is not stated here.
The Pile's Terms: MIT on the Datasheet, "Other" on the Hub
The datasheet states that the Pile "is licensed under the MIT License", while the EleutherAI/pile card on Hugging Face tags it "other" and tells users to refer to the license of each subset.
The card lists one subset license, PubMed Central under MIT. The loader script carries an Apache License 2.0 header, which covers the code, and the datasheet states "no fees, access restrictions, or import restrictions".
The authors' legal position is theirs and is dated 2020. The paper's section 7.1 argues that "non-commercial, not-for-profit use of copyright media is preemptively fair use" and that the use is transformative, with its own note that this "does not, and is not intended to, constitute legal advice".
The datasheet says Books3 "is almost entirely comprised of copyrighted works", that a substantial portion of arXiv and PhilPapers are as well, and that users may compile "a custom remix" to exclude components. Nothing on file settles the fair-use view for the Pile. The table summarizes the layers as of 2026-11-17.
| Layer | Terms stated | What it covers | Source |
|---|---|---|---|
| The Pile (datasheet) | "licensed under the MIT License"; no fees or access restrictions | the compilation, at most the compiler's own rights | arxiv.org/abs/2201.07311 |
| The Hub copy | license "other"; "refer to the specific license depending on the subset"; lists PubMed Central (MIT) only | the card's label | huggingface.co/datasets/EleutherAI/pile |
| The loader code | Apache License 2.0 header | the script, not the text | huggingface.co |
| The works inside | each author's or rightholder's own rights; Books3 "almost entirely comprised of copyrighted works" (datasheet) | n/a | arxiv.org |
A license on a compilation covers at most the compiler's own rights, so neither label licenses the books, captions or emails inside the Pile. How licenses split database and contents is covered on the page about dataset licenses for AI training.
Models Trained on the Pile and the Court Records That Name It
EleutherAI's own model cards state that GPT-Neo, GPT-J, GPT-NeoX-20B and Pythia were trained on the Pile, and one pending complaint alleges that Salesforce's XGen models were too. The table lists the records as of 2026-11-17.
| Model | Developer | Record | Record type | Source |
|---|---|---|---|---|
| GPT-J 6B | EleutherAI | "trained on the Pile" | developer's model card | huggingface.co/EleutherAI/gpt-j-6b |
| GPT-Neo 2.7B | EleutherAI | "trained on the Pile" | developer's model card | huggingface.co/EleutherAI/gpt-neo-2.7B |
| GPT-NeoX-20B | EleutherAI | "trained on the Pile" | developer's model card | huggingface.co/EleutherAI/gpt-neox-20b |
| Pythia | EleutherAI | trained on the Pile, and a second suite on the deduplicated Pile | developer's model card | huggingface.co/EleutherAI/pythia-12b |
| XGen | Salesforce | trained on the Pile, RedPajama and Books3 | complaint allegation | Tanzer complaint, 2025-10-15 |
Tanzer v. Salesforce, N.D. Cal. 3:25-cv-08862-CRB, was filed on 2025-10-15 as a proposed class action alleging that XGen was trained on the Pile, RedPajama and Books3. It is pending as of the last check on 2026-11-09, and the allegation is not a finding.
The case has its own page on In re Salesforce: Books3 and RedPajama training claims.
Huckabee v. Bloomberg L.P., S.D.N.Y. 1:23-cv-09152-MMG, was filed on 2023-10-17 and originally also named Meta, Microsoft and EleutherAI as defendants. The amended complaint of 2024-01-24 kept one count of direct infringement against Bloomberg, and the motion to dismiss was denied on 2025-11-24 (Dkt. 98); later entries were not checked.
No court record on file holds anything about EleutherAI's own conduct, and no ruling on file decides the Pile as a whole (as of 2026-11-17).
A Work in the Pile: Benefits, Risks, Downsides and Alternatives for the Rightholder
For an author, captioner or other rightholder whose text sits in the Pile, the corpus brought no payment and no contract, and its datasheet offers no way to revoke consent.
Benefits
- The paper and datasheet name every component and its source, so a rightholder can tell which part could hold their work.
- Records on the Hub copy keep a component label (pile_set_name).
Risks
- Copies and mirrors made since 2021 stay with whoever made them, and the datasheet itself mentions community-maintained mirrors.
- Models trained on the Pile are public, as the table above shows.
- The authors' fair-use view is untested in any ruling on the Pile.
Downsides
- The datasheet names no revocation route.
- The Hub license tag "other" gives no rightholder a license.
Alternatives
- Send a notice to the host of a copy, naming the files; the 2023 and 2024 notices show the route was used for Books3 re-uploads, and this page makes no claim about outcomes.
- Use the YouTube third-party training setting, which reaches future use only.
- Offer text directly to AI developers, as described on the page about text corpora for AI training.
- Not license at all.
Not licensing is a valid outcome.
The Successor Corpus, Related Sets and the Law Beyond This Page
EleutherAI and collaborators released Common Pile v0.1 on 2025-06-05, 8 TB of openly licensed text from 30 sources, which EleutherAI's post calls "the successor to the Pile". It has its own page, not yet published.
- Common Crawl: the crawl behind Pile-CC.
- C4 (Colossal Clean Crawled Corpus): another filtered web corpus.
- RedPajama: named next to the Pile in the XGen complaint.
- Dolma: another open text corpus.
- Common Corpus: an openly licensed corpus.
- AI copyright lawsuits over training data: the case tracker.
- Kadrey v. Meta claims, rulings and status: the books case built on the Pile's book component.
- Huckabee v. Bloomberg (BloombergGPT and Books3): the case that originally also named EleutherAI.
- Fair use and AI training in US law: the doctrine the authors relied on.
- Copyright and AI training across jurisdictions: outside the US.
- Is web scraping for AI training legal: scraped captions and terms of service.
- Privacy law and AI training data: emails and captions as personal data.
- Pirated books and shadow libraries in AI training: tracker-sourced books.
- Books as AI training data: licensed book text.
- Licensed vs scraped training data: the licensed alternative.
- Removing content from AI training datasets: removal routes across datasets.
- Alternatives to selling data, including not selling at all: not licensing at all.
Personal data in the email and caption components falls under privacy rules that differ by jurisdiction.
Frequently Asked Questions About the Pile
What is the Pile dataset?
The Pile is an 825 GiB English corpus of 22 components, assembled by EleutherAI and described in a paper submitted 2020-12-31. It was used to train GPT-Neo, GPT-J, GPT-NeoX-20B and Pythia, according to EleutherAI's model cards.
How big is the Pile?
The Pile is 825.18 GiB of raw text and 1254.20 GiB effective after repetition, per the paper's Table 1. The paper's title says 800GB, so the headline figure and the table differ.
What license does the Pile use?
The datasheet says MIT, while the Hugging Face card says "other" and points to each subset's license. Neither label licenses the works inside the Pile, which stay under their own rights.
Is my work in the Pile, and can it be removed?
The paper and datasheet name each component's source, so a work can be traced to a component. The datasheet offers no revocation route, and a copy can only be addressed through its host. This page is information, not legal advice.
Agents and advisers can use the introducer form.