DataPacts

The Pile

Checked

At a glance

The Pile is an 825 GiB English text corpus that EleutherAI assembled from 22 smaller datasets and described in a paper dated December 31, 2020, for training large language models. Its title calls it an 800GB dataset, and the paper's own table gives the exact figures.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

It mixes openly licensed and public sources with scraped and copied ones, including a book collection later removed from its host and a scrape of YouTube captions. EleutherAI calls its 2025 openly licensed corpus the Pile's successor.

This page reads the Pile from the side of the people whose text it holds. Within AI training data licensing, it is a free mixed corpus, the unpaid counterpart to licensed text.

Who Built the Pile: EleutherAI's 2020 Paper and 2022 Datasheet

The Pile was built by twelve researchers listed under EleutherAI, who introduced it in the paper "The Pile: An 800GB Dataset of Diverse Text for Language Modeling", submitted to arXiv on 2020-12-31. The table lists the identity facts as of 2026-11-17.

Attribute Value Source
Built by 12 authors listed under EleutherAI (Gao, Biderman, Black, Golding, Hoppe, Foster, Phang, He, Thite, Nabeshima, Presser, Leahy) arxiv.org/abs/2101.00027
Legal entity not stated on EleutherAI's site eleuther.ai
Paper "The Pile: An 800GB Dataset of Diverse Text for Language Modeling", arXiv 2101.00027, submitted 2020-12-31 arxiv.org
Datasheet "Datasheet for the Pile", arXiv 2201.07311, 2022-01-13; release date given as 2021-01-01 arxiv.org/abs/2201.07311
Hosting named by EleutherAI the Eye, plus community mirrors (datasheet); "hosted by the Eye" (project site) arxiv.org; pile.eleuther.ai
Hub copy EleutherAI/pile, license tag "other", not gated, last modified 2023-05-03; deduplicated copy EleutherAI/the_pile_deduplicated, created 2022-12-02 huggingface.co
Updates "EleutherAI does not plan to update the Pile" (datasheet) arxiv.org

The paper names EleutherAI as the only affiliation. This page writes EleutherAI as a brand, because the group's site names no legal entity. The datasheet, written by Biderman, Bicheno and Gao, gives the release date as January 1, 2021, one day after the paper's submission, so both dates stay visible with their source.

The paper also offers the Pile as a benchmark, called Pile BPB, and the Hugging Face card names Leo Gao and Stella Biderman as its primary curators. The Pile is one of the public corpora compared on the page about major AI training datasets.

What the Pile Holds: 22 Components by Size and Weight

The Pile holds 22 components, from Pile-CC, a filtered extract of Common Crawl at 227.12 GiB, to the Enron Emails at 0.88 GiB, and each enters the mix with its own weight and number of passes. The table repeats the paper's Table 1 as of 2026-11-17.

Component Raw size Weight Epochs Source
Pile-CC 227.12 GiB 18.11 % 1.0 arXiv 2101.00027, Table 1
PubMed Central 90.27 GiB 14.40 % 2.0 same
Books3 100.96 GiB 12.07 % 1.5 same
OpenWebText2 62.77 GiB 10.01 % 2.0 same
ArXiv 56.21 GiB 8.96 % 2.0 same
GitHub 95.16 GiB 7.59 % 1.0 same
FreeLaw 51.15 GiB 6.12 % 1.5 same
Stack Exchange 32.20 GiB 5.13 % 2.0 same
USPTO Backgrounds 22.90 GiB 3.65 % 2.0 same
PubMed Abstracts 19.26 GiB 3.07 % 2.0 same
Gutenberg (PG-19) 10.88 GiB 2.17 % 2.5 same
OpenSubtitles 12.98 GiB 1.55 % 1.5 same
Wikipedia (en) 6.38 GiB 1.53 % 3.0 same
DM Mathematics 7.75 GiB 1.24 % 2.0 same
Ubuntu IRC 5.52 GiB 0.88 % 2.0 same
BookCorpus2 6.30 GiB 0.75 % 1.5 same
EuroParl 4.59 GiB 0.73 % 2.0 same
HackerNews 3.90 GiB 0.62 % 2.0 same
YouTube Subtitles 3.73 GiB 0.60 % 2.0 same
PhilPapers 2.38 GiB 0.38 % 2.0 same
NIH ExPorter 1.89 GiB 0.30 % 2.0 same
Enron Emails 0.88 GiB 0.14 % 2.0 same
Total 825.18 GiB raw; 1254.20 GiB effective n/a n/a same

In the paper, Books3, Gutenberg, OpenSubtitles, Wikipedia, DM Mathematics, EuroParl and Enron Emails came from prior work with minimal preprocessing. The total is 825.18 GiB raw and 1254.20 GiB effective after repetition, while the title says 800GB. Twelve sources were new datasets introduced by the paper, and OpenWebText2 and BookCorpus2 extend earlier sets.

Pile-CC was extracted with jusText from WARC files of 2013 to 2020, of which the paper processed 22 of 3,679 chunks, and filtered with a classifier trained against OpenWebText2. The crawl itself belongs to the page about Common Crawl, linked below.

OpenWebText2 is the paper's extension of OpenWebText, which has its own page. The set it extends is covered on the page about OpenWebText.

The Pile's authors rated each component on three tiers, public, compliant with the host's terms of service, and consented to by the authors, and treated ambiguous or unknown consent as no consent. The paper puts it this way: "We adopted a strict model of consent, where ambiguous or unknown consent is treated as non-consensual."

The ratings are the authors' own findings. They wrote that Books3 and OpenSubtitles were used consistently with the host's terms, but that "the data host is not authorized to post the data online by the parties that own it".

The Enron emails were "not collected with the permission of the authors" and were gathered by the U.S. government in a criminal investigation.

Five components "were not collected and distributed in a ToS compliant fashion", and their authors had no ability to consent. Of those five, the paper names YouTube Subtitles as the only one it created itself.

The datasheet answers "No" to a mechanism for revoking consent and "No" to a data protection impact analysis. It adds that no formal ethical review was done, because EleutherAI has no institutional review board. How datasheets record such answers belongs to the page on dataset documentation, datasheets and data cards.

The Pile's YouTube Subtitles Component: How the Captions Were Gathered

The YouTube Subtitles component holds human-made closed captions from 173,651 YouTube videos, which EleutherAI gathered for the Pile itself, 3.73 GiB of text with a weight of 0.60 %. The method is in the paper's appendix C.19.

The authors built a list of search terms by prompting GPT-3 and filtering the output by hand, centred on educational topics. They took 1,000 video IDs per term, removed duplicates, and fetched all human-generated captions in every available language with the open-source youtube-transcript-api, which the paper calls "a very popular unofficial API".

The datasheet says the captions "were added by the video owners or volunteers, and then scraped directly from YouTube by EleutherAI", for videos from 2005-02-14 to 2020-07-01. The paper counts the component among the five not collected in a ToS-compliant way. This page states no view on YouTube's terms and no court view of them.

Today YouTube offers channel owners a third-party training setting that is off by default, and YouTube states that it "isn't facilitating payments" between AI companies and creators (checked 2026-11-09).

The setting governs future use and does not reach captions copied in 2020. Platform settings and their limits are covered on the page about opting out of AI training through platform settings.

What Left the Pile's Hosts: The Book Component and Later Takedowns

The Pile's book component, Books3, is the part that left its host: the Danish Rights Alliance announced on 2023-08-14 that the host had removed it in mid-July 2023.

The Pile paper describes Books3 as derived from the Bibliotik private tracker, at 100.96 GiB raw and a 12.07 % weight, and the datasheet says it was "made available by The Eye".

The paper itself was never withdrawn. The EleutherAI/pile repository still carries a loader whose data URLs point to the-eye.eu, last modified 2023-05-03. The table gives the dated events as of 2026-11-17.

Date Event Source
2020-12-31 paper submitted; Books3 listed as a component arxiv.org
2023-05-03 last change to the EleutherAI/pile Hub repository (loader points to the-eye.eu) huggingface.co
2023-07 (mid-month) host removes Books3 after the Danish Rights Alliance's approach; announced 2023-08-14 rettighedsalliancen.dk
2023-08-25 notice to Hugging Face on a books-3 re-upload huggingface.co/datasets/huggingface-legal/takedown-notices
2023-08-30 third-party pile-uncopyrighted copy created without five components, its uploader's claim huggingface.co/datasets/monology/pile-uncopyrighted
2024-07-04 notice to Hugging Face on a Books3 re-upload, asking to prevent future uploads huggingface.co/datasets/huggingface-legal/takedown-notices

On Hugging Face, the Danish Rights Alliance sent notices on 2023-08-25, for a books-3 re-upload of at least 109 works of its members, and on 2024-07-04, for a Books3 re-upload of "approximately 200.000 illegal e-books" in its words.

The second notice asked Hugging Face to prevent future uploads. None of the 59 text notices in the public takedown repository names the Pile or EleutherAI, and 18 PDF notices were not read. The book component's origin, counts, removal and dockets belong to the page on Books3.

A third-party copy on the Hub, pile-uncopyrighted, created 2023-08-30, states that it removed "everything from the Books3, BookCorpus2, OpenSubtitles, YTSubtitles, and OWT2 subsets" on its reading of the paper's section 7.1.

That is the uploader's claim, and it is not proof that the rest is free of copyright. Whether the-eye.eu still serves the files is not stated here.

The Pile's Terms: MIT on the Datasheet, "Other" on the Hub

The datasheet states that the Pile "is licensed under the MIT License", while the EleutherAI/pile card on Hugging Face tags it "other" and tells users to refer to the license of each subset.

The card lists one subset license, PubMed Central under MIT. The loader script carries an Apache License 2.0 header, which covers the code, and the datasheet states "no fees, access restrictions, or import restrictions".

The authors' legal position is theirs and is dated 2020. The paper's section 7.1 argues that "non-commercial, not-for-profit use of copyright media is preemptively fair use" and that the use is transformative, with its own note that this "does not, and is not intended to, constitute legal advice".

The datasheet says Books3 "is almost entirely comprised of copyrighted works", that a substantial portion of arXiv and PhilPapers are as well, and that users may compile "a custom remix" to exclude components. Nothing on file settles the fair-use view for the Pile. The table summarizes the layers as of 2026-11-17.

Layer Terms stated What it covers Source
The Pile (datasheet) "licensed under the MIT License"; no fees or access restrictions the compilation, at most the compiler's own rights arxiv.org/abs/2201.07311
The Hub copy license "other"; "refer to the specific license depending on the subset"; lists PubMed Central (MIT) only the card's label huggingface.co/datasets/EleutherAI/pile
The loader code Apache License 2.0 header the script, not the text huggingface.co
The works inside each author's or rightholder's own rights; Books3 "almost entirely comprised of copyrighted works" (datasheet) n/a arxiv.org

A license on a compilation covers at most the compiler's own rights, so neither label licenses the books, captions or emails inside the Pile. How licenses split database and contents is covered on the page about dataset licenses for AI training.

Models Trained on the Pile and the Court Records That Name It

EleutherAI's own model cards state that GPT-Neo, GPT-J, GPT-NeoX-20B and Pythia were trained on the Pile, and one pending complaint alleges that Salesforce's XGen models were too. The table lists the records as of 2026-11-17.

Model Developer Record Record type Source
GPT-J 6B EleutherAI "trained on the Pile" developer's model card huggingface.co/EleutherAI/gpt-j-6b
GPT-Neo 2.7B EleutherAI "trained on the Pile" developer's model card huggingface.co/EleutherAI/gpt-neo-2.7B
GPT-NeoX-20B EleutherAI "trained on the Pile" developer's model card huggingface.co/EleutherAI/gpt-neox-20b
Pythia EleutherAI trained on the Pile, and a second suite on the deduplicated Pile developer's model card huggingface.co/EleutherAI/pythia-12b
XGen Salesforce trained on the Pile, RedPajama and Books3 complaint allegation Tanzer complaint, 2025-10-15

Tanzer v. Salesforce, N.D. Cal. 3:25-cv-08862-CRB, was filed on 2025-10-15 as a proposed class action alleging that XGen was trained on the Pile, RedPajama and Books3. It is pending as of the last check on 2026-11-09, and the allegation is not a finding.

The case has its own page on In re Salesforce: Books3 and RedPajama training claims.

Huckabee v. Bloomberg L.P., S.D.N.Y. 1:23-cv-09152-MMG, was filed on 2023-10-17 and originally also named Meta, Microsoft and EleutherAI as defendants. The amended complaint of 2024-01-24 kept one count of direct infringement against Bloomberg, and the motion to dismiss was denied on 2025-11-24 (Dkt. 98); later entries were not checked.

No court record on file holds anything about EleutherAI's own conduct, and no ruling on file decides the Pile as a whole (as of 2026-11-17).

A Work in the Pile: Benefits, Risks, Downsides and Alternatives for the Rightholder

For an author, captioner or other rightholder whose text sits in the Pile, the corpus brought no payment and no contract, and its datasheet offers no way to revoke consent.

Benefits

  • The paper and datasheet name every component and its source, so a rightholder can tell which part could hold their work.
  • Records on the Hub copy keep a component label (pile_set_name).

Risks

  • Copies and mirrors made since 2021 stay with whoever made them, and the datasheet itself mentions community-maintained mirrors.
  • Models trained on the Pile are public, as the table above shows.
  • The authors' fair-use view is untested in any ruling on the Pile.

Downsides

  • The datasheet names no revocation route.
  • The Hub license tag "other" gives no rightholder a license.

Alternatives

  • Send a notice to the host of a copy, naming the files; the 2023 and 2024 notices show the route was used for Books3 re-uploads, and this page makes no claim about outcomes.
  • Use the YouTube third-party training setting, which reaches future use only.
  • Offer text directly to AI developers, as described on the page about text corpora for AI training.
  • Not license at all.

Not licensing is a valid outcome.

EleutherAI and collaborators released Common Pile v0.1 on 2025-06-05, 8 TB of openly licensed text from 30 sources, which EleutherAI's post calls "the successor to the Pile". It has its own page, not yet published.

Personal data in the email and caption components falls under privacy rules that differ by jurisdiction.

Frequently Asked Questions About the Pile

What is the Pile dataset?

The Pile is an 825 GiB English corpus of 22 components, assembled by EleutherAI and described in a paper submitted 2020-12-31. It was used to train GPT-Neo, GPT-J, GPT-NeoX-20B and Pythia, according to EleutherAI's model cards.

How big is the Pile?

The Pile is 825.18 GiB of raw text and 1254.20 GiB effective after repetition, per the paper's Table 1. The paper's title says 800GB, so the headline figure and the table differ.

What license does the Pile use?

The datasheet says MIT, while the Hugging Face card says "other" and points to each subset's license. Neither label licenses the works inside the Pile, which stay under their own rights.

Is my work in the Pile, and can it be removed?

The paper and datasheet name each component's source, so a work can be traced to a component. The datasheet offers no revocation route, and a copy can only be addressed through its host. This page is information, not legal advice.

Agents and advisers can use the introducer form.