Both sets were assembled from public sources without a license from the people whose pages, code, papers or books they hold, and the book slice of the first set is now marked defunct.
This page reads RedPajama from that side: within AI training data licensing, it is free text next to licensed text, with per-source terms, no data license of its own and one pending complaint.
Who Publishes RedPajama: Together and Its 2023 Research Partners
RedPajama is published by Together Computer, Inc., a Delaware corporation that trades as Together AI, which announced it on 2023-04-17 as a collaboration with Ontocord.ai, ETH DS3Lab, Stanford CRFM and Hazy Research. The table lists the identity facts as of 2026-11-17.
| Attribute | Value | Source |
|---|---|---|
| Publisher | Together Computer, Inc., "a Delaware corporation" per its Terms of Service (checked 2026-11-09); brand Together AI | together.ai/terms-of-service |
| Partners named in April 2023 | Ontocord.ai, ETH DS3Lab, Stanford CRFM, Hazy Research | together.ai/blog/redpajama |
| Partners thanked on the V2 card | Ontocord.ai, MILA Québec AI Institute, ETH DS3Lab, Université de Montréal, Stanford CRFM, Stanford Hazy Research, LAION | huggingface.co/datasets/togethercomputer/RedPajama-Data-V2 |
| RedPajama-Data-1T | announced 2023-04-17; Hub repository created 2023-04-17, last modified 2024-06-17, not gated | together.ai; huggingface.co |
| RedPajama-V2 | announced 2023-10-30; Hub repository created 2023-10-26, last modified 2024-11-21, not gated | together.ai; huggingface.co |
| Code | GitHub togethercomputer/RedPajama-Data, created 2023-04-14, Apache License 2.0 | github.com |
| Paper | Weber et al., arXiv 2411.12372, submitted 2024-11-19, NeurIPS 2024 Datasets and Benchmarks track | arxiv.org/abs/2411.12372 |
| Registry number | not checked | n/a |
The April 2023 post framed the data as the first of three steps, pre-training data, base models and instruction tuning, toward "fully open-source models". The later V2 card thanks a longer partner list than the first post, so the page shows both lists with their sources and never merges them.
Hugging Face, Inc. hosts both cards and neither repository is gated, but Hugging Face is the host, not the publisher. RedPajama is one of the corpora compared on the page about major AI training datasets.
What RedPajama-Data-1T Holds: The Slices Behind 1.2 Trillion Tokens
RedPajama-Data-1T holds about 1.2 trillion tokens in six slices that are still downloadable, from web pages, an existing web corpus, code, scientific papers, an encyclopedia and question-and-answer sites, plus a book slice that is now defunct.
The dataset card calls it "a clean-room, fully open-source implementation of the LLaMa dataset", built to follow the LLaMA paper as closely as possible and primarily English. The Hub copy has 2,084 JSONL files and a 1-billion-token sample, and the April 2023 post gives about 5 TB unzipped and about 3 TB compressed.
The table compares the slices of RedPajama-Data-1T as of 2026-11-17, in the order of the card, with the book slice last. The total on the card is 1.2 trillion tokens, Together's figure, and the page never adds the rows.
| Slice | Tokens, as Together states | Where it came from | Status | Source |
|---|---|---|---|---|
| Commoncrawl | 878 billion (checked 2026-11-09) | five Common Crawl dumps, 2019-30 to 2023-06, cc_net, paragraph deduplication, Wikipedia-reference classifier | available | dataset card; GitHub rp_v1 |
| C4 | 175 billion | C4 downloaded from Hugging Face, format change only | available | dataset card |
| GitHub | 59 billion | Google BigQuery; MIT, BSD or Apache projects only | available | dataset card |
| ArXiv | 28 billion | LaTeX sources, requester-pays bucket; preambles, comments, macros and bibliographies removed | available | dataset card |
| Wikipedia | 24 billion | Hugging Face Wikipedia set, 2023-03-20 dump, 20 languages | available | dataset card |
| StackExchange | 20 billion | Internet Archive, 28 largest sites, question-answer pairs | available | dataset card |
| Book | 26 billion (April 2023 post; no row on the current card) | pg19 subset of Project Gutenberg and the-pile-books3, from Hugging Face | "defunct and no longer accessible" | together.ai; GitHub rp_v1; dataset card |
The web slice uses five Common Crawl dumps, 2019-30, 2020-05, 2021-04, 2022-05 and 2023-06, so the crawl behind it is fixed at those dates. A linear classifier trained on pages cited by Wikipedia kept the pages it rated close to them.
The GitHub slice was selected by project license, MIT, BSD or Apache, and files were deduplicated. Each record carries a text field and a meta field with URL, timestamp and source, which lets a site owner trace a page.
The card calls the set primarily English, and only the Wikipedia slice is stated to span 20 languages.
The C4 slice is C4 "downloaded from Huggingface" with only a format change, so its contents and terms are C4's own. What C4 holds belongs to the page on C4, the Colossal Clean Crawled Corpus.
RedPajama's Book Slice and Why Together Marks It Defunct
The book slice of RedPajama-Data-1T is no longer available: the dataset card marks the "book" configuration as defunct due to reported copyright infringement for the Books3 collection it contained.
Together's repository, on its rp_v1 branch, says the slice was built from the pg19 subset of Project Gutenberg and "the-pile-books3", both downloaded from Hugging Face and deduplicated with SimHash.
The April 2023 post described the same slice as "a corpus of open books, deduplicated by content similarity" and gave it 26 billion tokens. The page states both descriptions with their sources and does not judge intent.
The card's notice reads that the configuration is "defunct and no longer accessible due to reported copyright infringement for the Book3 dataset contained in this config".
The card states no date and names no sender for the step. The current card table lists no book row while still giving a total of 1.2 trillion tokens, so the page never adds or subtracts slice counts.
The rp_v1 README had listed the licenses of the_pile_books3 and pg19 as the terms for this slice, and no rightholder license is recorded for either. The repository names the origin of the slice as two Hugging Face copies, not as a license from any author or publisher.
Where Books3 came from, how it left its host in 2023 and which dockets name it are covered on the page about what Books3 is and how it was removed.
What RedPajama-V2 Holds: 84 Web Snapshots, Five Languages and Quality Signals
RedPajama-V2 holds over 100 billion web documents from 84 Common Crawl snapshots in English, German, French, Spanish and Italian, of which about 20 billion are unique, according to its dataset card. The snapshots run from 2014-15 to 2023-14, and each was processed with CCNet and split into head, middle and tail buckets by perplexity.
Thirty billion documents carry quality signals, and duplicates stay in the set, marked, not removed. Together's October 2023 post states 30 trillion deduplicated tokens from more than 100 trillion raw, over 40 pre-computed quality annotations, and calls V2 "to our best knowledge, the largest public dataset released specifically for LLM training", which is Together's claim.
The table gives the deduplicated head and middle counts per language as of 2026-11-17, as the card states them under its section headings. The card's raw head and middle total is 32.9 billion documents and 50.6 trillion estimated tokens.
| Language | Documents | Estimated tokens | Source |
|---|---|---|---|
| en | 14.5 billion | 20.5 trillion | V2 dataset card |
| de | 1.9 billion | 3.0 trillion | V2 dataset card |
| fr | 1.6 billion | 2.7 trillion | V2 dataset card |
| es | 1.8 billion | 2.8 trillion | V2 dataset card |
| it | 0.9 billion | 1.5 trillion | V2 dataset card |
| Total | 20.8 billion | 30.4 trillion | V2 dataset card |
V2 is web text only, and Together pointed users to RedPajama-V1 for Wikipedia and other sources, and to The Stack for code.
Every page in V2 came through Common Crawl's archive, so the 84 snapshot identifiers fix which crawls can contain a page, and how that crawl collects pages is covered on the page about what Common Crawl contains.
RedPajama's Terms: Per-Source Rules for the Data, Apache 2.0 for the Code
RedPajama has no single data license: the RedPajama-Data-1T card tells users to "refer to the licenses of the data subsets you use", and the RedPajama-V2 card refers them to the Common Crawl Foundation Terms of Use, while Apache 2.0 covers only Together's code.
The 1T card adds a per-source list: Common Crawl's Terms of Use, the C4 license, GitHub limited to MIT, BSD or Apache projects, arXiv's Terms of Use, the Wikipedia license and the StackExchange license on the Internet Archive.
The V2 card says the code used to load and process the dataset is licensed under Apache 2.0, and the repository LICENSE is Apache License 2.0, with the README notice "Copyright 2023 Together Computer".
The table compares the layers as of 2026-11-17.
| Layer | Terms as stated | What it covers | Source |
|---|---|---|---|
| RedPajama-Data-1T data | "Please refer to the licenses of the data subsets you use" plus a per-source list | each slice under its source's terms | 1T dataset card |
| C4 slice inside 1T | ODC-BY plus Common Crawl Terms of Use | the compilation, not the pages inside it | allenai/c4 card; opendatacommons.org |
| Book slice (defunct) | links to the the_pile_books3 and pg19 license sections | no rightholder license recorded | GitHub rp_v1 README |
| RedPajama-V2 data | "Please refer to the Common Crawl Foundation Terms of Use for the data"; those terms last updated 2024-03-07 | use of the crawl content | V2 dataset card; commoncrawl.org/terms-of-use |
| Code (both releases) | Apache License 2.0, "Copyright 2023 Together Computer" | loading and processing scripts, not the data | V2 dataset card; GitHub LICENSE and README |
The C4 slice brings ODC-BY plus Common Crawl's terms, and ODC-BY governs only the rights over the collection, not the individual contents (section 2.4). A post or thread that calls RedPajama Apache-licensed reads the code license as a data license.
None of these terms is a license from the people whose content sits in the set, and none settles whether training on it is lawful.
The card's phrase "fully open-source" describes Together's intent for the project, and the page reads the terms above as the actual permissions. How ODC, CC, Apache and custom terms compare is covered on the page about dataset licenses for AI training.
Models Trained on RedPajama and the Pending Authors' Complaint
Together's own 2024 paper says RedPajama has been used to train production models such as Snowflake Arctic, Salesforce's XGen and AI2's OLMo. The paper is arXiv 2411.12372, submitted 2024-11-19 for the NeurIPS 2024 Datasets and Benchmarks track, and the statement is the authors' claim.
The V2 card, last modified 2024-11-21, thanks Cerebras for SlimPajama and counts "over 500 models built on RedPajam[a] to date", which is Together's own figure. None of these companies is a buyer or licensee of RedPajama.
The court record is Tanzer v. Salesforce, Inc., N.D. Cal. 3:25-cv-08862-CRB, filed 2025-10-15, a proposed class action that alleges Salesforce's XGen models were trained on The Pile, RedPajama and Books3. A related case, Alexander v.
Salesforce, N.D. Cal. 5:25-cv-09560-VKD, was filed 2025-11-05. The status is pending as of 2026-11-09, and the tracker lists the consolidated In re Salesforce. Every statement in the complaint is an allegation, and the full record sits on the page for In re Salesforce: Books3 and RedPajama training claims.
No court ruling or licensing deal on file names RedPajama, as of 2026-11-17. Together's paper and the complaint are different sources, so the page treats neither as proof of the other.
Content in RedPajama: Benefits, Risks, Downsides and Alternatives for the Holder
For a website owner, developer or author whose work sits in RedPajama, the dataset brings no payment and no contract, and nothing done today changes the snapshots it was built from.
Benefits
- Each 1T record keeps a meta field with URL, timestamp and source, so inclusion can be checked record by record.
- The card and the repository name each source, so the route of a page into the set can be traced.
Risks
- Copies downloaded since 2023 stay with whoever downloaded them, and Together's October 2023 post counts more than 190,000 downloads of 1T, which is Together's claim.
- Together says models in production were trained on the data.
- The only court record naming RedPajama is a pending complaint, with no ruling.
Downsides
- The cards describe no removal form, contact address or opt-out.
- The GitHub slice was selected by license type, which the card does not read against training use.
Alternatives
- Reserve rights in machine-readable form and block the upstream crawler, which affects future snapshots only.
- Send a notice to the host that names the files.
- License the text directly to AI developers, as described for text corpora for AI training.
- Do not license at all and leave the content as it is.
Not licensing is a valid outcome.
Source Corpora, Crawler Controls and Court Records Beyond This Page
This page covers RedPajama-Data-1T and RedPajama-V2; the corpora inside them, the crawler, the court records and the wider law each have their own page.
- Court records: AI copyright lawsuits over training data, and fair use and AI training in US law.
- Rules and crawler controls: the EU text and data mining exception, robots.txt for AI crawlers and CCBot, Common Crawl's crawler.
- Content types: licensing source code to AI companies, books as AI training data and pirated books and shadow libraries in AI training.
- Other corpora: The Pile and Dolma.
- Documents and routes: dataset documentation and data cards, removing content from AI training datasets and alternatives to selling data, including not selling at all.
This page is information about a dataset, not legal advice.
Frequently Asked Questions About RedPajama
What is RedPajama?
RedPajama is a family of text datasets for training language models, published by Together. It has RedPajama-Data-1T (2023-04-17), a rebuild of the LLaMA training mix, and the web-only RedPajama-V2 (2023-10-30).
How big is RedPajama?
RedPajama-Data-1T holds about 1.2 trillion tokens, Together's figure. RedPajama-V2 holds over 100 billion documents from 84 snapshots, about 20 billion unique, and 30 trillion deduplicated tokens per Together's post.
What license does RedPajama use?
RedPajama has no single data license. The 1T card points to each source's terms and the V2 card to Common Crawl's Terms of Use, while Apache 2.0 covers Together's code only.
Does RedPajama contain Books3?
The 1T book slice combined pg19 and the-pile-books3, and the card now marks it defunct over reported infringement for Books3. RedPajama-V2 holds web text only. This page is information, not legal advice.