DataPacts

RedPajama

Checked

At a glance

RedPajama is a family of text datasets for training large language models, published by Together with RedPajama-Data-1T on 2023-04-17, a rebuild of the LLaMA training mix of about 1.2 trillion tokens, and followed on 2023-10-30 by the web-only RedPajama-V2, which Together says holds 30 trillion deduplicated tokens from 84 Common Crawl snapshots.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

Both sets were assembled from public sources without a license from the people whose pages, code, papers or books they hold, and the book slice of the first set is now marked defunct.

This page reads RedPajama from that side: within AI training data licensing, it is free text next to licensed text, with per-source terms, no data license of its own and one pending complaint.

Who Publishes RedPajama: Together and Its 2023 Research Partners

RedPajama is published by Together Computer, Inc., a Delaware corporation that trades as Together AI, which announced it on 2023-04-17 as a collaboration with Ontocord.ai, ETH DS3Lab, Stanford CRFM and Hazy Research. The table lists the identity facts as of 2026-11-17.

Attribute Value Source
Publisher Together Computer, Inc., "a Delaware corporation" per its Terms of Service (checked 2026-11-09); brand Together AI together.ai/terms-of-service
Partners named in April 2023 Ontocord.ai, ETH DS3Lab, Stanford CRFM, Hazy Research together.ai/blog/redpajama
Partners thanked on the V2 card Ontocord.ai, MILA Québec AI Institute, ETH DS3Lab, Université de Montréal, Stanford CRFM, Stanford Hazy Research, LAION huggingface.co/datasets/togethercomputer/RedPajama-Data-V2
RedPajama-Data-1T announced 2023-04-17; Hub repository created 2023-04-17, last modified 2024-06-17, not gated together.ai; huggingface.co
RedPajama-V2 announced 2023-10-30; Hub repository created 2023-10-26, last modified 2024-11-21, not gated together.ai; huggingface.co
Code GitHub togethercomputer/RedPajama-Data, created 2023-04-14, Apache License 2.0 github.com
Paper Weber et al., arXiv 2411.12372, submitted 2024-11-19, NeurIPS 2024 Datasets and Benchmarks track arxiv.org/abs/2411.12372
Registry number not checked n/a

The April 2023 post framed the data as the first of three steps, pre-training data, base models and instruction tuning, toward "fully open-source models". The later V2 card thanks a longer partner list than the first post, so the page shows both lists with their sources and never merges them.

Hugging Face, Inc. hosts both cards and neither repository is gated, but Hugging Face is the host, not the publisher. RedPajama is one of the corpora compared on the page about major AI training datasets.

What RedPajama-Data-1T Holds: The Slices Behind 1.2 Trillion Tokens

RedPajama-Data-1T holds about 1.2 trillion tokens in six slices that are still downloadable, from web pages, an existing web corpus, code, scientific papers, an encyclopedia and question-and-answer sites, plus a book slice that is now defunct.

The dataset card calls it "a clean-room, fully open-source implementation of the LLaMa dataset", built to follow the LLaMA paper as closely as possible and primarily English. The Hub copy has 2,084 JSONL files and a 1-billion-token sample, and the April 2023 post gives about 5 TB unzipped and about 3 TB compressed.

The table compares the slices of RedPajama-Data-1T as of 2026-11-17, in the order of the card, with the book slice last. The total on the card is 1.2 trillion tokens, Together's figure, and the page never adds the rows.

Slice Tokens, as Together states Where it came from Status Source
Commoncrawl 878 billion (checked 2026-11-09) five Common Crawl dumps, 2019-30 to 2023-06, cc_net, paragraph deduplication, Wikipedia-reference classifier available dataset card; GitHub rp_v1
C4 175 billion C4 downloaded from Hugging Face, format change only available dataset card
GitHub 59 billion Google BigQuery; MIT, BSD or Apache projects only available dataset card
ArXiv 28 billion LaTeX sources, requester-pays bucket; preambles, comments, macros and bibliographies removed available dataset card
Wikipedia 24 billion Hugging Face Wikipedia set, 2023-03-20 dump, 20 languages available dataset card
StackExchange 20 billion Internet Archive, 28 largest sites, question-answer pairs available dataset card
Book 26 billion (April 2023 post; no row on the current card) pg19 subset of Project Gutenberg and the-pile-books3, from Hugging Face "defunct and no longer accessible" together.ai; GitHub rp_v1; dataset card

The web slice uses five Common Crawl dumps, 2019-30, 2020-05, 2021-04, 2022-05 and 2023-06, so the crawl behind it is fixed at those dates. A linear classifier trained on pages cited by Wikipedia kept the pages it rated close to them.

The GitHub slice was selected by project license, MIT, BSD or Apache, and files were deduplicated. Each record carries a text field and a meta field with URL, timestamp and source, which lets a site owner trace a page.

The card calls the set primarily English, and only the Wikipedia slice is stated to span 20 languages.

The C4 slice is C4 "downloaded from Huggingface" with only a format change, so its contents and terms are C4's own. What C4 holds belongs to the page on C4, the Colossal Clean Crawled Corpus.

RedPajama's Book Slice and Why Together Marks It Defunct

The book slice of RedPajama-Data-1T is no longer available: the dataset card marks the "book" configuration as defunct due to reported copyright infringement for the Books3 collection it contained.

Together's repository, on its rp_v1 branch, says the slice was built from the pg19 subset of Project Gutenberg and "the-pile-books3", both downloaded from Hugging Face and deduplicated with SimHash.

The April 2023 post described the same slice as "a corpus of open books, deduplicated by content similarity" and gave it 26 billion tokens. The page states both descriptions with their sources and does not judge intent.

The card's notice reads that the configuration is "defunct and no longer accessible due to reported copyright infringement for the Book3 dataset contained in this config".

The card states no date and names no sender for the step. The current card table lists no book row while still giving a total of 1.2 trillion tokens, so the page never adds or subtracts slice counts.

The rp_v1 README had listed the licenses of the_pile_books3 and pg19 as the terms for this slice, and no rightholder license is recorded for either. The repository names the origin of the slice as two Hugging Face copies, not as a license from any author or publisher.

Where Books3 came from, how it left its host in 2023 and which dockets name it are covered on the page about what Books3 is and how it was removed.

What RedPajama-V2 Holds: 84 Web Snapshots, Five Languages and Quality Signals

RedPajama-V2 holds over 100 billion web documents from 84 Common Crawl snapshots in English, German, French, Spanish and Italian, of which about 20 billion are unique, according to its dataset card. The snapshots run from 2014-15 to 2023-14, and each was processed with CCNet and split into head, middle and tail buckets by perplexity.

Thirty billion documents carry quality signals, and duplicates stay in the set, marked, not removed. Together's October 2023 post states 30 trillion deduplicated tokens from more than 100 trillion raw, over 40 pre-computed quality annotations, and calls V2 "to our best knowledge, the largest public dataset released specifically for LLM training", which is Together's claim.

The table gives the deduplicated head and middle counts per language as of 2026-11-17, as the card states them under its section headings. The card's raw head and middle total is 32.9 billion documents and 50.6 trillion estimated tokens.

Language Documents Estimated tokens Source
en 14.5 billion 20.5 trillion V2 dataset card
de 1.9 billion 3.0 trillion V2 dataset card
fr 1.6 billion 2.7 trillion V2 dataset card
es 1.8 billion 2.8 trillion V2 dataset card
it 0.9 billion 1.5 trillion V2 dataset card
Total 20.8 billion 30.4 trillion V2 dataset card

V2 is web text only, and Together pointed users to RedPajama-V1 for Wikipedia and other sources, and to The Stack for code.

Every page in V2 came through Common Crawl's archive, so the 84 snapshot identifiers fix which crawls can contain a page, and how that crawl collects pages is covered on the page about what Common Crawl contains.

RedPajama's Terms: Per-Source Rules for the Data, Apache 2.0 for the Code

RedPajama has no single data license: the RedPajama-Data-1T card tells users to "refer to the licenses of the data subsets you use", and the RedPajama-V2 card refers them to the Common Crawl Foundation Terms of Use, while Apache 2.0 covers only Together's code.

The 1T card adds a per-source list: Common Crawl's Terms of Use, the C4 license, GitHub limited to MIT, BSD or Apache projects, arXiv's Terms of Use, the Wikipedia license and the StackExchange license on the Internet Archive.

The V2 card says the code used to load and process the dataset is licensed under Apache 2.0, and the repository LICENSE is Apache License 2.0, with the README notice "Copyright 2023 Together Computer".

The table compares the layers as of 2026-11-17.

Layer Terms as stated What it covers Source
RedPajama-Data-1T data "Please refer to the licenses of the data subsets you use" plus a per-source list each slice under its source's terms 1T dataset card
C4 slice inside 1T ODC-BY plus Common Crawl Terms of Use the compilation, not the pages inside it allenai/c4 card; opendatacommons.org
Book slice (defunct) links to the the_pile_books3 and pg19 license sections no rightholder license recorded GitHub rp_v1 README
RedPajama-V2 data "Please refer to the Common Crawl Foundation Terms of Use for the data"; those terms last updated 2024-03-07 use of the crawl content V2 dataset card; commoncrawl.org/terms-of-use
Code (both releases) Apache License 2.0, "Copyright 2023 Together Computer" loading and processing scripts, not the data V2 dataset card; GitHub LICENSE and README

The C4 slice brings ODC-BY plus Common Crawl's terms, and ODC-BY governs only the rights over the collection, not the individual contents (section 2.4). A post or thread that calls RedPajama Apache-licensed reads the code license as a data license.

None of these terms is a license from the people whose content sits in the set, and none settles whether training on it is lawful.

The card's phrase "fully open-source" describes Together's intent for the project, and the page reads the terms above as the actual permissions. How ODC, CC, Apache and custom terms compare is covered on the page about dataset licenses for AI training.

Models Trained on RedPajama and the Pending Authors' Complaint

Together's own 2024 paper says RedPajama has been used to train production models such as Snowflake Arctic, Salesforce's XGen and AI2's OLMo. The paper is arXiv 2411.12372, submitted 2024-11-19 for the NeurIPS 2024 Datasets and Benchmarks track, and the statement is the authors' claim.

The V2 card, last modified 2024-11-21, thanks Cerebras for SlimPajama and counts "over 500 models built on RedPajam[a] to date", which is Together's own figure. None of these companies is a buyer or licensee of RedPajama.

The court record is Tanzer v. Salesforce, Inc., N.D. Cal. 3:25-cv-08862-CRB, filed 2025-10-15, a proposed class action that alleges Salesforce's XGen models were trained on The Pile, RedPajama and Books3. A related case, Alexander v.

Salesforce, N.D. Cal. 5:25-cv-09560-VKD, was filed 2025-11-05. The status is pending as of 2026-11-09, and the tracker lists the consolidated In re Salesforce. Every statement in the complaint is an allegation, and the full record sits on the page for In re Salesforce: Books3 and RedPajama training claims.

No court ruling or licensing deal on file names RedPajama, as of 2026-11-17. Together's paper and the complaint are different sources, so the page treats neither as proof of the other.

Content in RedPajama: Benefits, Risks, Downsides and Alternatives for the Holder

For a website owner, developer or author whose work sits in RedPajama, the dataset brings no payment and no contract, and nothing done today changes the snapshots it was built from.

Benefits

  • Each 1T record keeps a meta field with URL, timestamp and source, so inclusion can be checked record by record.
  • The card and the repository name each source, so the route of a page into the set can be traced.

Risks

  • Copies downloaded since 2023 stay with whoever downloaded them, and Together's October 2023 post counts more than 190,000 downloads of 1T, which is Together's claim.
  • Together says models in production were trained on the data.
  • The only court record naming RedPajama is a pending complaint, with no ruling.

Downsides

  • The cards describe no removal form, contact address or opt-out.
  • The GitHub slice was selected by license type, which the card does not read against training use.

Alternatives

  • Reserve rights in machine-readable form and block the upstream crawler, which affects future snapshots only.
  • Send a notice to the host that names the files.
  • License the text directly to AI developers, as described for text corpora for AI training.
  • Do not license at all and leave the content as it is.

Not licensing is a valid outcome.

Source Corpora, Crawler Controls and Court Records Beyond This Page

This page covers RedPajama-Data-1T and RedPajama-V2; the corpora inside them, the crawler, the court records and the wider law each have their own page.

This page is information about a dataset, not legal advice.

Frequently Asked Questions About RedPajama

What is RedPajama?

RedPajama is a family of text datasets for training language models, published by Together. It has RedPajama-Data-1T (2023-04-17), a rebuild of the LLaMA training mix, and the web-only RedPajama-V2 (2023-10-30).

How big is RedPajama?

RedPajama-Data-1T holds about 1.2 trillion tokens, Together's figure. RedPajama-V2 holds over 100 billion documents from 84 snapshots, about 20 billion unique, and 30 trillion deduplicated tokens per Together's post.

What license does RedPajama use?

RedPajama has no single data license. The 1T card points to each source's terms and the V2 card to Common Crawl's Terms of Use, while Apache 2.0 covers Together's code only.

Does RedPajama contain Books3?

The 1T book slice combined pg19 and the-pile-books3, and the card now marks it defunct over reported infringement for Books3. RedPajama-V2 holds web text only. This page is information, not legal advice.