This page reads OpenWebText from the side of the people whose pages it holds. Within AI training data licensing, it is free web text: it brings the authors of the linked pages no payment and no contract. The page also separates WebText, OpenWebText and OpenWebText2, which are three different sets with three different selection rules.
Who Built OpenWebText and What It Set Out to Reproduce
OpenWebText was created by Aaron Gokaslan and Vanya Cohen of Brown University, who announced it on their project page as a beta release and "an open source effort to reproduce OpenAI's WebText dataset". The table lists the identity facts as of 2026-11-17.
| Attribute | Value | Source |
|---|---|---|
| Created by | Aaron Gokaslan and Vanya Cohen of Brown University, per the project page | skylion007.github.io/OpenWebTextCorpus |
| Release | beta, 2019 (citation year); "an open source effort to reproduce OpenAI's WebText dataset" | skylion007.github.io/OpenWebTextCorpus |
| Reproduces | WebText, described in the GPT-2 paper (OpenAI, 2019) | cdn.openai.com GPT-2 paper |
| Hosted copy | Skylion007/openwebtext on Hugging Face; not gated; last modified 2026-07-21 | huggingface.co |
| Packaging license | CC0 (cc0-1.0) | skylion007.github.io; huggingface.co |
| Takedown contact | openwebtext at gmail.com (project page); the card adds datasets at huggingface.co | skylion007.github.io; huggingface.co |
| Registry number | not applicable to a project page; none checked | n/a |
The creators wrote that, given OpenAI's "limited release of information around WebText and GPT-2", there may be further room for improvement, and they invited contributions. The creators are named as the project page names them.
The copy most users load is Skylion007/openwebtext on Hugging Face, whose card calls it "an open-source replication of the WebText dataset from OpenAI, that was used to train GPT-2". OpenWebText is one of the public corpora compared on the page about major AI training datasets.
WebText, the OpenAI Set Behind It: Reddit Links With at Least 3 Karma
WebText is the dataset OpenAI researchers described in the 2019 GPT-2 paper, built from outbound links on Reddit that received at least 3 karma.
The paper's reason is that a scrape of pages "curated/filtered by humans" avoids the quality problems of a raw crawl, with karma serving as "a heuristic indicator for whether other users found the link interesting, educational, or just funny". The paper counts 45 million links and extracts their text with the Dragnet and Newspaper extractors.
The version used in the paper excludes links created after December 2017. After de-duplication and heuristic cleaning it holds slightly over 8 million documents and 40 GB, and the authors removed all Wikipedia documents to avoid overlap with test tasks.
The acknowledgements thank "everyone who wrote the text, shared the links, and upvoted the content in WebText".
The project page calls OpenAI's release of information around WebText limited, and that is the gap OpenWebText was built to fill. The paper's own description of WebText is therefore the only primary account of its selection, and OpenWebText's makers could rebuild the rule only from that description.
The 2019 rule is historical: it describes one dataset and says nothing about how any model is trained today. How community votes and forum archives are licensed to AI developers belongs to the page on forum and community archives as AI training data.
How OpenWebText Was Built and What One Record Holds
OpenWebText was built from the URLs of Reddit posts, downloaded page by page and filtered in six steps that the project page lists. The table compares the three sets as of 2026-11-17.
- Reddit post URLs are taken from "the Reddit submissions dataset", deduplicated, stripped of non-HTML content and shuffled.
- The pages are downloaded on several machines in parallel and their text is extracted with the newspaper Python package.
- Non-English pages are removed with Facebook FastText.
- Near-duplicates are removed with locality-sensitive hashing on 5-gram sets, above a 0.5 similarity threshold.
- Documents under 128 tokens are removed.
- The result is 38 GB (40 GB in SI units) from 8,013,769 documents.
| Set | Selection rule as its makers state it | Time range | Size as stated | Source |
|---|---|---|---|---|
| WebText (OpenAI) | outbound Reddit links with at least 3 karma; 45 million links; Wikipedia documents removed | links created up to December 2017 in the paper's version | slightly over 8 million documents, 40 GB | GPT-2 paper, 2019 |
| OpenWebText | all Reddit post URLs from the Reddit submissions dataset; no karma threshold stated | not stated | 38 GB (40 GB SI), 8,013,769 documents; card: 8,013,769 rows, 39.77 GB generated | project page, 2019; Hugging Face card |
| OpenWebText2 (plug and play) | minimum combined Reddit score 3 | Reddit submissions 2005 to April 2020 | 17,103,059 documents, 65.86 GB uncompressed | OpenWebText2 documentation |
The project page states no karma threshold and no date range, unlike WebText. The hosted copy has one train split of 8,013,769 rows and a single text field, with about 39.77 GB generated, as the card states.
No URL, title or date travels with a record, so a page owner cannot see from the file which page a record came from. This differs from the C4 copy on Hugging Face, where each record keeps its URL, and it means a check for one page has to search the text itself.
OpenWebText fetched the linked pages itself, in parallel, instead of drawing them from a crawl archive. How a crawl archive feeds other corpora is covered on the page about what Common Crawl contains, the source behind sets such as the one the C4 page describes.
OpenWebText2: The Extended Reddit Scrape Inside a Larger Corpus
OpenWebText2 is the extended version that its documentation describes as covering "all Reddit submissions from 2005 up until April 2020", released in a filtered plug-and-play version and a larger raw version. The table lists both as of 2026-11-17, with the weight the corpus carries in The Pile.
| Version | Filtering | Documents | Uncompressed | Compressed | Source |
|---|---|---|---|---|---|
| Plug and play | minimum combined Reddit score 3; deduplicated by URL and at document level with MinHashLSH | 17,103,059 | 65.86 GB | 28 GB | openwebtext2.readthedocs.io |
| Raw scrapes | deduplicated by URL only | 69,547,149 | 193.89 GB | 79 GB | openwebtext2.readthedocs.io |
| In The Pile | raw size 62.77 GiB, weight 10.01 %, 2.0 epochs | n/a | n/a | n/a | arXiv 2101.00027, Table 1 |
The plug-and-play version keeps URLs with a minimum combined Reddit score of 3, which differs from the project page of the original, where no threshold is stated. Both versions carry "useful metadata for each document, including several original Reddit fields".
The Pile paper, whose authors give EleutherAI as their affiliation, describes OpenWebText2 as "a generalized web scrape dataset inspired by WebText" that uses "net upvotes on Reddit submissions as a proxy for outgoing link quality" and extracts text with Newspaper "for consistency with OpenWebTextCorpus".
In The Pile, OpenWebText2 is one of 22 components. The documentation's license page carries the MIT License, "Copyright (c) 2020 EleutherAI", worded for software and its documentation, and the documentation names no takedown contact.
The weight of 10.01 % and the 2.0 epochs are The Pile authors' choices of how often to sample the component, not properties of OpenWebText2 itself. What the whole corpus holds and its legal history belong to the page about The Pile.
OpenWebText's Terms: CC0 on the Packaging, Not on the Text
OpenWebText's creators license "the actual packaging" of the corpus under Creative Commons CC0 and state, in the same section, "We do not own any of the text from which these data has been extracted." The card's license field reads cc0-1.0. The table sets the layers side by side as of 2026-11-17.
| Layer | Terms | What it covers | What it leaves out | Source |
|---|---|---|---|---|
| The packaging (OpenWebText) | CC0, "no rights reserved" | "the actual packaging of these parallel data" | the text itself: "We do not own any of the text" | skylion007.github.io; huggingface.co |
| The pages' text | the page authors' own rights | n/a | no license from the creators | n/a |
| OpenWebText2 | MIT License, Copyright (c) 2020 EleutherAI, on the documentation's license page | "the Software" and "associated documentation files" | no statement on the scraped text | openwebtext2.readthedocs.io |
CC0 waives the creators' own rights in the packaging, meaning the selection, arrangement and files, and it does not license the pages' text, whose rights stay with the page authors.
Creative Commons states that its licenses apply only where copyright permission is required (checked 2026-11-09), so CC0 on the packaging does not settle whether training on the pages needs permission.
The two layers read together mean that the creators grant what they hold and state plainly what they do not. This page does not describe OpenWebText as public domain, free for commercial use or open source. How CC0, ODC and CDLA terms compare is covered on the page about dataset licenses for AI training.
Takedown Requests and OpenWebText's Legal Record
OpenWebText has a written notice-and-takedown policy: an owner who believes the corpus contains their material can write to openwebtext at gmail.com, and the creators say they will remove the affected sources "from the next release of the corpus".
A notice must identify the claimant with contact details, name the copyrighted work and identify the material with enough information to locate it.
The card repeats the policy and adds datasets at huggingface.co as a contact. The policy is written for copyright claims, and it describes a removal from a future release, not from copies already distributed.
The record shows a hosted repository that was last modified on 2026-07-21 and is not gated. The card's section on personal and sensitive information reads "More Information Needed", so it offers no account of what personal data the 8,013,769 documents contain.
The project page names no release after the 2019 beta, so whether any source was removed is not stated.
No court ruling or case on file names OpenWebText, WebText or OpenWebText2 in its recorded claims, as of 2026-11-17, and no licensing deal on file covers them. Tanzer v.
Salesforce (N.D. Cal. 3:25-cv-08862-CRB), pending as of 2026-11-09, alleges training on The Pile, which contains OpenWebText2. The search covered the court records and deals tracked on this site. Pending and decided matters over training data are listed in the tracker of AI copyright lawsuits over training data.
A Page Linked From Reddit: Benefits, Risks, Downsides and Alternatives for the Owner
For the owner of a page that a Reddit user linked before 2019, inclusion in OpenWebText brings no payment and no contract, and blocking a crawler today does not change copies already downloaded. The four lists below separate what inclusion does from what it costs and end with the owner's options, including doing nothing.
Benefits
- The creators publish a takedown route.
- The GPT-2 paper credits, in general terms, the people who wrote and linked the text.
Risks
- Copies downloaded since 2019 stay with whoever downloaded them.
- The card says nothing about personal information.
- No court has ruled on training with OpenWebText, as of 2026-11-17.
Downsides
- A record carries no URL, so finding one's page means searching the text.
- Removal is promised only for a next release that the page does not date.
Alternatives
- Send a notice with the three items the policy lists.
- Reserve rights in machine-readable form for future collection, under Article 4(3) of Directive (EU) 2019/790.
- License the text directly to AI developers, as the page on text corpora for AI training describes.
- Not license at all and leave the pages as they are.
The benefits are narrow and the risks depend on court outcomes that have not arrived. Not licensing is a valid outcome, and the choice stays with the owner.
Reddit's Own Licenses, Other Corpora and Removal Beyond This Page
This page covers OpenWebText, WebText and OpenWebText2; Reddit's own licenses, the crawl archives, other corpora and the law each have their own page. Personal-data rules for training sets differ by jurisdiction.
- Reddit's AI data licensing deals: how Reddit licenses its own data.
- OpenAI as a data buyer: the lab behind WebText, as a buyer today.
- C4 (Colossal Clean Crawled Corpus): its webtextlike config uses OpenWebText URLs.
- FineWeb: a newer filtered web set.
- Common Pile: EleutherAI's later, openly licensed corpus.
- robots.txt for AI crawlers: blocking future collection.
- Machine-readable TDM reservations: reservations a machine can read.
- EU text and data mining exception: Article 4 and the reservation.
- Copyright and AI training across jurisdictions: fair use and TDM exceptions.
- Is web scraping for AI training legal: copyright, contract and computer-misuse law.
- Privacy law and AI training data: personal data in training sets.
- Dataset documentation, datasheets and data cards: what a dataset card should record.
- Licensed vs scraped training data: the licensed alternative.
- Alternatives to selling data, including not selling at all: not licensing at all.
- Removing content from AI training datasets: removal routes across datasets.
Frequently Asked Questions About OpenWebText
What is OpenWebText?
OpenWebText is an open reproduction of OpenAI's WebText, created by Aaron Gokaslan and Vanya Cohen of Brown University in 2019. The project page states 38 GB from 8,013,769 documents, packaged under CC0.
Is OpenWebText the same as WebText?
No: WebText is OpenAI's set from the GPT-2 paper, built from Reddit links with at least 3 karma and about 40 GB. OpenWebText rebuilt it from Reddit post URLs with its own filters and states no karma threshold.
What license does OpenWebText use?
The creators license the packaging under CC0 and state that they do not own any of the text. The pages inside keep their authors' rights, and CC0 does not settle whether training on them needs permission.
What is OpenWebText2?
OpenWebText2 is the extended scrape of Reddit submissions from 2005 to April 2020, with 17,103,059 documents in its plug-and-play version. It is one of the 22 components of The Pile. This page is information, not legal advice.
Agents and advisers can use the introducer form.