DataPacts

FineWeb

Checked

At a glance

FineWeb is an English web-text dataset for pretraining language models, published by Hugging Face and filtered from Common Crawl's public web archive, which the 2024 paper describes as 15 trillion tokens from 96 crawl snapshots.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

Each record keeps the text of one page together with its source URL and crawl date, so the owner of a website can check whether a page is in it.

The set is released under ODC-By 1.0 and is also bound by the crawl's Terms of Use. It sits among the open web-text corpora that a licensed text deal competes with.

This page reads it from the side of the people whose pages it holds. Within AI training data licensing, FineWeb is the free counterpart to licensed text: it brings its source sites no payment and no contract, and the card describes no domain-level removal route.

Hugging Face's FineWeb: Publisher, Paper and Stated Purpose

FineWeb is published by Hugging Face, Inc., a Delaware corporation, which curated and funded the dataset and hosts it on its own Hub. The table lists the identity facts as of 2026-11-16.

Attribute Value Source
Publisher Hugging Face, Inc., a Delaware corporation per its Terms of Service huggingface.co/terms-of-service
Curated and funded by Hugging Face, per the paper's datasheet arxiv.org/abs/2406.17557
Paper "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale", arXiv 2406.17557, submitted 2024-06-25; NeurIPS 2024 Datasets and Benchmarks Track per the card's citation arxiv.org; huggingface.co
First version v1.0.0, 2024-04-21 huggingface.co/datasets/HuggingFaceFW/fineweb
Stated purpose "primarily intended to be used as a research artifact on public data" for pretraining dataset card
Hosting Hugging Face Hub, DOI 10.57967/hf/2493 arxiv.org (Appendix B, C)
Point of contact a discussion on the dataset's Community tab dataset card
Registry number not checked n/a

The card says FineWeb was originally meant to be a fully open replication of RefinedWeb, released in full under ODC-By 1.0.

The stated purpose is research, and the card does not describe any program that pays the owners of the pages it holds. The registry number of the company has not been checked, so none is given.

The dataset, the paper and the Hub page are three separate records, and each is dated here by what its own page states, so a reader can trace every figure to the document that carries it.

Hugging Face is named by its full legal name throughout, and no other dataset creator is merged with it. FineWeb is one of the public corpora compared on the page about major AI training datasets.

What FineWeb Holds: Crawl Snapshots, Record Fields and Size

FineWeb holds English text extracted from web pages in Common Crawl's archive, drawn from 110 crawl dumps from CC-MAIN-2013-20 to CC-MAIN-2025-26, with one record per page. The dataset is English only, selected by a language score of at least 0.65.

Each record keeps the extracted text, the page URL, the crawl date, the dump ID and the path of the WARC file the page came from, together with the language score and a token count.

The URL and the WARC path are the fields that matter to a site owner, because together they point from a record back to the crawl file where the page was captured. The dataset does not hold images or other media, only the extracted text of each page.

The size is stated in two places, and the two differ in date. The paper of 2024-06-25 gives 15 trillion tokens from 96 snapshots.

The card's own summary paragraph still says 96 dumps from the summer of 2013 to April 2024 and around 15 trillion tokens by the gpt2 tokenizer, while its dump table lists 110 dumps. This page shows both as the publisher states them and does not state a current total.

Attribute Value Source
Source Common Crawl web archive commoncrawl.org/about
Dumps in the card's table 110, CC-MAIN-2013-20 to CC-MAIN-2025-26 dataset card, checked 2026-11-09 and 2026-11-16
Paper (2024-06-25) 15 trillion tokens from 96 snapshots arxiv.org/abs/2406.17557
Card summary paragraph 96 dumps, summer 2013 to April 2024, around 15 trillion gpt2 tokens dataset card, checked 2026-11-16
Language English only; language score at least 0.65 dataset card
Record fields text, id, dump, url, date (crawl date), file_path (WARC path), language, language_score, token_count dataset card; paper Appendix A

The pages came from Common Crawl's crawler and archive. How that crawl collects pages and how a site keeps them out is covered on the page about what Common Crawl contains, and the crawler has its own entry, CCBot, Common Crawl's crawler.

What FineWeb's Filters Keep and Drop

FineWeb keeps only pages that pass a URL blocklist, an English language check, quality and repetition filters and per-crawl deduplication, and it rewrites email and public IP addresses before release. The card lists the steps in this order.

  1. URL filtering against a blocklist of malicious and NSFW sites.
  2. Main-text extraction from the HTML.
  3. An English language score of at least 0.65.
  4. Quality and repetition filters.
  5. MinHash deduplication within each crawl.
  6. PII formatting, which rewrites email and public IP addresses.

For a site owner, being dropped depends on these filters and not on any request. A page in English that passes them is kept, and the filters themselves read no signal its owner sent after the crawl.

The card describes the filters as a pipeline that runs before release, and the paper presents the choice of each step as an engineering decision about text quality.

A page is also not kept because it is valuable: the filters judge form and language, and none of the steps checks a license or a rights reservation.

FineWeb's Terms: ODC-By 1.0 Plus the Crawl's Terms of Use

Hugging Face releases FineWeb under the Open Data Commons Attribution License (ODC-By) 1.0, and the card adds that use of the dataset is also subject to Common Crawl's Terms of Use.

The paper's Appendix B states the same two layers. Common Crawl's Terms of Use were last updated on 2024-03-07, as read on 2026-11-09. The table sets the layers side by side as of 2026-11-16.

Layer Terms What it covers What it leaves out Source
The dataset ODC-By 1.0 rights over the database (s.2.4) the individual pages inside it, which stay under their own rights huggingface.co; opendatacommons.org
The upstream crawl Common Crawl Terms of Use, last updated 2024-03-07 use of the crawl data, which the card adds to the dataset's terms detail on the Common Crawl page commoncrawl.org/terms-of-use

ODC-By section 2.4 states that the license governs rights over the database and not its individual contents. The license on FineWeb therefore does not license the pages inside it, whose text stays under its authors' rights.

The license also does not settle whether training on those pages is lawful, which the legal standing section below addresses.

This page does not characterize the license as permitting any particular use of the pages, and the paper's own word "permissive" is the paper's characterization. How ODC licenses compare with CC and CDLA terms is covered on the page about dataset licenses for AI training.

Personal Data and Takedown Notices in FineWeb

FineWeb's card states that Hugging Face replaced email addresses and public IP addresses in the text, and it offers a form for anyone who finds their own personal data in the dataset.

Emails are replaced with placeholder addresses such as email@example.com, and public IP addresses are replaced with randomly generated addresses. Phone numbers are left as they are, because a pattern for them produced too many false positives.

The card itself says that it is very likely some personal data is still present. For that case it points to a PII removal form, and it names a discussion on the dataset's Community tab as the point of contact.

Personal-data rules differ by jurisdiction, and no legal conclusion is drawn here. The anonymization step handles two kinds of identifier only, so names, postal addresses and other personal details that appear in running text are not covered by it, and the card's own caveat is the guide to what remains.

One removal dispute is on record. Torstar sent Hugging Face a cease-and-desist letter over FineWeb, dated 2025-01-22, which listed 361 domains in Schedule A and asked for a response by 2025-01-31. Hugging Face publishes the notices it receives in a public takedown-notice repository.

Hugging Face's repository lists the letter with its date, so anyone can read what was asked. The notice is a letter and not a lawsuit, and this page does not state which version of the dataset was changed afterwards or when.

Notices like this one work as provenance records, a role described on the page about training data provenance.

The card describes no route for copyright holders or site owners to request removal of a domain, and it describes no filtering based on robots.txt.

The absence was checked on the card and the paper on 2026-11-16. This page makes no statement about how Hugging Face answers requests other than the personal-data form. Removal routes across datasets are collected on the page about removing content from AI training datasets.

FineWeb-Edu and FineWeb2: The Releases Built Alongside It

FineWeb-Edu is a subset of FineWeb filtered for educational content, and FineWeb2 is a separate multilingual release covering 1,868 language-script pairs.

FineWeb-Edu holds 1.3 trillion tokens according to the paper and the card, and 5.4 trillion tokens in the version with the lower score threshold. The paper states that both datasets are released under "the permissive ODC-By License", which is its wording, not a DataPacts characterization.

FineWeb2 is about 20 TB, with 5 billion documents and over 3 trillion words, and its latest version on the card is v2.1.1 of 2025-10-27. Its license and its crawl sources have not been checked here, so none is stated. The table lists the releases alphabetically as of 2026-11-16.

Release What it is Size, as the publisher states Terms Source
FineWeb English web text from crawl dumps paper: 15 trillion tokens, 96 snapshots (2024); card table: 110 dumps ODC-By 1.0 plus crawl Terms of Use huggingface.co; arxiv.org
FineWeb-Edu educational subset of FineWeb 1.3 trillion tokens; 5.4 trillion for the score-2 version ODC-By per the paper huggingface.co/datasets/HuggingFaceFW/fineweb-edu; arxiv.org
FineWeb2 multilingual release 1,868 language-script pairs; about 20 TB, 5 billion documents, over 3 trillion words; v2.1.1, 2025-10-27 not recorded huggingface.co/datasets/HuggingFaceFW/fineweb-2

No court ruling or pending case on file names FineWeb, as of 2026-11-16, and the only dispute on record is Torstar's notice to its host.

No licensing deal on file names FineWeb either, so the dataset has no price, no counterparty and no court-set value on record. The search covered the court records and deals tracked on this site on 2026-11-16.

In the EU, a rightholder's reservation against text and data mining of content made publicly available online must be made in an appropriate manner, such as machine-readable means, under Article 4(3) of Directive (EU) 2019/790.

A reservation that reached the crawler before a snapshot is the one that could matter for later copies of that snapshot. The card does not say that FineWeb honors such reservations, so no claim is made that it does. The rule is set out on the page about the EU text and data mining exception.

The upstream crawler states that it checks robots.txt before fetching a page, but the files of past crawls stay unchanged: Common Crawl says its archive files cannot be edited after publication and that URLs it is asked to remove are filtered from subsequent crawls and its public indexes (statement of 2025-11-04).

A block placed in 2026 therefore does not change FineWeb dumps already published. What a block does change is the next snapshot: pages that the crawler cannot fetch are not captured, and a later FineWeb release built from later dumps would not contain them.

Whether and when such a release appears is Hugging Face's decision, and no schedule is stated on the card.

A Website in FineWeb: Benefits, Risks, Downsides and Alternatives for the Owner

For the owner of a website whose pages sit in FineWeb, the dataset brings no payment and no contract, and blocking the crawler today does not reach dumps already released. The four lists below separate what inclusion does from what it costs, and they end with the options the owner has, including doing nothing.

Benefits

  • Attribution travels with the database under ODC-By, as a database-level duty only.
  • Each record names its source URL, so inclusion can be checked page by page.

Risks

  • Copies downloaded before any removal stay with whoever downloaded them.
  • Personal data may remain despite the anonymization step.
  • No court has ruled on training with FineWeb.

Downsides

  • Text available free in a public set can be harder to license for a fee.
  • The card offers no domain or copyright removal route.

Alternatives

  • Reserve rights in machine-readable form and block the upstream crawler for future snapshots.
  • Send a takedown notice naming each domain, as Torstar did.
  • License the text directly to AI developers, as the page on text corpora for AI training describes.
  • Not license at all and leave the pages as they are.

The benefits are real but narrow, and the risks depend on court outcomes that have not arrived. Nothing in the dataset's terms creates a claim to payment for an owner whose pages are included.

Not licensing is a valid outcome. The decision to reserve rights, to send a notice, to license or to do nothing stays with the owner.

Crawler Controls, Dataset Terms and Removal Beyond This Page

This page covers FineWeb and its two companion releases; crawler rules, removal routes, other corpora and the wider law each have their own page.

Frequently Asked Questions About FineWeb

What is FineWeb?

FineWeb is a set of English web text filtered from Common Crawl's archive by Hugging Face, Inc. The 2024 paper describes it as 15 trillion tokens from 96 crawl snapshots. It is released under ODC-By 1.0 and is also subject to the crawl's Terms of Use. FineWeb-Edu and FineWeb2 are separate releases with their own sizes.

How big is FineWeb?

The paper of 2024-06-25 gives 15 trillion tokens from 96 snapshots, and the card's dump table lists 110 dumps up to CC-MAIN-2025-26. The two figures are different dates of one growing set. This page states no current total, because the card's own summary and table differ, and it does not add or reconcile them.

What license does FineWeb use?

FineWeb is released under the Open Data Commons Attribution License 1.0, and use is also subject to Common Crawl's Terms of Use. The license covers the database, not the pages inside it, which stay under their own rights.

Is my website in FineWeb, and can it be removed?

Each record keeps its source URL, so a site owner can check a page. The card offers a removal form for personal data only, and Torstar used a cease-and-desist letter naming 361 domains. Blocking the crawler affects only future snapshots. This page is information, not legal advice.

Agents and advisers can use the introducer form.