DataPacts

DataComp, CommonPool and DCLM: What They Contain, License, Legal Status

Checked

At a glance

DataComp is a benchmark project, started in 2023 and extended to language models in 2024, whose two releases are CommonPool, a list of 12.8 billion image-text pairs from Common Crawl, and DCLM-Pool, a 240 trillion token text corpus from the same crawl.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

The project also publishes two baselines drawn from them, DataComp-1B and DCLM-Baseline. CommonPool is an index of links and captions; DCLM-Pool holds the extracted text of documents.

This page reads both from the side of the people whose images and writing they index: inclusion is unpaid and unlicensed. Within AI training data licensing, these are web-sourced sets that researchers filter, with license terms that cover the list and not the content.

Who Built DataComp, CommonPool and DCLM: Two Papers, One Project and No Named Legal Entity

DataComp and DataComp-LM were built by a research collaboration of universities and companies and are described in two arXiv papers, of April 27, 2023 and June 17, 2024. The table lists the identity facts as of 2026-11-17.

Attribute Value Source
Built by the DataComp project (publishing as mlfoundations); legal entity not stated datacomp.ai
Image paper "DataComp: In search of the next generation of multimodal datasets", arXiv 2304.14108, v1 2023-04-27, v5 2023-10-20, 34 authors arxiv.org/abs/2304.14108
Language paper "DataComp-LM: In search of the next generation of training sets for language models", arXiv 2406.11794, v1 2024-06-17, v4 2025-04-21, 59 authors arxiv.org/abs/2406.11794
Releases CommonPool and DataComp-1B (image-text), DCLM-Pool and DCLM-Baseline (text) the two papers
Host Hugging Face, organization mlfoundations; the three cards read are not gated huggingface.co
Contact contact@datacomp.ai as printed in the papers arXiv 2304.14108; 2406.11794

The first paper lists affiliations including the University of Washington, Columbia, Tel Aviv University, Apple, UT Austin, LAION, AI2 and Juelich. The datacomp.ai site names no legal entity, so this page writes "the DataComp project". Hugging Face hosts the files and did not build them.

The paper's datasheet says takedown requests are handled together with LAION. The language paper lists 59 authors, a larger team than the first paper's 34, and both papers print the same contact address, so the two releases share one point of contact.

How CommonPool's 12.8 Billion Image-Text Pairs Were Drawn From Common Crawl

CommonPool starts from every Common Crawl metadata file from 2014 through November 2022 and keeps a random 12.8 billion of the image-text pairs that survived retrieval and screening. The table gives the paper's counts as of 2026-11-17, as printed.

Stage Count Source
URL-text pairs parsed from Common Crawl WAT files, 2014 to November 2022, after uid-hash deduplication 88 billion arXiv 2304.14108, Appendix D
Fetch attempted 40 billion samples Section 3.2
Downloaded with img2dataset 16.8 billion Section 3.2
After NSFW removal (about 19 % flagged) 13.6 billion Section 3.2
After evaluation-set deduplication (about 3 % flagged) 13.1 billion Section 3.2
Random subset kept as the xlarge pool 12.8 billion Section 3.2
Smaller scales 12.8 million, 128 million, 1.28 billion Table 2

The caption of each pair is the alt-text of its image. Deduplication by uid hash removes repeated URL-text pairs but keeps identical images that sit at different URLs.

The four scales are 12.8 million, 128 million, 1.28 billion and 12.8 billion samples, each a random subset of the next larger one. All of it starts from the archive described on the page about what Common Crawl contains.

What the CommonPool and DataComp-1B Files Hold: Links, Captions and Metadata, Not Images

The CommonPool files list image URLs with captions and per-sample metadata, and they hold no images. The table compares the four releases as of 2026-11-17; all results are claims of the authors.

Release What it is Size or result stated Source
CommonPool URL-text index plus metadata 12.8 billion pairs (paper); Hub card total file size 91.1 TB arXiv 2304.14108; Hub card
DataComp-1B best baseline subset of CommonPool 1.4 billion samples; CLIP ViT-L/14 79.2 % zero-shot ImageNet, 3.7 points above OpenAI's ViT-L/14 arXiv 2304.14108
DCLM-Pool text documents extracted from Common Crawl 200 billion documents, 370 TB gzip, 240 trillion tokens arXiv 2406.11794
DCLM-Baseline filtered subset 4 trillion tokens, 3 billion documents, 7.2 TB Hub card

The metadata fields include the URL, the caption, image dimensions, CLIP scores, face bounding boxes and a sha256 hash. The datasheet says raw data is "not available or distributed due to safety considerations", and only URLs not flagged as NSFW are listed.

The images load from their original sites, so an image removed by its host drops out of a later retrieval.

The Hub card for DataComp-1B repeats the CommonPool license wording, so the baseline subset carries the same split between the index and the images it lists.

The paper calls DataComp-1B currently the largest public image-text dataset, which is the authors' own description. LAION-5B is also a link set built this way, covered under LAION's link set.

DCLM-Pool and DCLM-Baseline: Web Text Extracted From Common Crawl

DCLM-Pool is an unfiltered text corpus of all Common Crawl data before 2023, re-extracted from HTML with resiliparse, and DCLM-Baseline is the roughly 4 trillion token subset the project built from it. The paper and the card state:

  • DCLM-Pool: 200 billion documents, 370 TB gzip, 240 trillion GPT-NeoX tokens.
  • DCLM-Baseline pipeline: RefinedWeb heuristic filters, Bloom-filter deduplication, then a fastText classifier trained on instruction-style data (OpenHermes 2.5 and r/ExplainLikeImFive posts) that keeps the highest-scoring 10 % of documents.
  • DCLM-Baseline card: 4T tokens, 3B documents, an estimated 3,017,780,768 rows, 7.2 TB, last modified 2024-07-22.
  • Redactions: the pool maps one-to-one to Common Crawl WARC files, so it can follow Common Crawl's redactions.

The results are the authors' claims: a 7B model reaching 64 % on 5-shot MMLU at 2.6 trillion training tokens. The project did not decontaminate the pool against benchmarks and released decontamination tooling instead.

Unlike CommonPool, DCLM-Pool holds the text itself, so a page that was crawled before 2023 can appear in it as a full extracted copy, and the paper says the pool was collected without filtering, with no special treatment for private content. The paper compares its baseline with other crawl-derived sets, among them RedPajama.

License Terms Across the DataComp Releases: CC BY 4.0 on the Index, Own Rights on the Content

The DataComp releases license the index and the metadata under CC BY 4.0, and say the content they point to stays under its own rights. The table shows the stated terms as of 2026-11-17.

Release Stated license or terms Stated limits Source
CommonPool index CC BY 4.0 (url-text samples and metadata) images "under their own copyrights"; "as is", a research artifact; use at the user's risk Hub card; paper datasheet Q48
DataComp-1B same wording same Hub card
DCLM-Pool competition subsets CC BY 4.0 also subject to Common Crawl's Terms of Use; robots.txt respected arXiv 2406.11794, distribution section
DCLM-Baseline cc-by-4.0 tag card: "intended for research use only"; not for production-ready models Hub card and API

For DCLM-Baseline the research-only language sits in the card's intended-use text beside the license tag. Both are shown here, and this page does not say which one controls. CC BY 4.0 on the index covers the list and the metadata, not any image or page it points to.

The paper's datasheet adds that the authors do not copyright the samples. Common Crawl's terms of use are a second layer for the DCLM text, and the paper says the pool respects robots.txt.

Neither the CC BY 4.0 notice nor the research-only wording is a license from the people whose pages were crawled. How CC BY 4.0 and other terms compare is set out in the page on dataset licenses for AI training.

Faces, NSFW Content and Personal Data in CommonPool: What Was Filtered and What an Audit Found

CommonPool's builders filtered NSFW content and detected faces for blurring, but the blurring runs in the image-fetching tooling, and a 2025 audit estimates that at least 100 million face images stay unblurred with default parameters. The paper describes three steps:

  • A Detoxify text screen at a 0.1 threshold and a classifier on CLIP image features; about 19 % of pairs were flagged.
  • The SCRFD-10G face detector, with 90.53 % recall on 3,293 sampled images; bounding boxes are stored as metadata.
  • Blurring: "images are by default blurred" when the tooling fetches them (Appendix G).

The datasheet says no raw images are distributed, and the Hub card does not mention blurring. The builders add that CommonPool "is still likely to contain some harmful data" and is "not suitable for any software that makes decisions involving people".

The audit "A Common Pool of Privacy Problems" (6 authors, arXiv 2506.17185, v1 2025-06-20, v2 2026-04-06) reports significant personal information despite sanitization.

It estimates at least 136,000 resume images among the 12.8 billion pairs, a lower bound from a random subset, and found credit cards, passports and similar documents, with counts for the audited sample only.

These are the authors' findings. For DCLM-Pool, the paper says the text "will contain some PII" and was not filtered for it. The seller's side of the same problem is covered under PII removal and de-identification before licensing.

Takedown and Opt-out Routes the DataComp Project Describes

The project describes two takedown routes, one per release, and neither Hub card describes a removal form.

For CommonPool, the datasheet names laion.ai/dataset-requests, a form keyed by sample ID, and says the project will try to tell a requester the URL where the content is hosted so the requester can ask the host to remove it. The form page was live on 2026-11-17.

For DCLM-Pool, the datasheet says "users can contact us through contact@datacomp.ai" and "specify the offending URL", "until we establish an automated method". The datasheet adds that Common Crawl honors deletion requests, that the pool follows its redactions and that the pool respects robots.txt.

The audit notes that since 2023 hosts have changed robots.txt to restrict scraping, while CommonPool's snapshots span 2014 to 2022. No count of requests, removals or responses is published for either release. Routes across datasets are compared on the page about removing content from AI training datasets.

One court record on file names a DataComp release: the Third Amended Complaint of February 27, 2026 in Andersen v. Stability AI (Dkt. 431) pleads a count over DataComp xLarge against Midjourney. That is an allegation, and this page states no outcome.

No other case, deal, regulator action or EU training-content summary on file names DataComp, CommonPool or DCLM, as of 2026-11-17. The case page covers the claims and their status: Andersen v. Stability AI.

The audit's legal analysis under GDPR, CCPA and Oregon law is the view of its authors, not a ruling. The builders' own datasheet calls CommonPool a research artifact and says it is not intended for production products, which is a statement of intended use and not a legal finding.

A Work Indexed in CommonPool or DCLM: Benefits, Risks, Downsides and Alternatives for the Rights Holder

For a photographer, blogger or publisher whose page sits in Common Crawl before 2023, inclusion in CommonPool or DCLM follows from the crawl, not from any contract, and brings no payment.

Benefits

  • The index is a list of links, so the image stays on the original site.
  • The datasheets and a takedown address are public.

Risks

  • A list that points to the work can be used by anyone to fetch the image from its host.
  • DCLM text is a full copy of the extracted page.
  • No outcome record exists for removal requests.

Downsides

  • Inclusion brings no payment and involved no consent step.
  • Blurring is a tooling option, not guaranteed in every retrieval.

Alternatives

  • A robots.txt rule or machine-readable reservation for future crawls.
  • A removal request through the routes above.
  • Licensing images or text directly, as covered under selling photos and images for AI training.
  • Not licensing at all.

Not licensing is a valid outcome, and nothing here suggests that inclusion creates a claim to payment.

Other Web Corpora, License Families and Removal Routes Beyond This Page

This page covers DataComp's releases; other web corpora, license families, opt-out settings and the law each have their own page.

Frequently Asked Questions About DataComp

What is DataComp?

DataComp is a benchmark project with a 2023 image-text track (CommonPool) and a 2024 language track (DCLM). Its papers are dated 2023-04-27 and 2024-06-17.

What is CommonPool?

CommonPool is a list of 12.8 billion image URLs with captions drawn from Common Crawl. It is filtered for NSFW content, and no images are distributed.

What license does DataComp use?

DataComp uses CC BY 4.0 on the index and metadata, while the images stay under their own copyrights. DCLM-Baseline is tagged cc-by-4.0 and described as intended for research use only.

Are faces blurred in CommonPool?

The paper says the image-fetching tooling blurs detected faces by default, and the Hub card does not say so. An audit estimates that at least 100 million face images stay unblurred with default parameters. This page is information, not legal advice.