The project also publishes two baselines drawn from them, DataComp-1B and DCLM-Baseline. CommonPool is an index of links and captions; DCLM-Pool holds the extracted text of documents.
This page reads both from the side of the people whose images and writing they index: inclusion is unpaid and unlicensed. Within AI training data licensing, these are web-sourced sets that researchers filter, with license terms that cover the list and not the content.
Who Built DataComp, CommonPool and DCLM: Two Papers, One Project and No Named Legal Entity
DataComp and DataComp-LM were built by a research collaboration of universities and companies and are described in two arXiv papers, of April 27, 2023 and June 17, 2024. The table lists the identity facts as of 2026-11-17.
| Attribute | Value | Source |
|---|---|---|
| Built by | the DataComp project (publishing as mlfoundations); legal entity not stated | datacomp.ai |
| Image paper | "DataComp: In search of the next generation of multimodal datasets", arXiv 2304.14108, v1 2023-04-27, v5 2023-10-20, 34 authors | arxiv.org/abs/2304.14108 |
| Language paper | "DataComp-LM: In search of the next generation of training sets for language models", arXiv 2406.11794, v1 2024-06-17, v4 2025-04-21, 59 authors | arxiv.org/abs/2406.11794 |
| Releases | CommonPool and DataComp-1B (image-text), DCLM-Pool and DCLM-Baseline (text) | the two papers |
| Host | Hugging Face, organization mlfoundations; the three cards read are not gated |
huggingface.co |
| Contact | contact@datacomp.ai as printed in the papers | arXiv 2304.14108; 2406.11794 |
The first paper lists affiliations including the University of Washington, Columbia, Tel Aviv University, Apple, UT Austin, LAION, AI2 and Juelich. The datacomp.ai site names no legal entity, so this page writes "the DataComp project". Hugging Face hosts the files and did not build them.
The paper's datasheet says takedown requests are handled together with LAION. The language paper lists 59 authors, a larger team than the first paper's 34, and both papers print the same contact address, so the two releases share one point of contact.
How CommonPool's 12.8 Billion Image-Text Pairs Were Drawn From Common Crawl
CommonPool starts from every Common Crawl metadata file from 2014 through November 2022 and keeps a random 12.8 billion of the image-text pairs that survived retrieval and screening. The table gives the paper's counts as of 2026-11-17, as printed.
| Stage | Count | Source |
|---|---|---|
| URL-text pairs parsed from Common Crawl WAT files, 2014 to November 2022, after uid-hash deduplication | 88 billion | arXiv 2304.14108, Appendix D |
| Fetch attempted | 40 billion samples | Section 3.2 |
| Downloaded with img2dataset | 16.8 billion | Section 3.2 |
| After NSFW removal (about 19 % flagged) | 13.6 billion | Section 3.2 |
| After evaluation-set deduplication (about 3 % flagged) | 13.1 billion | Section 3.2 |
| Random subset kept as the xlarge pool | 12.8 billion | Section 3.2 |
| Smaller scales | 12.8 million, 128 million, 1.28 billion | Table 2 |
The caption of each pair is the alt-text of its image. Deduplication by uid hash removes repeated URL-text pairs but keeps identical images that sit at different URLs.
The four scales are 12.8 million, 128 million, 1.28 billion and 12.8 billion samples, each a random subset of the next larger one. All of it starts from the archive described on the page about what Common Crawl contains.
What the CommonPool and DataComp-1B Files Hold: Links, Captions and Metadata, Not Images
The CommonPool files list image URLs with captions and per-sample metadata, and they hold no images. The table compares the four releases as of 2026-11-17; all results are claims of the authors.
| Release | What it is | Size or result stated | Source |
|---|---|---|---|
| CommonPool | URL-text index plus metadata | 12.8 billion pairs (paper); Hub card total file size 91.1 TB | arXiv 2304.14108; Hub card |
| DataComp-1B | best baseline subset of CommonPool | 1.4 billion samples; CLIP ViT-L/14 79.2 % zero-shot ImageNet, 3.7 points above OpenAI's ViT-L/14 | arXiv 2304.14108 |
| DCLM-Pool | text documents extracted from Common Crawl | 200 billion documents, 370 TB gzip, 240 trillion tokens | arXiv 2406.11794 |
| DCLM-Baseline | filtered subset | 4 trillion tokens, 3 billion documents, 7.2 TB | Hub card |
The metadata fields include the URL, the caption, image dimensions, CLIP scores, face bounding boxes and a sha256 hash. The datasheet says raw data is "not available or distributed due to safety considerations", and only URLs not flagged as NSFW are listed.
The images load from their original sites, so an image removed by its host drops out of a later retrieval.
The Hub card for DataComp-1B repeats the CommonPool license wording, so the baseline subset carries the same split between the index and the images it lists.
The paper calls DataComp-1B currently the largest public image-text dataset, which is the authors' own description. LAION-5B is also a link set built this way, covered under LAION's link set.
DCLM-Pool and DCLM-Baseline: Web Text Extracted From Common Crawl
DCLM-Pool is an unfiltered text corpus of all Common Crawl data before 2023, re-extracted from HTML with resiliparse, and DCLM-Baseline is the roughly 4 trillion token subset the project built from it. The paper and the card state:
- DCLM-Pool: 200 billion documents, 370 TB gzip, 240 trillion GPT-NeoX tokens.
- DCLM-Baseline pipeline: RefinedWeb heuristic filters, Bloom-filter deduplication, then a fastText classifier trained on instruction-style data (OpenHermes 2.5 and r/ExplainLikeImFive posts) that keeps the highest-scoring 10 % of documents.
- DCLM-Baseline card: 4T tokens, 3B documents, an estimated 3,017,780,768 rows, 7.2 TB, last modified 2024-07-22.
- Redactions: the pool maps one-to-one to Common Crawl WARC files, so it can follow Common Crawl's redactions.
The results are the authors' claims: a 7B model reaching 64 % on 5-shot MMLU at 2.6 trillion training tokens. The project did not decontaminate the pool against benchmarks and released decontamination tooling instead.
Unlike CommonPool, DCLM-Pool holds the text itself, so a page that was crawled before 2023 can appear in it as a full extracted copy, and the paper says the pool was collected without filtering, with no special treatment for private content. The paper compares its baseline with other crawl-derived sets, among them RedPajama.
License Terms Across the DataComp Releases: CC BY 4.0 on the Index, Own Rights on the Content
The DataComp releases license the index and the metadata under CC BY 4.0, and say the content they point to stays under its own rights. The table shows the stated terms as of 2026-11-17.
| Release | Stated license or terms | Stated limits | Source |
|---|---|---|---|
| CommonPool index | CC BY 4.0 (url-text samples and metadata) | images "under their own copyrights"; "as is", a research artifact; use at the user's risk | Hub card; paper datasheet Q48 |
| DataComp-1B | same wording | same | Hub card |
| DCLM-Pool | competition subsets CC BY 4.0 | also subject to Common Crawl's Terms of Use; robots.txt respected | arXiv 2406.11794, distribution section |
| DCLM-Baseline | cc-by-4.0 tag | card: "intended for research use only"; not for production-ready models | Hub card and API |
For DCLM-Baseline the research-only language sits in the card's intended-use text beside the license tag. Both are shown here, and this page does not say which one controls. CC BY 4.0 on the index covers the list and the metadata, not any image or page it points to.
The paper's datasheet adds that the authors do not copyright the samples. Common Crawl's terms of use are a second layer for the DCLM text, and the paper says the pool respects robots.txt.
Neither the CC BY 4.0 notice nor the research-only wording is a license from the people whose pages were crawled. How CC BY 4.0 and other terms compare is set out in the page on dataset licenses for AI training.
Faces, NSFW Content and Personal Data in CommonPool: What Was Filtered and What an Audit Found
CommonPool's builders filtered NSFW content and detected faces for blurring, but the blurring runs in the image-fetching tooling, and a 2025 audit estimates that at least 100 million face images stay unblurred with default parameters. The paper describes three steps:
- A Detoxify text screen at a 0.1 threshold and a classifier on CLIP image features; about 19 % of pairs were flagged.
- The SCRFD-10G face detector, with 90.53 % recall on 3,293 sampled images; bounding boxes are stored as metadata.
- Blurring: "images are by default blurred" when the tooling fetches them (Appendix G).
The datasheet says no raw images are distributed, and the Hub card does not mention blurring. The builders add that CommonPool "is still likely to contain some harmful data" and is "not suitable for any software that makes decisions involving people".
The audit "A Common Pool of Privacy Problems" (6 authors, arXiv 2506.17185, v1 2025-06-20, v2 2026-04-06) reports significant personal information despite sanitization.
It estimates at least 136,000 resume images among the 12.8 billion pairs, a lower bound from a random subset, and found credit cards, passports and similar documents, with counts for the audited sample only.
These are the authors' findings. For DCLM-Pool, the paper says the text "will contain some PII" and was not filtered for it. The seller's side of the same problem is covered under PII removal and de-identification before licensing.
Takedown and Opt-out Routes the DataComp Project Describes
The project describes two takedown routes, one per release, and neither Hub card describes a removal form.
For CommonPool, the datasheet names laion.ai/dataset-requests, a form keyed by sample ID, and says the project will try to tell a requester the URL where the content is hosted so the requester can ask the host to remove it. The form page was live on 2026-11-17.
For DCLM-Pool, the datasheet says "users can contact us through contact@datacomp.ai" and "specify the offending URL", "until we establish an automated method". The datasheet adds that Common Crawl honors deletion requests, that the pool follows its redactions and that the pool respects robots.txt.
The audit notes that since 2023 hosts have changed robots.txt to restrict scraping, while CommonPool's snapshots span 2014 to 2022. No count of requests, removals or responses is published for either release. Routes across datasets are compared on the page about removing content from AI training datasets.
Legal Standing of the DataComp Releases: One Court Record and No Notice on File
One court record on file names a DataComp release: the Third Amended Complaint of February 27, 2026 in Andersen v. Stability AI (Dkt. 431) pleads a count over DataComp xLarge against Midjourney. That is an allegation, and this page states no outcome.
No other case, deal, regulator action or EU training-content summary on file names DataComp, CommonPool or DCLM, as of 2026-11-17. The case page covers the claims and their status: Andersen v. Stability AI.
The audit's legal analysis under GDPR, CCPA and Oregon law is the view of its authors, not a ruling. The builders' own datasheet calls CommonPool a research artifact and says it is not intended for production products, which is a statement of intended use and not a legal finding.
A Work Indexed in CommonPool or DCLM: Benefits, Risks, Downsides and Alternatives for the Rights Holder
For a photographer, blogger or publisher whose page sits in Common Crawl before 2023, inclusion in CommonPool or DCLM follows from the crawl, not from any contract, and brings no payment.
Benefits
- The index is a list of links, so the image stays on the original site.
- The datasheets and a takedown address are public.
Risks
- A list that points to the work can be used by anyone to fetch the image from its host.
- DCLM text is a full copy of the extracted page.
- No outcome record exists for removal requests.
Downsides
- Inclusion brings no payment and involved no consent step.
- Blurring is a tooling option, not guaranteed in every retrieval.
Alternatives
- A robots.txt rule or machine-readable reservation for future crawls.
- A removal request through the routes above.
- Licensing images or text directly, as covered under selling photos and images for AI training.
- Not licensing at all.
Not licensing is a valid outcome, and nothing here suggests that inclusion creates a claim to payment.
Other Web Corpora, License Families and Removal Routes Beyond This Page
This page covers DataComp's releases; other web corpora, license families, opt-out settings and the law each have their own page.
- Corpora: major AI training datasets, C4, FineWeb, Dolma, Common Pile and CCBot.
- Opt-out and records: opting out of AI training and dataset documentation.
- Law and cases: GDPR and AI training data, copyright and AI training across jurisdictions and AI copyright lawsuits over training data.
- Data and options: text corpora for AI training and alternatives to selling data, including not selling at all.
Frequently Asked Questions About DataComp
What is DataComp?
DataComp is a benchmark project with a 2023 image-text track (CommonPool) and a 2024 language track (DCLM). Its papers are dated 2023-04-27 and 2024-06-17.
What is CommonPool?
CommonPool is a list of 12.8 billion image URLs with captions drawn from Common Crawl. It is filtered for NSFW content, and no images are distributed.
What license does DataComp use?
DataComp uses CC BY 4.0 on the index and metadata, while the images stay under their own copyrights. DCLM-Baseline is tagged cc-by-4.0 and described as intended for research use only.
Are faces blurred in CommonPool?
The paper says the image-fetching tooling blurs detected faces by default, and the Hub card does not say so. An audit estimates that at least 100 million face images stay unblurred with default parameters. This page is information, not legal advice.