DataPacts

Dolma

Checked

At a glance

Dolma is an openly released English text corpus from the Allen Institute for AI (Ai2), first published on 2023-08-18 with about 3 trillion tokens from web pages, papers, code, books, Reddit and encyclopedic text, and used to train Ai2's OLMo models.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

It grew from v1 to v1.7 (2024-04-15) and then into the Dolma 3 pool, the Dolma 3 Mix and the Dolma 3.5 pool, whose size Ai2 states as nearly 10 trillion tokens.

Ai2's cards carry the Open Data Commons Attribution License (ODC-BY), which replaced Ai2's ImpACT license on 2024-04-15, and Ai2 offers one removal route, a personal data form.

This page reads Dolma from the side of the people whose text it holds. Within AI training data licensing, Dolma is free crawled text, the unpaid counterpart to licensed text: it brings its source authors no payment and no contract.

Who Publishes Dolma: Ai2, the Paper and the Hosted Releases

Dolma is published by The Allen Institute for Artificial Intelligence, a Seattle-based nonprofit research institute that calls itself Ai2, and is described in the paper "Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research". The table lists the identity facts as of 2026-11-17.

Attribute Value Source
Publisher The Allen Institute for Artificial Intelligence (Ai2), "a Seattle based non-profit AI research institute"; a 501(c)(3) nonprofit per its Responsible Use Guidelines allenai.org/terms; allenai.org/responsible-use
Paper "Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research", Soldaini et al., arXiv 2402.00159, v1 2024-01-31, v2 2024-06-06, accepted at ACL 2024 arxiv.org/abs/2402.00159
First release v1, 2023-08-18 huggingface.co/datasets/allenai/dolma; allenai.org blog
Hosted copies allenai/dolma (repository created 2023-06-30, not gated, last modified 2024-04-17); dolma3_pool, dolma3_mix-6T, dolma3.5_pool huggingface.co
Host Hugging Face, Inc.; a further copy on Ai2's cloud storage per the datasheet huggingface.co/terms-of-service; arxiv.org
Models trained OLMo-1B (v1.5), OLMo-7B (v1.5 sample), OLMo-7B-v1.7 (v1.7), Olmo-3-1125-32B (Dolma 3 Mix), as the cards state dataset cards
Registry number not checked n/a

The paper was first posted on 2024-01-31 and its second version carries the note "Accepted at ACL 2024". Its datasheet names Ai2 as the maintainer, directs support requests to GitHub issues and other inquiries to ai2-info@allenai.org.

Hugging Face, Inc. hosts the files and did not create the corpus. The Responsible Use Guidelines describe Ai2's tools as "intended for research and educational purposes". Dolma is one entry among the major AI training datasets compared on the parent page.

Dolma's Releases From v1 to Dolma 3.5

Ai2 has published Dolma in six versions on its original card, from v1 on 2023-08-18 to v1.7 on 2024-04-15, and later as the Dolma 3 pool, the Dolma 3 Mix and the Dolma 3.5 pool.

The table orders the releases by date and keeps each size in the unit and wording of Ai2's card, as of 2026-11-17.

Release Released or created Size, as Ai2 states it Notes from the card Source
v1 2023-08-18 6.0 TB gzip "The first version of Dolma" allenai/dolma card
v1.5 2023-10-31 6.4 TB gzip; roughly 3 trillion tokens used to train OLMo-1B; a 2.9 TB sample of roughly 1.9 trillion tokens trained OLMo-7B allenai/dolma card
v1.6 2024-01-31 5.4 TB gzip; 3,059 billion Llama tokens more deduplication; a 16.4 GB sample of roughly 10 billion tokens allenai/dolma card
v1.7 (default) 2024-04-15 (announced 2024-04-17) 4.5 TB gzip; 2,308.5 billion OLMo tokens used to train OLMo-7B-v1.7; new sources, quality filtering, fuzzy deduplication allenai/dolma card
Dolma 3 pool repository created 2025-10-16 over 9 trillion tokens (table total 9.31 trillion, 9.97 billion documents) pool before quality upsampling and mixing allenai/dolma3_pool card
Dolma 3 Mix (6T) repository created 2025-11-24 about 6 trillion tokens pretraining data of Olmo-3-1125-32B; majority from Common Crawl allenai/dolma3_mix-6T card
Dolma 3.5 pool repository created 2026-03-25 nearly 10 trillion tokens more PDFs, code pool from 137 billion to over 1,351 billion tokens allenai/dolma3.5_pool card

The units differ by release: the original card counts gzip terabytes and, for some versions, tokens, while the later cards count tokens, and the original card itself uses OLMo tokens for v1.7 and Llama tokens for v1.6. The two tokenizers are never compared or added here, and neither figure is called wrong.

The v1.7 release is the card's default, and the card's version table dates it 2024-04-15 while its announcement line reads 2024-04-17; both are shown. The card attaches the "roughly 3 trillion tokens" of the paper's title and Ai2's 2023 post to v1.5.

The Dolma 3 pool is the stage before "upsampling and mixing", and the Dolma 3 Mix, at about 6 trillion tokens, is the data Ai2 states it used in pretraining.

What Dolma Draws From: Web Crawls, Papers, Code, Forums and Books

Most of Dolma comes from Common Crawl's web archive, and the rest from academic papers, code repositories, Reddit, Wikipedia and public domain books, each listed with its own cutoff date on Ai2's card.

For v1.7, the largest sources by OLMo tokens are Dolma's own Common Crawl subset at 1,195.5 billion, RefinedWeb at 456.4 billion, StarCoder at 263.8 billion, C4 at 138.4 billion and Reddit, collected through the PushShift API, at 79.9 billion.

The card gives a total of 2,308.5 billion tokens in 2,532.0 million documents and says 1.715 trillion tokens were used to train OLMo-7B-v1.7 after sampling.

The C4 portion carries an April 2019 cutoff, which ties it to the snapshot described on the page about C4 (Colossal Clean Crawled Corpus).

The paper's web subset used 25 Common Crawl snapshots from 2020-05 to 2023-06. Dolma 3 starts from 104 Common Crawl dumps from 2013 to 2024 with a cutoff of 2024-12-31, according to Ai2's Dolma 3 documentation.

Release Source Type Tokens, as Ai2 states Cutoff Source
v1.7 Dolma's CC (Common Crawl) web pages 1,195.5 billion March 2023 allenai/dolma card
v1.7 RefinedWeb web pages 456.4 billion February 2023 allenai/dolma card
v1.7 StarCoder code 263.8 billion May 2023 allenai/dolma card
v1.7 C4 web pages 138.4 billion April 2019 allenai/dolma card
v1.7 Reddit (PushShift API) social media 79.9 billion March 2023 allenai/dolma card
v1.7 Semantic Scholar (peS2o) papers 57.2 billion March 2023 allenai/dolma card
v1.7 arXiv; StackExchange; Flan; CC News; OpenWebMath; Algebraic Stack; Project Gutenberg; MegaWika; Wikipedia and Wikibooks papers, Q&A, instructions, news, math, books, encyclopedic 28.0; 19.6; 16.5; 14.3; 12.6; 12.6; 5.3; 4.6; 3.7 billion February to October 2023 allenai/dolma card
Dolma 3 pool Common Crawl web pages 8.14 trillion (9.67 billion documents) 2024-12-31 allenai/dolma3_pool card
Dolma 3 pool olmOCR Science PDFs academic documents 972 billion n/a allenai/dolma3_pool card
Dolma 3 pool StackEdu; arXiv; FineMath 3+; Wikipedia and Wikibooks code, papers, math pages, encyclopedic 137; 21.4; 34.1; 3.69 billion n/a allenai/dolma3_pool card

Some text is collected by Ai2's own pipeline. The Dolma 3 PDF pool comes from 238,475,382 PDFs crawled by the Semantic Scholar Anansi pipeline, which the documentation lists as a "Disclosed agent" with "No paywall circumventing", and the Dolma 3.5 card moves the cutoff for crawled PDFs to November 2025.

Separately, Ai2's crawler notice says "The AI2 Bot explores certain domains to find web content.

This web content is used to train open language models." and gives the user agent "Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)". The notice names no dataset, so this page does not tie that crawler to any Dolma release.

Forum text such as Reddit's is covered on the page about forum and community archives as AI training data and papers such as peS2o on the page about scientific data and papers as AI training data.

The upstream archive has its own page. For the crawl's contents and terms, see what Common Crawl contains. The Dolma 3 pool's Common Crawl figure of 8.14 trillion tokens appears on this page only in the table above.

What Dolma's Filters Remove: Personal Data, Toxic Text and Deny-Listed Documents

Ai2 filters Dolma for personal data, toxic content and duplicates, and for the Dolma 3 PDF pool it also removes documents on legal deny lists, according to the paper and the Dolma 3 documentation. The filters are listed as Ai2 describes them.

  • Personal data: regular expressions find emails, IP addresses and phone numbers; documents with 5 or fewer such spans have them masked (0.02 % of documents), and denser documents are removed (0.001 %).
  • Toxic text: FastText classifiers for hate speech and NSFW content.
  • Reddit: 26,123 banned or NSFW subreddits are excluded.
  • Dolma 3 PDFs: PII filtering affects 4.9 %, and legal deny lists remove 0.3 % for the Olmo 3 7B data and a further 6.6 % for the 32B model.
  • Sources left out: the paper says it avoids sources such as Books3, "the center of ongoing legal cases".

The paper says its regular expressions "sacrifice some accuracy", and Ai2's 2024 post says its removal of personal and hateful content "is not perfect". The contents of the deny lists are not published in the documents read. The excluded Books3 corpus has its own entry in the page on the Books3 dataset.

Dolma's License History: From Ai2's ImpACT License to ODC-BY

Dolma was first released on 2023-08-18 under Ai2's ImpACT license as a medium-risk artifact, and Ai2 moved it to the Open Data Commons Attribution License (ODC-BY) on 2024-04-15. Ai2's 2023 post states four conditions of the ImpACT period: users give contact information and intended use, disclose their derivatives, distribute derivatives under the same restrictions, and avoid prohibited uses "such as military surveillance or generating disinformation".

The ImpACT license text itself was not reachable on 2026-11-17, so this page repeats only what Ai2's posts say.

The 2024 post gives the reason that "creative uses of Dolma require a more flexible license".

It says people who downloaded Dolma under ImpACT do not have to redownload, that derivatives may keep ImpACT or use another license, and that users must attach a notice that the work was obtained from Dolma. The table sets out the layers as of 2026-11-17.

Period or layer Terms What Ai2 states Source
2023-08-18 to 2024-04-14 Ai2 ImpACT license, medium-risk artifact contact details and intended use; disclose derivatives; derivatives under the same restrictions; no prohibited uses such as military surveillance or disinformation allenai.org blog, 2023-08-18
From 2024-04-15 ODC-BY 1.0 earlier downloads need no redownload; derivatives may keep ImpACT or another license; attribution notice that the work was obtained from Dolma allenai.org blog, 2024-04-15; card
Contents inside the database not licensed by ODC-BY (s. 2.4); "also bound" by the original sources' license agreements and terms of use original sources card; opendatacommons.org
Upstream web archive Common Crawl Terms of Use, last updated 2024-03-07 see the Common Crawl page commoncrawl.org/terms-of-use
Dolma 3, Dolma 3 Mix, Dolma 3.5 ODC-By v1.0, "intended for research and educational use", Responsible Use Guidelines dataset cards dataset cards; allenai.org/responsible-use

The original card states: "We are releasing this dataset under the terms of ODC-BY. By using this dataset, you are also bound any license agreements and terms of use of the original data sources." ODC-BY 1.0 governs rights over the database, not its individual contents (section 2.4), and the repository's license file says the same.

The Dolma 3 and 3.5 cards add that the data is "intended for research and educational use", and this page quotes the phrase without interpreting it. How ODC, CC and CDLA terms differ is compared on the page about dataset licenses for AI training.

Ai2 did not ask the people whose text is in Dolma for consent, and it offers one route out: a Google Form titled "Dolma Personal Data Removal Request". The paper's datasheet says the authors of the texts "were not directly informed" and that "no consent was collected from individuals".

It describes Reddit data as "acquired through a public API adherent to terms of service", with its authors "not contacted directly", and says Ai2 "did not contact Project Gutenberg".

The form asks for the requester, on whose behalf the request is made, the data subject, file names, the Dolma version and document IDs, followed by sworn statements and a signature.

It says Ai2 must "balance privacy and free expression" and that Ai2 "does not guarantee that all information about you will be removed from OLMo outputs".

The dataset card and the paper's footnote link to the same form. How datasheets such as this one are meant to work is explained on the page about dataset documentation, datasheets and data cards.

The form concerns personal data only. The original, Dolma 3 and Dolma 3.5 cards describe no route for copyright holders or site owners, and the Dolma 3 and 3.5 cards do not link the form.

No response time or count of removals is stated in the documents read. Removal routes across datasets are collected on the page about removing content from AI training datasets.

No court ruling or pending case on file names Dolma or the OLMo models, as of 2026-11-17, and no licensing deal on file covers Dolma. The search covered the court records and deals tracked on this site, and it does not reach every proceeding or notice anywhere.

The Dolma paper has its own copyright paragraph, and its statements are the authors' own. They write that the legal landscape is "largely undetermined", that "large web crawls may also contain copyrighted material" and that it is "not possible to reliably or scalably detect copyrighted materials".

They describe their sources as "publicly available and already being used" and say they "engaged with legal and ethics experts". This page draws no conclusion on whether the corpus is lawful.

The Data Provenance Initiative (Longpre et al., arXiv 2407.14933) audited consent signals, robots.txt and terms of service, on the web domains behind C4, RefinedWeb and Dolma.

Under Article 4(3) of Directive (EU) 2019/790, a rightholder may reserve text and data mining "in an appropriate manner, such as machine-readable means in the case of content made publicly available online".

A reservation reaches Dolma's web text only through the crawls it was built from, and the Dolma 3 crawl cutoff was 2024-12-31, so a reservation added after that date was not in place when those crawls were made. The rule is set out on the page about the EU text and data mining exception.

Content in Dolma: Benefits, Risks, Downsides and Alternatives for the Holder

For a person or company whose text sits in Dolma, the corpus brings no payment and no contract, and the only removal route Ai2 describes covers personal data. The four lists below separate what inclusion does from what it costs and end with the holder's options, including doing nothing.

Benefits

  • Ai2 documents sources, cutoffs and filters for each release.
  • ODC-BY carries an attribution notice, "obtained from Dolma", as a database-level duty only.

Risks

  • Downloads made in the ImpACT period stay in use under ODC-BY without redownload, and the documents read describe no way for a removal to reach copies already downloaded.
  • Ai2 does not guarantee removal from OLMo outputs.
  • No court ruling on file concerns Dolma, so its legal standing is untested, as of 2026-11-17.

Downsides

  • No notice and no consent were given to the authors.
  • The cards describe no route for copyright holders or site owners.

Alternatives

  • Reserve rights in machine-readable form and block CCBot, which obeys robots.txt, for future collections.
  • Filter the AI2Bot user agent, as Ai2's notice allows, for future collections.
  • Use the personal data form where the text is about a person.
  • License the text directly to AI developers, as the page on text corpora for AI training describes.
  • Not license at all and leave the text where it is.

The benefits are narrow and the risks depend on outcomes that have not arrived. Not licensing is a valid outcome, and the choice stays with the holder.

Other Corpora, Crawler Controls and the Law Beyond This Page

This page covers Dolma and its releases; the corpora inside it, the crawlers, removal routes and the wider law each have their own page. Privacy rules for training sets differ by jurisdiction.

Frequently Asked Questions About Dolma

What is the Dolma dataset?

Dolma is Ai2's English text corpus, first released on 2023-08-18. It is mostly Common Crawl web text plus papers, code, Reddit, books and Wikipedia, and Ai2 used it to train its OLMo models.

How big is Dolma?

Dolma v1.7 is 4.5 TB gzip and 2,308.5 billion OLMo tokens, as Ai2 states. The Dolma 3 pool is over 9 trillion tokens and the Dolma 3.5 pool nearly 10 trillion, each by Ai2's own count.

What license does Dolma use?

Dolma has used ODC-BY since 2024-04-15, after Ai2's ImpACT license. The original sources' terms also bind, and the Dolma 3 cards say the data is "intended for research and educational use".

Can I have my content removed from Dolma?

Ai2 offers a personal data removal form that asks for file names, the Dolma version and document IDs. The cards describe no route for copyright holders or site owners. This page is information, not legal advice.

Agents and advisers can use the introducer form.