The Brown Corpus of 1961 and the Penn Treebank of 1993 came first, and the earliest training license a party dates is Shutterstock's announcement of 2022-10-25.
The sequence behind AI training data licensing is told here as of 2026-11-17, and every date names its stating party. How we source, date and check every figure sets out the method.
What a History of Training Data Follows, and How Each Date Is Sourced
This history follows how data reached AI developers, from whom and on what terms, not how the models themselves developed.
Corpus documentation and papers date the research corpora, dockets date the lawsuits and settlements, and announcements and SEC filings date the licenses. A value appears only where the stating party gives it.
The Timeline From 1961 to 2026, Event by Event
The table lists the dated events of this history in order, from the Brown Corpus texts of 1961 to micro1's acquisition commitment of 2026-10-09, each with the party that states it, as of 2026-11-17.
| Date | Era | Event | Stated by | Source |
|---|---|---|---|---|
| 1961 (texts), 1964 (release) | Research corpora | Brown Corpus released, 1 million+ words | VARIENG corpus documentation | varieng.helsinki.fi |
| 1993 | Research corpora | Penn Treebank paper published | Marcus, Santorini, Marcinkiewicz | aclanthology.org |
| 1999 | Research corpora | Treebank-3 distributed under LDC agreements | Linguistic Data Consortium | catalog.ldc.upenn.edu |
| 2007 | Web crawl | Common Crawl founded | Common Crawl | commoncrawl.org |
| 2008 | Web crawl | Common Crawl collection begins | Common Crawl | commoncrawl.org |
| 2009-06 | Research corpora | ImageNet paper at IEEE CVPR 2009 | IEEE (Crossref record) | doi.org/10.1109/CVPR.2009.5206848 |
| 2019 | Web crawl | OpenAI's WebText described in the GPT-2 paper | OpenAI | cdn.openai.com |
| 2019-01-01 | Web crawl | Japan Copyright Act Art. 30-4 in force | Government of Japan | laws.e-gov.go.jp |
| 2019-04 | Web crawl | Common Crawl snapshot used for C4 | Raffel et al. | arxiv.org |
| 2019-06-06 | Web crawl | DSM Directive (EU) 2019/790 in force | EU | eur-lex.europa.eu |
| 2020-12-31 | Mixed corpora | The Pile, including Books3, described | EleutherAI paper | arxiv.org |
| 2022-03-31 | Mixed corpora | LAION-5B announced | LAION | laion.ai |
| 2022-10-25 | Licensing and channels | Shutterstock expands its OpenAI partnership and launches the Contributor Fund | Shutterstock | submit.shutterstock.com |
| 2023-01-13 | Licensing and channels | Andersen v. Stability AI filed | court record | archive.org (RECAP) |
| 2023-01-16 | Licensing and channels | Getty Images claim against Stability AI issued (UK) | court record | judiciary.uk |
| 2023-07-07 | Licensing and channels | Kadrey v. Meta filed | court record | archive.org (RECAP) |
| 2023-07-13 | Licensing and channels | OpenAI licenses part of AP's text archive | The Associated Press | ap.org |
| 2023-08-14 | Licensing and channels | Books3 removal announced | Danish Rights Alliance | rettighedsalliancen.dk |
| 2023-12-13 | Licensing and channels | Axel Springer and OpenAI partnership announced | Axel Springer | axelspringer.com |
| 2023-12-19 | Licensing and channels | LAION-5B taken down | LAION | laion.ai |
| 2023-12-27 | Licensing and channels | NYT v. Microsoft and OpenAI filed | court record | archive.org (RECAP) |
| 2024-01 | Licensing and channels | Reddit data licensing arrangements, USD 203.0 million aggregate contract value, counterparties not named | Reddit S-1 (filed 2024-02-22) | sec.gov |
| 2024-02-22 | Licensing and channels | Google and Reddit partnership, value not disclosed | blog.google | |
| 2024-06-13 | Licensing and channels | Wiley USD 23 million content rights project, buyer not named | Wiley 8-K exhibit | sec.gov |
| 2024-07-20 | Licensing and channels | Robots.txt restrictions on C4 sources measured | Longpre et al. | arxiv.org |
| 2024-08-01 | Licensing and channels | EU AI Act in force | EU | eur-lex.europa.eu |
| 2024-08-30 | Licensing and channels | Re-LAION-5B released | LAION | laion.ai |
| 2024-09-27 | Licensing and channels | Kneschke v. LAION decided, LG Hamburg | Federal Court of Justice case notice | bundesgerichtshof.de |
| 2024-11-19 | Licensing and channels | HarperCollins licenses at USD 5,000 per title, buyer not named | Authors Guild | authorsguild.org |
| 2025-04-17 | Licensing and channels | Tempus and Pathos AI order form, USD 200 million in fees over three years | Tempus 10-K and 10-Q | sec.gov |
| 2025-06-23 | Licensing and channels | Bartz v. Anthropic Order on Fair Use | court record | archive.org (RECAP) |
| 2025-06-25 | Licensing and channels | Kadrey v. Meta summary judgment | court record | archive.org (RECAP) |
| 2025-10-29 | Licensing and channels | UMG and Udio agreements, value not disclosed | Universal Music Group | universalmusic.com |
| 2025-11-04 | Licensing and channels | Getty Images v. Stability AI judgment (UK) | High Court | judiciary.uk |
| 2025-11-11 | Licensing and channels | GEMA v. OpenAI judgment, not final | LG München I | justiz.bayern.de |
| 2025-12-10 | Licensing and channels | Google News AI pilot, amounts not disclosed | blog.google | |
| 2025-12-19 | Licensing and channels | Kopinor collective license, NOK 45,000,000 for 2026 | Kopinor agreement | kopinor.no |
| 2026-02-03 | Licensing and channels | Microsoft Publisher Content Marketplace announced | Microsoft | about.ads.microsoft.com |
| 2026-07-20 | Licensing and channels | Bartz v. Anthropic final approval, USD 1.5 billion | court record | archive.org (RECAP) |
| 2026-08-14 | Licensing and channels | Spirit Airlines auction: Google LLC successful bid USD 10,000,000, not approved as of 2026-11-09 | Bankr. S.D.N.Y. Dkt. 1463 | courtlistener.com |
| 2026-09-29 | Licensing and channels | Thomson Reuters v. Ross affirmed, Third Circuit | court | ca3.uscourts.gov |
| 2026-10-09 | Licensing and channels | micro1 states a USD 1 billion commitment | micro1 | micro1.ai |
The era column uses the four bands of the comparison table below. A disclosure date can differ from the event date: Reddit's January 2024 arrangements were disclosed in its filing of 2024-02-22.
Research Corpora Compiled and Distributed Under Agreements, 1961 to 2009
Before the web crawl, training and test data for language and vision research came from corpora that universities compiled for study and distributed under agreements.
The Brown Corpus holds 500 samples of US texts printed in 1961, over 1 million words, compiled by W. Nelson Francis and Henry Kučera at Brown University and released in 1964. VARIENG's corpus documentation describes it as the first computer-readable general corpus of texts prepared for linguistic research on modern English.
Marcus, Santorini and Marcinkiewicz described the Penn Treebank in Computational Linguistics 19(2) in 1993. It is distributed as Treebank-3 (LDC99T42, 1999), one million words of 1989 Wall Street Journal material, under the LDC User Agreement for Non-Members.
LDC states that its For-Profit Membership Agreement covers commercial development, at USD 34,000 (Standard) and USD 40,000 (Subscription) as checked 2026-11-09, so fee-based licensing of research corpora predates the crawl. The Linguistic Data Consortium still licenses corpora under agreements.
ImageNet followed in June 2009, when Deng and five co-authors published "ImageNet: A large-scale hierarchical image database" at IEEE CVPR 2009. These corpora were small, named and documented.
The Web Crawl Becomes the Default Source, 2007 to 2020
From 2008 the open web became the main supply of training text, through a nonprofit archive open to anyone and through filtered sets built from it.
Common Crawl was founded in 2007 and states that it has regularly collected petabytes of data since 2008 (checked 2026-11-09). It states that at least 64 % of 47 LLMs published from 2019 to 2023 trained on its data, citing a 2024 Mozilla Foundation study.
OpenAI's GPT-2 paper of 2019 built WebText from 45 million outbound Reddit links with at least 3 karma. Raffel et al. built C4 from the April 2019 snapshot. The Common Crawl profile covers what the archive holds and its terms.
Two statutes arrived in the same years: Japan's Copyright Act Art. 30-4, in force since 2019-01-01 with a proviso on uses that unreasonably prejudice the owner, and Directive (EU) 2019/790, in force on 2019-06-06, whose Art. 4(3) reservation must be machine-readable.
Mixed Corpora With Copied Books and Scraped Images, 2020 to 2022
From 2020 open corpora began to mix the crawl with books copied without the rightholders' license, and image sets of billions of scraped links.
The Pile's paper of 2020-12-31 describes 825 GiB in 22 subsets, of which Books3, derived from a private book tracker, is 100.96 GiB raw and a 12.07 % weight. Anthropic's copy of Books3 was downloaded in January or February 2021, per the Order on Fair Use of 2025-06-23 (Dkt. 231).
LAION announced LAION-5B on 2022-03-31: 5.85 billion image-text pairs, holding links and captions rather than images, with metadata under CC BY 4.0 and images under their own copyright, as LAION states. The dataset profiles hold each corpus's contents. The LAION-5B and Re-LAION-5B profile covers the image set and its 2024 rerelease.
Takedowns, Court Filings and Crawl Restrictions, 2023 and 2024
In 2023 and 2024 rightholders answered the scraped corpora with lawsuits, takedowns and crawl restrictions, and each kind of response is dated on the record.
- Filings: Andersen v. Stability AI (3:23-cv-00201-WHO) on 2023-01-13, Getty Images' UK claim against Stability AI on 2023-01-16, Kadrey v. Meta (3:23-cv-03417-VC) on 2023-07-07 and NYT v. Microsoft and OpenAI (1:23-cv-11195-SHS-OTW) on 2023-12-27, pending as of 2026-11-09.
- Takedowns: the Danish Rights Alliance announced on 2023-08-14 that Books3 was removed from its host, about 200,000 books by its own figure. LAION took LAION-5B down on 2023-12-19 and released Re-LAION-5B on 2024-08-30.
- Restrictions: Longpre et al. (2024-07-20) measured more than 5 % of C4 tokens, and more than 28 % of the most critical sources, as fully restricted by robots.txt.
The Regional Court of Hamburg decided Kneschke v. LAION (310 O 227/23) on 2024-09-27. The Higher Regional Court dismissed the appeal on 2025-12-10, and the Federal Court of Justice has set pronouncement for 2026-12-17. The AI copyright lawsuits over training data tracker follows every filing above to its ruling.
The First Paid Licenses for Training Data, 2022 to 2024
Paid licenses for training data appear on the record from late 2022, first for stock images, then for news archives, platform posts and books.
- 2022-10-25: Shutterstock announces an expanded partnership with OpenAI and the Contributor Fund, with contributor earnings from OpenAI datasets first issued in Q4 2022; value not disclosed.
- 2023-07-13: the Associated Press states that OpenAI licenses part of its text archive; value not disclosed.
- 2023-12-13: Axel Springer announces its partnership with OpenAI; value not disclosed.
- 2024-02-22: Reddit's S-1 states USD 203.0 million in aggregate contract value for January 2024 data licensing arrangements of two to three years, counterparties not named. The same day Google states that the partnership gives it Reddit Data API access to "display, train on, and otherwise use" content, with no value.
- 2024-06-13: a Wiley filing exhibit describes a USD 23 million content rights project; buyer not named.
- 2024-11-19: the Authors Guild reports HarperCollins at USD 5,000 per title, split 50-50 with the opted-in author; buyer not named.
These values are what the parties state. The AI data licensing deals with parties, dates and values tracker carries the full record.
Programs, Filed Revenue, Settlements and Court Sales, 2025 and 2026
In 2025 and 2026 single deals gave way to standing channels: multi-year data contracts in filings, publisher programs paid by usage, a national collective license, a bankruptcy auction of company data and court-approved settlements.
Tempus states in its filings USD 200 million in data license fees over three years under an order form with Pathos AI dated 2025-04-17. Google states on 2025-12-10 that it pays participating publications in a news pilot.
Kopinor and the National Library of Norway signed a collective license on 2025-12-19 for NOK 45,000,000 for 2026. Microsoft states on 2026-02-03 that publishers "will be paid on delivered value" in its marketplace for grounding Copilot answers.
In Spirit Airlines' bankruptcy (Bankr. S.D.N.Y. 25-11897), the auction of 2026-08-14 produced a successful bid from Google LLC and an alternate bid from Mercor.io Corporation (Dkt. 1463).
The sale was not approved as of 2026-11-09, and a hearing was set for 2026-11-17 (Dkt. 1779); its outcome is not stated here. micro1 states a USD 1,000,000,000 enterprise data acquisition commitment over 12 months, announced on 2026-10-09 (checked 2026-11-09).
In Bartz the Order on Fair Use of 2025-06-23 held purchased-and-scanned print books fair use but not pirated library copies, and final approval of 2026-07-20 (Dkt. 680) fixed a USD 1.5 billion non-reversionary settlement for 482,460 works.
The Bartz v. Anthropic and the $1.5 billion settlement page follows the rulings and payments as of 2026-11-17. In Kadrey the court granted Meta summary judgment on 2025-06-25, writing that plaintiffs were "not entitled to the market for licensing their works as AI training data" and that the ruling "does not stand for the proposition that Meta's use ... is lawful".
The Third Circuit, affirming on 2026-09-29 in Thomson Reuters v. Ross (No. 25-2153), wrote that "the market for licensing headnotes as text to train AI is rapidly developing" and that ROSS's AI "was not a generative AI"; a rehearing petition is filed.
Neither statement is presented as the law. Getty's UK judgment of 2025-11-04 is on appeal with judgment reserved since 2026-11-12, and the GEMA v. OpenAI judgment of 2025-11-11 is not final.
What Changed From Era to Era: Source, Permission and Payment
Across the four eras the typical source, the basis for using it and the party that was paid all changed, while earlier methods never disappeared, as the table shows as of 2026-11-17.
| Era | Typical source | Basis for use on record | Who was paid on record | Example | Source |
|---|---|---|---|---|---|
| Research corpora, 1961 to 2009 | compiled and annotated text, speech and image collections | corpus agreements; LDC user agreements and memberships | the distributing institution (LDC fees) | Brown Corpus, Treebank-3, ImageNet | corpus documentation, LDC catalog |
| Web crawl, 2007 to 2020 | the open web via Common Crawl and filtered sets | terms of use of the archive; TDM exceptions in Japan (in force 2019) and the EU (Directive 2019/790, transposition deadline 2021-06-07) | no one on record | Common Crawl, WebText, C4 | commoncrawl.org, research papers, eur-lex |
| Mixed corpora, 2020 to 2022 | crawl plus copied books and scraped image links | dataset licenses on metadata only (CC BY 4.0 for LAION-5B); no rightholder license for Books3 | no one on record | The Pile, Books3, LAION-5B | arxiv.org, laion.ai |
| Licensing and channels, 2022 to 2026 | licensed archives, platform APIs, books, music, clinical and company data | signed licenses, collective licenses, settlements, bankruptcy sale bids | rightholders, platforms, publishers, settlement classes | Shutterstock, AP, Reddit, Kopinor, Bartz, Spirit | parties' announcements, SEC filings, dockets |
Crawling continues: Common Crawl announced on 2026-10-21 its crawl CC-MAIN-2026-43, collected from 2026-10-05 to 2026-10-18 with 2.16 billion pages. The legal exposure of each route is in licensed vs scraped training data.
Dates That Circulate but Are Not Confirmed Here
Some dates and values repeated in histories of training data have no primary source on record, so this page leaves them out.
- The start date of Shutterstock's original OpenAI license.
- The term of any 2023 expansion of that Shutterstock partnership.
- Yearly values for Google's use of Reddit content.
- The composition of GPT-3's training mix.
- Any 2026 agreement between Anthropic and the Associated Press.
ImageNet's image counts and access terms, the Treebank-3 fee and the launch dates of training crawlers are likewise not stated here.
What the History Shows a Data Holder Today
The history shows that data holders were paid only once their data was licensed rather than crawled, and that the amounts on record come from filings, dockets and the parties themselves.
Reddit's aggregate contract value, the Tempus fees, the Kopinor amount and the Bartz settlement each rest on a filing, an agreement or a docket, while micro1's commitment is its own claim.
Value depends on rights, exclusivity, data type and buyer, and not licensing remains an option. The valuation guide covers what company data is worth to AI companies.
Where the History Connects to the Deal Tracker, the Datasets, the Cases and the Market Section
History is one measure in the market section; the deal tracker, the dataset profiles and the case tracker carry each event in full.
- the AI training data market: the parent
- AI training data market size: estimates by publisher
- AI training data prices: per-unit prices
- funding and valuations of AI data companies: rounds and valuations
- major AI training datasets: every corpus named here
- The Pile: the 2020 corpus
- Books3: the book component
- Shutterstock's AI licensing deals: the first dated license
- The Associated Press AI licensing deals: the 2023 news license
- Reddit's AI data licensing deals: the filed contract value
Counts of data and estimates of its supply sit in AI training data statistics, and the court sale of company data is in Spirit Airlines' data archive sale in bankruptcy.
How the Timeline Is Sourced and Updated
Each event is dated from the stating party's own record, a court docket, an official journal or a filing, and a new row is added when a party or court records a new step, such as the Spirit Airlines sale hearing. The checks follow how we source, date and check every figure.
Frequently Asked Questions
When did AI companies start paying for training data?
The earliest training license a party dates is Shutterstock's: on 2022-10-25 it announced an expanded partnership with OpenAI, with contributor earnings from OpenAI datasets first paid in Q4 2022. Research corpora were already distributed for fees decades earlier.
What were the first corpora used to train language and vision models?
Research corpora such as the Brown Corpus (texts of 1961, released 1964), the Penn Treebank (1993) and ImageNet (2009) came first, followed by the web crawl from 2008. None is described here as the first AI training dataset.
Why did AI developers move from scraped data to licensed data?
No developer states one reason on the record, but the same years brought lawsuits over scraped corpora from 2023, takedowns of Books3 and LAION-5B, and robots.txt restrictions on crawled sources. No motive is attributed.
Are AI companies running out of training data?
Epoch AI estimates an effective stock of about 300 trillion tokens of public human-generated text, fully used between 2026 and 2032 (published 2024-06-06, checked 2026-11-09); it is an estimate, not a measurement. Related figures are collected under AI training data statistics.