How AI Companies Get Training Data: Crawling, Licensing, Users, Humans and Synthetic

AI companies get training data through nine routes: they crawl the web, copy corpora built from crawls, license content, train on their own users, commission people, buy company records, use out-of-copyright works, have used unlicensed book copies, and generate data with their own models. Only licensing and commissioned work put a payment on record, while the free layer keeps growing: Common Crawl's October 2026 archive alone holds 2.16 billion web pages.

This hub maps each route to how the data is obtained, whether the holder is paid and what the holder controls. It belongs to an independent reference on AI training data licensing that takes no cut of any sale.

How do AI companies get training data?

Most training data is taken rather than bought: the public web, crawled by the developers' own bots or by Common Crawl, is the base layer, and licenses, contractors and purchased records are added on top. Anthropic's Form S-1 of 2026-10-26 lists three sources: public web data, datasets licensed or commissioned from third parties, and synthetic data generated by its own models. xAI's public summary of training content for Grok 4.7, of 2026-09-21 from SpaceXAI LLC (formerly xAI LLC), answers Yes for commercial licenses with rightsholders (text and image only), private third-party datasets, user interaction data and synthetic data.

The licensed share is small where stated. OpenAI told the Office of the Privacy Commissioner of Canada that licensed third-party datasets made up under 1 % of its pre-training data; the regulator recorded the statement in PIPEDA Findings #2026-002, published 2026-05-06, and could not verify it.

The nine routes at a glance

The table sorts the routes by how the data is obtained, whether the holder is paid, what the holder controls and whether a developer names the route in its own disclosure, as of 2026-11-10.

• "Paid on record" means a documented payment to the holder or contributor, never a price.
• "Named" cites a developer's own disclosure, never an inference.
• Child-page routes link from the final table.

How each route works

The nine routes below follow the table's order, from the unpaid and largest to the paid and narrow, then synthetic data.

Crawling the public web

Developers run their own crawlers, such as OpenAI's GPTBot, and treat what the crawler collects as training material unless the site opts out. OpenAI states that disallowing GPTBot signals content should not be used in training (checked 2026-11-09), and xAI names the SpaceXAI Web Crawler, which crawled for Grok 4.7 from January 2024 to August 2026. Cloudflare measured that nearly 80 % of AI crawling between 2025-07-01 and 2025-07-28 served training.

Downloading public corpora built from the crawl

Many models start from free corpora that others built from the crawl, such as Common Crawl's archives and the filtered sets made from them. The Common Crawl Foundation, a 501(c)(3) nonprofit founded in 2007, published its latest archive, CC-MAIN-2026-43, on 2026-10-21, and The Pile (2020) packs 825 GiB from 22 subsets. A Data Provenance Initiative audit of over 1,800 text datasets (2023-10-25) found licenses omitted in over 70 % of cases on hosting sites and wrong in over 50 %.

Licensing content from platforms, publishers and catalogs

Licensing is the route that pays the holder, and the few values stated by a party come from filings rather than announcements. Reddit's S-1 of 2024-02-22 gives an aggregate contract value of USD 203.0 million for January 2024 data licenses with counterparties it does not name, and Wiley's 10-K reports AI licensing revenue for the year ended 2026-04-30 of USD 33.1 million (Research) and USD 16.0 million (Learning). Google states over 3,000 content partnerships in more than 50 countries (2025-12-10), not limited to training.

Training on the content of their own users

Services that host content or conversations train on it under their own terms, and the user's control is a setting, an objection form or nothing. Anthropic keeps consumer chats for 5 years where users allow training, Meta trains on public EU posts from 2025-05-27 (Irish Data Protection Commission), GitHub on Copilot Free, Pro and Pro+ interaction data from 2026-04-24, and LinkedIn unless members turn off "Data for Generative AI Improvement". Slack trains on customer data only with opt-in consent, and Microsoft states that Microsoft 365 Copilot data does not train foundation models (both checked 2026-11-09).

Commissioning people to write, rate and demonstrate

Developers commission contributors, mostly through human-data companies, to write answers, rank model outputs and record demonstrations that the web does not contain. Anthropic's S-1 lists datasets commissioned from third parties, and its HH-RLHF paper, submitted on 2022-04-12, is built on human preference data collected from crowdworkers. Outlier, Scale AI's contributor platform, states USD 500M+ paid to experts on its home page and USD 600M+ paid out on its about page, both checked 2026-11-09.

Buying company records through intermediaries and insolvency sales

Operating companies' internal records reach developers mostly through human-data companies that acquire them for model builders, and occasionally through court-supervised sales. The human-data company micro1 states a commitment of USD 1 billion over 12 months for enterprise data acquisition, announced 2026-10-09, a pledge rather than money paid. On 2026-08-14 the Spirit Airlines auction named Google LLC successful bidder at USD 10,000,000 and Mercor.io Corporation alternate at USD 7,500,000 (Bankr. S.D.N.Y. 25-11897 (SHL), Dkt. 1463); the sale is not approved as of 2026-11-09, hearing set for 2026-11-17 (Dkt. 1779).

Using out-of-copyright and permissively licensed works

Works whose copyright has expired, US federal government works and openly licensed collections can be trained on without paying the author, within each license's terms. In the US, 17 U.S.C. 105(a) denies copyright to federal works and 304(b) limits renewal-term works to 95 years, while EU terms run for life plus 70 years from 1 January of the following year (Directive 2006/116/EC, Arts. 1(1), 8). Pleias states 2,267,302,720,836 tokens for Common Corpus, and Institutional Books 1.0 releases 983,004 volumes whose commercial AI use requires contacting its publisher IDI.

Unlicensed book copies: what the court record shows

Court records show that an AI developer downloaded millions of books from shadow libraries, and that a US court refused to treat keeping those pirated copies as fair use. In Bartz v. Anthropic (N.D. Cal. 3:24-cv-05417), the Order on Fair Use of 2025-06-23 (Dkt. 231) records 196,640 Books3 books, at least five million LibGen copies and at least two million PiLiMi copies; the case settled with final approval on 2026-07-20 (Dkt. 680). The Danish rights group RettighedsAlliancen announced on 2023-08-14 that it had Books3, about 200,000 books by its count, removed from its host.

Generating data with their own models

Developers also train on data their own earlier models generate, and both Anthropic's S-1 and xAI's Grok 4.7 summary list it as a source. The model-collapse study by Shumailov et al. in Nature (online 2024-07-24) marks the limit, and Dohmatob et al. (2024-10-07) find in a regression setting that as little as 1 % synthetic data can trigger collapse. Buyers of real records reject the substitute: Mercor lists "Synthetic or sample charts" and "Synthetic or scripted traffic" as not a fit, checked 2026-11-09.

How much of the data is paid for: the evidence on record

No developer publishes the share of its training data it paid for, but papers, filings and a regulator finding put the unpaid web at the center of the mix. The Commission's template for EU training content summaries (C(2025) 8311 final, 2025-12-05) asks only Yes or No plus modality for licensed data, so even full compliance yields no paid share. Lab by lab, see how frontier AI labs acquire data.

The table lists each figure with the party that states it and its date, as of 2026-11-10; the rows measure different things and are never combined.

Restrictions on the unpaid route are growing, as the Consent in Crisis audit of about 14,000 domains and Cloudflare's 2026 figure show. A restriction does not reach archives already made: Common Crawl states that its WARC files are immutable and that affected URLs are removed or filtered from later crawls.

The rules for each row are set out in how we source, date and check every figure.

What decides whether a route is lawful: consent, opt-outs and fair use rulings

Whether a route is lawful depends on where the developer trains and how it obtained the copy, not on whether the data was public. In the United States, the Bartz order of 2025-06-23 held training "exceedingly transformative" and fair use, and scanning purchased print books fair use, but did not excuse the pirated copies. Kadrey v. Meta (N.D. Cal. 3:23-cv-03417-VC, Dkt. 598, 2025-06-25) granted Meta summary judgment on training on that record, yet the ruling "does not stand for the proposition that Meta's use ... is lawful".

In the EU, rightholders can reserve online content from text and data mining, but the reservation must be machine-readable (Directive (EU) 2019/790, Art. 4(3)), and Art. 7(1) does not protect that exception against contract override. From 2025-08-02 the AI Act (Regulation (EU) 2024/1689, Art. 53) requires general-purpose model providers to publish a training content summary, which under the Commission template lists the top 10 % of scraped domain names (SMEs 5 % or 1,000 domains).

As of 2026-11-10, the Bundesgerichtshof is set to pronounce in Kneschke v. LAION (I ZR 281/25) on 2026-12-17, and Advocate General Szpunar's non-binding opinion of 2026-11-03 in Like Company v. Google (C-250/25) calls the training questions inadmissible, adding that training is as a rule permitted under the TDM exception; the judgment is pending. For the user route, FTC Office of Technology staff wrote on 2024-02-13 that adopting "more permissive data practices" may be unfair or deceptive, in a staff blog post, not a Commission rule.

This section is information, not legal advice; the instruments are compared under copyright and AI training across jurisdictions.

Where a holder's data enters these routes: benefits, risks, downsides and alternatives, including not licensing

A data holder meets these routes in three places: its published content is crawled, its content on other platforms is trained on under their terms, and its internal records reach developers only if licensed.

Benefits of licensing

• Licensing is the route where a payment to the holder is on record, in Reddit's and Wiley's filings.
• A license fixes scope, term and deletion terms that the crawl never offers.
• Licensed and commissioned data is named in developers' own disclosures, such as Anthropic's S-1.

Risks

• Data that has trained a model cannot be withdrawn from it, so the terms fixed at signing govern its use.
• Employee and customer content may carry rights the company does not hold, which a rights audit of the data to be licensed settles before any offer.
• Courts and regulators examine how a copy was obtained (the Bartz order, the FTC staff statement).

Downsides

• Values stated by a party are rare: the licensing route above rests on two filings.
• Opt-out signals are uneven: the __usage_ai-training token is an individual Internet-Draft of 2026-10-30, not an IETF standard, and Google aims to support a shared AIPREF training token in 2027 without naming its string (checked 2026-11-10).

Alternatives

• Block training crawlers by name in robots.txt, or charge them where the host offers it.
• Leave platform settings off: YouTube's third-party training setting is off by default, and YouTube states it "isn't facilitating payments" to creators (checked 2026-11-09).
• Release the content under an open license.
• Do not license at all and keep internal records private.

Each alternative keeps control with the holder but forgoes the paid route. The decision page weighs whether to sell or license data to AI companies at all.

Where the sources hub leads: its five pages and the buyer side

The five pages below each take one route or question in depth (alphabetical, as of 2026-11-10); the sections after them cover who acquires data, who holds it and what it is worth.

The reference continues on the buyer and seller side:

• Who buys AI training data: every buyer and program.
• Who sells data to AI companies: every type of seller.
• Data types AI companies buy: what sells and for how much.
• The AI training data market: size and prices.
• Training data provenance: the chain of rights.
• Ethics of selling data for AI training: risks beyond the contract.

Frequently asked questions

Where did AI get its information from?

Mostly from the public web, crawled by developers' own bots or taken from Common Crawl's archives. Developers add licensed and commissioned datasets and synthetic data, all named in Anthropic's Form S-1 of 2026-10-26.

How do I stop AI from collecting my data?

A site can disallow named training crawlers such as GPTBot in robots.txt, and a user can switch off training settings where a platform offers them. OpenAI notes that user-initiated fetchers may ignore robots.txt. Rules and examples are in robots.txt for AI crawlers; platform settings are on the opt-out page listed above.

Is scraped training data the same as licensed data?

No: scraped content is copied without a contract, while licensed content comes under one that sets scope, term and payment. Rights and risk differ. Lawfulness turns on copyright, contract and computer-misuse law, covered under whether web scraping for AI training is legal.

Do AI companies train on pirated books?

Court records show that at least one developer built its library from them: the Bartz order of 2025-06-23 records over seven million pirated book copies kept by Anthropic. The case settled for USD 1.5 billion with final approval on 2026-07-20. It is tracked with other AI copyright lawsuits over training data.