Scraping web content to train an AI model is not lawful or unlawful as such: the answer depends on which of five bodies of law a site owner can invoke, namely computer-misuse law, contract, anti-circumvention, copyright and database right, and data protection. In the United States, public pages escape the Computer Fraud and Abuse Act, but the other four still apply, as of 2026-11-09.
The page covers the United States, the European Union, Japan, Singapore and Canada. It treats scraping as one route within AI training data licensing, so it states what a developer can face and what a holder can invoke.
Is Scraping Public Web Pages for AI Training Legal?
Scraping pages anyone can open is generally not a computer-misuse offense in the United States, but it can still breach the site's terms, circumvent an access control, copy protected works or process personal data unlawfully. The Supreme Court's Van Buren decision (No. 19-783, 2021-06-03) limits "exceeds authorized access" to off-limits areas of a computer, and the Ninth Circuit in hiQ v. LinkedIn (No. 17-16783, 2022-04-18) wrote that "the concept of without authorization does not apply to public websites". The same opinion says trespass to chattels claims "may still be available" and copyright, contract and privacy claims "may also lie".
The European Union has no general rule on scraping, but four regimes apply at once: the text and data mining reservation in Art. 4(3) of Directive (EU) 2019/790, the database right in Art. 7(1) of Directive 96/9/EC, the GDPR, and contract, as Ryanair v PR Aviation (C-30/14, 2015-01-15) shows. Singapore ties its copyright exception to lawful access (Copyright Act 2021 ss.243-244, in force 2021-11-21), and Japan's Art. 30-4 carries a proviso.
Seven Claims a Scraper for AI Training Can Face
The table sets the seven claims side by side with the provision, what it protects, its effect on scraping pages that need no login, and the authority that decided it, as of 2026-11-09.
| Claim | Citation and date | What it protects | Effect on scraping pages without a login | Leading authority | Source |
|---|---|---|---|---|---|
| Computer misuse (US) | 18 U.S.C. 1030(a)(2) as construed 2021-06-03 | areas of a computer that are off-limits | none for public pages; gated areas protected | Van Buren, No. 19-783; hiQ, 9th Cir. No. 17-16783 (2022-04-18) | supremecourt.gov; ca9.uscourts.gov |
| Contract (terms of use) | state contract law; EU: C-30/14 (2015-01-15); Dir. (EU) 2019/790 Art. 7(1) | the site's own terms | claim survives where terms bind; not preempted by copyright (N.D. Cal. 2026-03-30) | Reddit v. Anthropic; Ryanair v PR Aviation | courtlistener.com; publications.europa.eu |
| State torts and unfair competition (US) | trespass to chattels, unjust enrichment, Cal. Bus. & Prof. Code 17200 | servers, competitive position | case by case; unjust enrichment and unfair competition dismissed in Reddit v. SerpApi (2026-07-31) | hiQ (claims "may also lie"); Reddit v. SerpApi Dkt. 104 | ca9.uscourts.gov; courtlistener.com |
| Anti-circumvention (US) | 17 U.S.C. 1201 | technological measures controlling access | applies only where a measure is circumvented | Reddit v. SerpApi Dkt. 104 (1201(a)(1)(A) sustained) | govinfo.gov; courtlistener.com |
| Copyright in the copies | 17 U.S.C. 107; Dir. (EU) 2019/790 Art. 4; Japan Art. 30-4; Singapore ss.243-244 | protected works on the page | depends on the jurisdiction's exception and its conditions | Bartz v. Anthropic Dkt. 231 (2025-06-23) | govinfo.gov; eur-lex.europa.eu |
| EU database right | Dir. 96/9/EC Art. 7(1), 7(3), 10(1) | databases with substantial investment, 15 years | extraction of a substantial part needs consent or an exception | Ryanair (unprotected databases: contract only) | eur-lex.europa.eu |
| Data protection | GDPR; PIPEDA; national laws | people named or shown on the page | legal basis needed; public status alone is not enough (Canada 2026-05-06) | EDPB Opinion 28/2024; PIPEDA Findings #2026-002 | edpb.europa.eu; priv.gc.ca |
Computer-Misuse Law: Gates Up or Down After Van Buren and hiQ
Under the US Computer Fraud and Abuse Act a scraper commits no offense by reading pages that require no authorization, because the Supreme Court reads "exceeds authorized access" as a gates-up-or-down question. In Van Buren v. United States (No. 19-783, decided 2021-06-03) the Court held that a person exceeds authorized access under 18 U.S.C. 1030(a)(2) only by entering areas of a computer that are off-limits to that person.
In hiQ Labs v. LinkedIn (9th Cir. No. 17-16783, opinion 2022-04-18) the court affirmed a preliminary injunction and held that "the concept of without authorization does not apply to public websites". Areas that require authorization remain protected. Winning on the CFAA therefore leaves other claims open. Computer-misuse statutes outside the United States are not covered here yet.
Terms of Use and Contract Claims Against Scrapers
A site's terms of use are the claim most often left once computer-misuse law falls away, and a US federal court has held that such state-law claims over scraping for AI training are not preempted by copyright. In Reddit v. Anthropic (San Francisco Superior Court CGC-25-625892; federal 3:25-cv-05643-TLT) Reddit pleaded breach of contract, unjust enrichment, trespass to chattels, tortious interference and unfair competition over automated access to train Claude. The N.D. Cal. remanded the case to state court on 2026-03-30 (Dkt. 60), and Anthropic's S-1/A reports that the claims narrowed in September 2026 to breach of contract, interference with contract and unfair competition.
In the EU, Ryanair v PR Aviation (C-30/14, 2015-01-15) held that Directive 96/9/EC does not apply to a database protected by neither copyright nor the sui generis right, so contractual limits on its use are not precluded. Art. 7(1) of Directive (EU) 2019/790 protects the research exceptions in Arts. 3, 5 and 6 against contract override but does not list Art. 4, so a term can override the exception described in the EU text and data mining exception page.
Singapore goes the other way for its own exception: under s.244(2)(d) of the Copyright Act 2021, copies made in breach of database terms of use lack lawful access, yet s.187(1)(c) voids any term that excludes the permitted use. Longpre et al. found 45 % of the C4 corpus restricted by terms of service in 2023 to 2024.
Circumventing Access Controls: Anti-Circumvention, Paywalls and robots.txt
Getting around a technical barrier is a separate claim from copying: in the US, 17 U.S.C. 1201 makes circumventing a technological measure actionable, and courts have let such claims proceed against scrapers. In Reddit v. SerpApi et al. (S.D.N.Y. 1:25-cv-08736-PAE, Dkt. 104, 2026-07-31) the court sustained 1201(a)(1)(A) against SerpApi and Perplexity, 1201(a)(2) against SerpApi and a New York civil conspiracy claim, while 1201(b) against SerpApi and the state unjust enrichment and unfair competition claims were dismissed. Reddit alleges almost three billion search result pages were accessed; the case concerns an answer engine, not training. In Google v. SerpApi (N.D. Cal. 4:25-cv-10826-YGR) the original complaint was dismissed with leave to amend on 2026-07-20 (Dkt. 42), and no ruling on the amended complaint was recorded as of 2026-10-24.
Measure 1.2 of the voluntary GPAI Code of Practice (2025-07-10) asks providers to respect paywalls and subscription restrictions and Singapore's s.244(2)(d) gives circumventing a paywall as an example of no lawful access.
A robots.txt file is a request, not a lock, so it has legal effect only through a rule. Art. 4(3) of Directive (EU) 2019/790 makes a machine-readable reservation effective, Japan's Agency for Cultural Affairs reads circumvention of robots.txt or ID-password measures into the Art. 30-4 proviso (May 2024, non-binding), and Measure 1.3 of the voluntary Code asks crawlers to follow robots.txt under IETF RFC 9309.
Copying the Scraped Content: Copyright Exceptions and the EU Database Right
Each page a scraper stores is a copy, so copyright decides whether the developer needed a license, and in the EU the database right can protect a site's collection as a whole. US fair use under 17 U.S.C. 107 did not excuse pirated library copies in Bartz v. Anthropic (Dkt. 231, 2025-06-23); the EU, Japan and Singapore use exceptions with conditions, compared in copyright and AI training across jurisdictions.
Under Art. 7(1) of Directive 96/9/EC a maker with substantial investment can prevent extraction or re-utilization of a substantial part of a database, Art. 7(3) makes the right licensable, and Art. 10(1) sets a term of 15 years. Providers also face Art. 53(1)(d) of Regulation (EU) 2024/1689, applicable from 2025-08-02, whose training summary template lists the top 10 % of domain names by content scraped (SMEs: top 5 % or 1,000 domains, whichever is lower).
Scraped Pages That Contain Personal Data
Scraped pages that name or show people are personal data, and privacy regulators treat their collection for AI training as processing that needs a legal basis, whatever the page's public status. EDPB Opinion 28/2024 (adopted 2024-12-17) addresses legitimate interest for model development and the effect of unlawful development, and para. 114 lists corrective measures up to erasure of the dataset or model. The EDPB adopted Guidelines 03/2026 on web scraping in the context of generative AI for public consultation on 2026-07-07, the consultation closed on 2026-10-30, and no final version is recorded.
Twelve authorities issued a joint statement on data scraping on 2023-08-24, and 16 signed the concluding statement of October 2024. In PIPEDA Findings #2026-002 (concluded 2026-05-06) the Canadian regulators found no implied consent for web-scraped data and held that indiscriminate scraping does not meet the publicly-available exception. The same duties are explained in GDPR and AI training data.
On the site-owner side, the Italian Garante's note of 2024-05-20 suggests four measures: reserved areas, clauses in the terms of service, traffic monitoring and bot countermeasures. CNIL's non-binding sheets 8 and 8 bis cover legitimate interest for web scraping.
Scraping Disputes and Rulings on Record, Including the YouTube Lawsuits
Eleven disputes and decisions show how each claim fares, and the three YouTube suits recorded here are anti-circumvention or copyright claims over how the audio, video or transcripts were taken, not computer-misuse claims. The table is ordered by the last dated step, and Reddit v. SerpApi and Google v. SerpApi concern an answer engine or search results, not a training ruling.
| Dispute | Court, number | Claims tested | Status as of 2026-11-09 | Source |
|---|---|---|---|---|
| Kneschke v. LAION | BGH I ZR 281/25 | copyright, TDM exceptions (§§ 44b, 60d UrhG), natural-language reservation; photograph copied to build LAION-5B | pronouncement scheduled 2026-12-17 | bundesgerichtshof.de |
| Sony v. Uncharted Labs (Udio) | S.D.N.Y. 1:24-cv-04777-AKH; 1:26-cv-06120-AKH | copyright; 1201 (YouTube rolling cipher) | Sony claims continue; conference 2026-12-11; UMG and Warner settled | courtlistener.com |
| In re OpenAI (Millette, YouTube transcripts) | MDL No. 3143; N.D. Cal. 5:24-cv-04710 | copyright over video transcripts | pending in the MDL, no separate ruling | jpml.uscourts.gov |
| Reddit v. SerpApi, Oxylabs, AWMProxy and Perplexity | S.D.N.Y. 1:25-cv-08736-PAE | 1201; state claims | 2026-07-31 motions predominantly denied (Dkt. 104); Oxylabs motion undecided as of 2026-11-06 | courtlistener.com |
| Google v. SerpApi | N.D. Cal. 4:25-cv-10826-YGR | copyright; anti-circumvention over search results | motion to dismiss amended complaint briefed; no ruling as of 2026-10-24 | courtlistener.com |
| UMG v. Suno | D. Mass. 1:24-cv-11611-FDS; 1:26-cv-14275 | copyright; 1201 (YouTube stream ripping) | discovery; second action 2026-09-18 (60,202 recordings); Warner settled | courtlistener.com |
| Reddit v. Anthropic | San Francisco Superior Court CGC-25-625892 (federal 3:25-cv-05643-TLT) | contract, interference, unfair competition | remanded 2026-03-30 (Dkt. 60); narrowed September 2026 (Anthropic S-1/A) | courtlistener.com; sec.gov |
| OPC joint investigation of OpenAI | PIPEDA Findings #2026-002 | consent for scraped personal information | concluded 2026-05-06, well-founded, conditionally resolved | priv.gc.ca |
| hiQ Labs v. LinkedIn | 9th Cir. No. 17-16783 | CFAA | preliminary injunction affirmed 2022-04-18; later steps not recorded | ca9.uscourts.gov |
| Van Buren v. United States | U.S. No. 19-783 | CFAA "exceeds authorized access" | decided 2021-06-03 | supremecourt.gov |
| Ryanair v PR Aviation | CJEU C-30/14 | contract limits on screen-scraping a database | decided 2015-01-15 | publications.europa.eu |
Collecting or Licensing Web Content: Benefits, Risks, Downsides and Alternatives, Including Not Licensing
A site owner can rely on these claims, license access instead, or do nothing, and each path has costs.
Benefits of licensing access
- A license replaces uncertain claims with agreed scope and payment: Reddit's S-1 of 2024-02-22 gives USD 203.0 million as the aggregate contract value of January 2024 licenses, with the counterparties not named.
- Google and Reddit announced Data API access to "display, train on, and otherwise use" Reddit content on 2024-02-22, value not disclosed.
- The developer gets documented provenance for the collected content.
Risks
- Claims are decided case by case: hiQ left several claims open, and Reddit v. SerpApi was dismissed in part.
- An EU reservation that is not machine-readable may fail to take effect.
- Public pages give a site owner no computer-misuse claim in the United States.
Downsides
- Litigation and monitoring cost time and money for the holder.
- Terms of use bind only where the visitor accepted them, a point this page does not resolve.
Alternatives, including not licensing
- Block or reserve content with crawler rules.
- License through an API or a content licensing platform.
- Join collective licensing routes.
- Do nothing, a valid choice.
The first alternative is covered in AI crawlers and how to block or charge them.
Beyond Scraping: Jurisdiction Pages, Crawlers and Licensed Sources
Each jurisdiction, collection route and related rule has its own page.
- AI training data law by jurisdiction: the parent hub for every jurisdiction and topic.
- United States AI training data law: federal and state rules beyond the CFAA.
- Fair use and AI training in US law: the US copyright defense in depth.
- EU rules for AI training data: the other EU instruments.
- Japan rules for licensing data for AI training: Article 30-4 and its proviso.
- How AI companies get training data: scraping among the other routes.
- Selling video for AI training: the licensing route for video holders.
- Licensed vs scraped training data: rights, cost and risk compared.
- robots.txt for AI crawlers: rules and limits of robots.txt.
- Machine-readable TDM reservations: reservation formats.
- Biometric data in AI training: scraped faces and voices.
- Trade secrets in AI data licensing: confidential data behind a login.
What This Page Does Not Cover
This page covers collecting content to train AI models, not how to build or run a scraper. Scraping for marketing lists, people search and price monitoring, and reviews of scraping tools or proxies, are outside the reference.
How This Page Is Sourced and Dated
Every rule cites the statute, regulator document or docket entry, with the date it was checked. The approach is described in how we source, date and check every figure. This page is general information about the law on scraping, not legal advice for any specific collection or deal.
Frequently Asked Questions
Is it legal to scrape data for AI?
There is no single answer. Public pages escape the CFAA, but contract, 17 U.S.C. 1201, copyright and data protection claims still apply, as Reddit v. Anthropic (remanded 2026-03-30) and Reddit v. SerpApi (2026-07-31) show.
Is scraping YouTube videos for AI training legal?
No court has decided it. The recorded suits allege copyright infringement and circumvention of YouTube's protections, through stream ripping (Suno) and a rolling cipher (Udio), and none is decided as of 2026-11-09.
Does ignoring robots.txt make scraping illegal?
Not by itself. It matters only through EU Art. 4(3), the Japanese reading of the Art. 30-4 proviso, or a contract that the scraper is bound by.
Can a website's terms of use ban AI scraping?
They can found a contract claim, as the Reddit v. Anthropic remand shows. In the EU Art. 4 can be overridden by contract, while Singapore voids a term that excludes the permitted use.