DataPacts

Cloudflare Pay per Crawl, AI Crawl Control and Human Native: Program Terms for Publishers

Checked

At a glance

Cloudflare, Inc. is a US network provider that lets site owners see, block or charge AI crawlers, and under pay per crawl it bills the crawler as Merchant of Record and passes the earnings to the site owner, so it does not buy the content itself.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

Cloudflare sits between crawler operator and site owner: the owner allows, charges (HTTP 402) or blocks each crawler and sets the price per zone; the crawler pays Cloudflare as Merchant of Record, which pays the owner monthly.

Cloudflare states that pay per crawl is a closed beta, that the site owner sets the price and that payouts are monthly (checked 2026-11-10).

This page is part of an independent reference on AI training data licensing. Every term below is Cloudflare's own statement with its check date.

Each figure on this page is dated and traced to a primary page, as set out in how DataPacts sources and dates every figure.

What Cloudflare Offers Site Owners for AI Crawlers

Cloudflare offers site owners five things for AI crawlers: crawler defaults and per-purpose controls, AI Crawl Control, pay per crawl, the Content Signals Policy and, since its acquisition, Human Native.

  • Defaults and controls: block, allow or restrict crawlers by purpose.
  • AI Crawl Control: the dashboard for AI traffic.
  • Pay per crawl: charging crawlers per request, in closed beta.
  • Content Signals Policy: usage preferences added to robots.txt.
  • Human Native: a UK-based AI data marketplace Cloudflare acquired.

Site owners on Cloudflare's network are the users, and crawler operators are the paying side where a site charges; Cloudflare sits in between.

Cloudflare states that it manages and protects traffic for 20% of the web (press release 2025-07-01). Cloudflare is not among those who buy AI training data, and the wider directory of who buys AI training data keeps that distinction.

Cloudflare Company Identity

Cloudflare, Inc. is the legal name on record, with its website at cloudflare.com and its home country the United States. The table lists the identity attributes as of 2026-11-10.

Attribute Value Source
Brand Cloudflare cloudflare.com
Legal name Cloudflare, Inc. cloudflare.com; blog.cloudflare.com
Home country United States cloudflare.com
Registry number none found on primary pages checked 2026-11-09 cloudflare.com
Founded not on record here n/a
Type network provider; platform, intermediary (DataPacts classification) DataPacts
Programs on this page pay per crawl (closed beta); AI Crawl Control; Content Signals Policy; Human Native (acquired, announced 2026-01-15) developers.cloudflare.com; blog.cloudflare.com
Website https://www.cloudflare.com cloudflare.com

No registry number and no founding year appear on the primary pages checked. DataPacts classifies Cloudflare as a platform, an intermediary between site owners and crawler operators.

Who Pays Whom Under Cloudflare Pay per Crawl

Under pay per crawl the crawler operator pays Cloudflare, which acts as Merchant of Record, and Cloudflare pays the site owner, so the money passes through Cloudflare without Cloudflare taking the content.

Cloudflare's launch post says it "aggregates all the events, charges the crawler, and distributes the earnings to the publisher" (checked 2026-11-10). The site owner chooses to allow, charge or block each crawler.

Cloudflare's documentation states that payouts are monthly, to publishers in good standing, through a dedicated Cloudflare Stripe Connect account (page of 2026-04-23, checked 2026-11-10).

It names "settlement periods and minimum payout thresholds" but gives no amount, and no Cloudflare fee is stated. No crawler operator is on record as paying through pay per crawl, and this page names none.

The HTTP 402 exchange and the comparison with TollBit and RSL sit under pay-per-crawl and crawler licensing, not here.

Cloudflare Pay per Crawl Terms, Dated

Cloudflare publishes no price of its own for pay per crawl: the site owner sets the price, and the table lists each term Cloudflare states with the date it was checked. Values are as of 2026-11-10, and "not stated" means absent from Cloudflare's pages on the check date.

Term Cloudflare's stated value Page date Checked Source
Status "currently in closed beta" (pay-per-crawl page); "(private beta)" (AI Crawl Control overview) 2026-07-28; 2026-08-14 2026-11-09; re-checked 2026-11-10 developers.cloudflare.com
Launch private beta, 2025-07-01 post 2025-07-01 (modified 2026-07-15) 2026-11-10 blog.cloudflare.com/introducing-pay-per-crawl
Who sets the price site owner, "by setting a price per zone" (docs); "a flat, per-request price across their entire site" (launch post) 2026-07-28; 2025-07-01 2026-11-10 developers.cloudflare.com; blog.cloudflare.com
Options per crawler allow, charge or block; charged requests answered with HTTP 402 2026-07-28 2026-11-09 developers.cloudflare.com
Billing role Cloudflare is Merchant of Record 2026-07-28 2026-11-09 developers.cloudflare.com
How to join pay per crawl signup; Enterprise customers via account executive 2026-07-28 2026-11-09 developers.cloudflare.com
Eligibility closed beta participants 2026-07-28 2026-11-09 developers.cloudflare.com
Payouts monthly, to publishers in good standing, via a Cloudflare Stripe Connect account 2026-04-23 2026-11-09; re-checked 2026-11-10 developers.cloudflare.com (manage payouts)
Minimum payout threshold "minimum payout thresholds", amount not published 2026-04-23 2026-11-10 developers.cloudflare.com (manage payouts)
Cloudflare's fees not stated 2026-07-28; 2026-04-23 2026-11-10 developers.cloudflare.com
Referral program none on record n/a 2026-11-09 n/a

Each value is Cloudflare's own statement on the date shown. Where two Cloudflare pages word a term differently, both are shown.

Cloudflare's own pages word two terms differently. The pay-per-crawl page, updated 2026-07-28, says the program is "currently in closed beta", while the AI Crawl Control overview, updated 2026-08-14, lists pay per crawl as "(private beta)"; both mean it is not generally available.

The launch post describes "a flat, per-request price across their entire site", while the documentation says the owner sets "a price per zone".

The terms leave out what a site owner most wants to know: the price level, Cloudflare's fee and the payout threshold. DataPacts states no figure for any of them, and no referral program is on record, so there is no referral fee to disclose.

How these terms differ from other platforms is laid out in the program terms compared table.

What Cloudflare Lists Under AI Crawl Control

AI Crawl Control, formerly AI Audit, is the Cloudflare dashboard area that shows which AI services access a site and lets the owner manage them, and Cloudflare states it is available on all plans (docs updated 2026-08-14, checked 2026-11-10).

Cloudflare lists four features:

  • Manage AI crawlers.
  • Analyze AI traffic.
  • Track robots.txt: health and the crawlers that violate directives.
  • Pay per crawl, listed as "(private beta)".

The robots.txt tracking reports violations; it does not enforce robots.txt by itself. What a site can and cannot stop with a text file is covered under robots.txt for AI crawlers.

Cloudflare's AI Crawler Defaults Since 2024, Dated

Cloudflare moved from an optional one-click block in September 2024 to asking every new domain about AI crawlers on 2025-07-01 and to three independent controls on 2026-09-15. The table lists the announcements in Cloudflare's own words, with figures that are Cloudflare's measurements of its own network, as of 2026-11-10.

Date What Cloudflare announced Source
September 2024 One-click option to block AI crawlers; more than one million customers had used it by 2025-07-01 Press release 2025-07-01
2025-07-01 Blocks AI crawlers "accessing content without permission or compensation, by default"; every new domain asked whether to allow AI crawlers; crawler operators can state their purpose as training, inference or search Press release 2025-07-01
2025-07-01 Pay per crawl in private beta blog.cloudflare.com
2025-09-24 Content Signals Policy, CC0 blog.cloudflare.com
2026-01-15 Human Native joins Cloudflare, price not disclosed blog.cloudflare.com
2026-09-15 Three independent controls for search, AI training and AI agents replace Block AI Bots; Bot Preference Sync replaces Managed Robots.txt; default settings depend on whether a site carries ads; 17% of sites restrict AI training and fewer than 1% block search crawlers; mixed-use crawlers are 36.6% of verified crawler traffic Press release 2026-09-15

Under the 2026-09-15 settings Cloudflare proposes, a site that carries ads has search allowed, AI training disallowed and AI agents blocked on ad-carrying pages, while a site without ads has all three allowed; these are suggested defaults, not fixed ones.

Which bots collect training data, and how to allow, block or charge them in general, sit under AI crawlers and how to allow, block or charge them.

The Content Signals Policy: What Cloudflare's Signals Say and Do Not Do

The Content Signals Policy, published by Cloudflare on 2025-09-24 under a CC0 license, adds lines to robots.txt that state how content may be used after it has been accessed. Cloudflare defines three signals:

  • search: "building a search index and providing search results".
  • ai-input: "inputting content into one or more AI models".
  • ai-train: "training or fine-tuning AI models".

Cloudflare states that its managed robots.txt was on for over 3.8 million domains and adds Content-Signal: search=yes, ai-train=no for them, with no ai-input signal (checked 2026-11-10). In Cloudflare's own words the signals "express preferences; they are not technical countermeasures", and "Some companies might simply ignore them".

Cloudflare calls the restrictions "express reservations of rights" but does not state their legal effect, and no AI company's adoption of the signals is on record. Directive (EU) 2019/790, Article 4(3), requires a reservation for online content to be machine-readable, as set out under the EU text and data mining exception and its opt-out.

The policy sits beside the 2026-09-15 controls, so a site owner checks which one its robots.txt now carries. This page is information, not legal advice.

Human Native as Part of Cloudflare

Cloudflare announced on 2026-01-15 that it had acquired Human Native, which it calls a UK-based AI data marketplace, without disclosing a price. The post was last modified on 2026-07-15 (checked 2026-11-10).

Cloudflare's announcement names AI Crawl Control and pay per crawl beside Human Native but does not say how Human Native connects to them, and no marketplace terms are on record here. The company record, Human Native's marketplace status and history are covered on that profile.

For a holder of content, an approach that cites Human Native goes to Cloudflare in writing, and the terms are asked for before any data moves.

Cloudflare's Record: Disputes, Measurements and Open Questions

No lawsuit, regulator action or complaint about Cloudflare's pay per crawl, AI Crawl Control or Content Signals Policy is on record in the primary sources DataPacts checked as of 2026-11-10.

The public dispute on record on this topic is Cloudflare's own report of 2025-08-04, in which it de-listed Perplexity from its verified bots and measured the declared crawler (Perplexity-User/1.0) at 20 to 25 million requests a day against 3 to 6 million undeclared.

These are Cloudflare's measurements of its own network, not a census, and Cloudflare's crawl figures are read that way throughout this page.

Open questions a publisher can put to Cloudflare are which crawler operators pay, the fee, the threshold amount and which countries Stripe Connect payouts cover. The full account sits under PerplexityBot and the stealth-crawling dispute.

Benefits, Risks, Downsides and Alternatives to Charging Crawlers Through Cloudflare

Cloudflare gives a site on its network one place to see, block or charge AI crawlers, but pay per crawl is still a beta with unpublished fees and no paying crawler on record.

Benefits

  • The site owner sets the price and chooses allow, charge or block per crawler.
  • Cloudflare handles billing as Merchant of Record and pays monthly.
  • AI Crawl Control is listed as available on all plans.

Risks

  • Signals and charges reach only crawlers that honor them; Cloudflare itself says some companies might ignore content signals.
  • Beta terms can change, as the wording differences between its pages show.
  • Defaults set by Cloudflare can change a site's crawler access unless the owner checks them.

Downsides

  • Fees and the minimum payout amount are not published.
  • Pay per crawl is open only to sites on Cloudflare's network and to closed beta participants.
  • No crawler operator is on record as paying.

Alternatives

  • A content licensing platform such as TollBit or the RSL Collective.
  • A direct license with an AI company.
  • robots.txt and other opt-outs without charging.
  • Not licensing at all, with other alternatives to selling data to AI labs.

Cloudflare Next to TollBit, the RSL Collective and Other Publisher Routes

Cloudflare charges at the network edge, while other routes license through a publisher platform or a machine-readable license, and each route differs in who signs and who pays. The parent page places Cloudflare among content licensing platforms for publishers on one dated table.

More About Cloudflare's AI Crawler Programs

Frequently Asked Questions About Cloudflare Pay per Crawl

Does Cloudflare buy content for AI training?

No: under pay per crawl Cloudflare acts as Merchant of Record, charges the crawler and passes the earnings to the site owner, and no Cloudflare page describes it buying content. The licensee side is the crawler operator.

How do site owners join Cloudflare pay per crawl?

Through Cloudflare's pay per crawl signup, or through their account executive for existing Enterprise customers. The program is a closed beta per documentation updated 2026-07-28.

How and when does Cloudflare pay site owners?

Monthly, to publishers in good standing, through a dedicated Cloudflare Stripe Connect account, subject to settlement periods and minimum payout thresholds whose amount is not published (docs 2026-04-23, checked 2026-11-10). Dated changes to these terms are logged under dated changes to payouts and terms.