DataPacts

Pay-per-Crawl and Crawler Licensing: Cloudflare, TollBit and RSL

Checked

At a glance

Pay per crawl is a way for a site owner to charge AI crawlers for each request instead of only allowing or blocking them, and crawler licensing extends the idea to written license terms a crawler can read.

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

Three lanes from an AI crawler to the site owner: Cloudflare network metering with HTTP 402, the TollBit bot paywall with direct licenses, and an RSL License line with the Collective; each shows signal, price setter and contract.

It is one route in AI training data licensing for web content, next to direct deals and collective licenses.

This page compares three routes: Cloudflare's pay per crawl, TollBit's publisher platform and the RSL Collective's licenses on the RSL standard. Each row rests on the intermediary's own statement with a check date.

Cloudflare describes pay per crawl as a closed beta (documentation updated 2026-07-28), none of the three publishes a per-request price, and none is recommended here.

Every status and term below is the intermediary's own dated claim, and how we source, date and check every figure sets out the method.

What pay per crawl and crawler licensing mean

Charging crawlers turns access to web content into a metered license: the crawler operator pays per request or per use, and an intermediary meters, bills or declares the terms. The intermediary is not the licensee and does not take the data for itself.

The routes split into three kinds, and the hub page on AI crawlers and how to allow, block or charge them covers the choice between allowing, blocking and charging, plus the bot list, so this page does not repeat it.

  • Network metering: the network answers a crawler with a price before serving the page.
  • Publisher platform: a platform meters bots and the publisher licenses directly to the AI developer.
  • License declaration: a file states terms and a collective collects royalties.

Three routes for charging crawlers, compared on dated terms

The three routes differ in who sets the price, who signs the license and how the crawler learns the terms, and each row below is the intermediary's own statement with its check date.

The table lists three routes alphabetically as of 2026-11-09 unless the Checked cell says otherwise.

Route Operator (role) Status Who sets the price Who is party to the license Signal to the crawler Payout Not published Checked Source
Pay per crawl Cloudflare, Inc. (network operator, Merchant of Record) closed beta (docs updated 2026-07-28); launched in private beta 2025-07-01 site owner sets a price per zone, charged each time an AI crawler accesses a page (docs, checked 2026-11-16) Cloudflare acts as Merchant of Record HTTP 402 response with the price; allow, charge or block per crawler monthly, through Stripe Connect (docs 2026-04-23) fees; minimum payout threshold amount 2026-11-09 developers.cloudflare.com/ai-crawl-control
RSL Collective licenses (on the RSL standard) RSL Collective (nonprofit rights organization by its own description) open standard, RSL 1.0 specification finalized; licenses open to join ("Join for Free") the Collective; rates not published publisher opts in to the Collective's license License: line in robots.txt or an RSL file pointing to royalty.xml or attribution.xml royalties "whenever an AI application uses your content to generate a result" (paid license) or attribution links rates; distribution terms; exclusivity, term and deletion 2026-11-09 rslcollective.org/license; rslstandard.org
TollBit publisher platform Novoscribe, Inc. (publisher-side platform) operating; "Free for publishers" publisher, in the TollBit portal publisher and AI developer directly; TollBit not a party (agreement 2025-10-29) bot paywall payout timing not in the agreement payout timing 2026-11-09 tollbit.com/solutions; tollbit.com/legal/publisher-platform-agreement
  • "Not published" means absent from the intermediary's pages on the check date.
  • "Party to the license" comes from the intermediary's own terms.
  • No column ranks the routes, and none of them is a buyer of the content.

Charging at the network edge: Cloudflare's HTTP 402 model

Cloudflare's pay per crawl lets a site on its network allow, charge or block each AI crawler, answering a charged request with HTTP 402 and a price the site owner sets.

Cloudflare launched it on 2025-07-01 in private beta, describing "a flat, per-request price across their entire site" in its launch post; its documentation (checked 2026-11-16) has the owner set "a price per zone". Cloudflare acts as Merchant of Record.

Cloudflare's documentation, updated 2026-07-28, still describes pay per crawl as a closed beta. Sites join through a signup, and Enterprise customers go through their account executive. Payouts are monthly according to the payouts page of 2026-04-23, which names Stripe Connect.

Cloudflare's fees and the amount of its minimum payout threshold were not published on 2026-11-09. Crawler operators have their own documentation on the paying side, and its steps are not described here.

Cloudflare states that it has blocked AI crawlers by default for new domains since 2025-07-01, and on 2026-09-15 it announced separate Search, AI training and AI agents controls to replace "Block AI Bots".

It announced the acquisition of Human Native on 2026-01-15 without disclosing a price. For the program detail see Cloudflare pay per crawl and AI Crawl Control.

Charging through a publisher platform: TollBit's bot paywall and direct licenses

TollBit, operated by Novoscribe, Inc., meters AI bots for publishers and lets them set their own rates, but the license itself runs directly between the publisher and the AI developer.

TollBit describes its products as analytics, a bot paywall, licensed RAG access and content controls, and states "Free for publishers". Publishers set rates for their data in the TollBit portal, and the agreement does not give a payout timing.

TollBit's Publisher Platform Agreement, last updated 2025-10-29, says TollBit is not a party to Content License Agreements. The publisher warrants its rights and indemnifies TollBit, and the agreement is governed by New York law with JAMS arbitration.

A separate Developer Platform Agreement of 2025-02-06 covers the AI side. A rights problem in the content therefore sits with the publisher under these terms.

TollBit states that it has raised over USD 31 million (checked 2026-11-09). Its home page also claims, for the first half of 2026, 987 billion visits analyzed and over 2.6 billion bots sent to its paywall, next to a placeholder reading "Over 0 sites".

These are TollBit's claims, not independent measurements. See TollBit's profile for its terms in detail.

Declaring license terms in machine-readable form: RSL and the RSL Collective

RSL (Really Simple Licensing) is an open standard for stating AI licensing terms in machine-readable form, and the RSL Collective uses it to offer a paid and an attribution license for AI training.

The RSL 1.0 specification is finalized, and the standard states that "A license token is required even if free". It recognizes these license types:

  • free
  • attribution
  • subscription
  • usage-based
  • purchase
  • collective
  • custom

The Collective's paid license permits ai-train and pays royalties "whenever an AI application uses your content to generate a result", and it publishes no rates or distribution terms.

The attribution license permits ai-train and ai-input. Exclusivity, term and deletion are not stated. The Collective calls itself a "nonprofit rights organization" and offers "Join for Free", both its own claims checked 2026-11-09.

A publisher points to a license with a line such as License: https://rslcollective.org/royalty.xml or .../attribution.xml in robots.txt or an RSL file, with optional limits on what is permitted and prohibited. The line blocks nothing by itself. Where it sits among the other rules is covered under robots.txt for AI crawlers.

Usage-based licensing beyond crawling: grounding marketplaces, attribution and paid APIs

Several routes charge for AI use of content without metering crawls, and they matter here because a site can combine them with crawler charging. The table compares five of them alphabetically, each as of the date in its row, and each has its own owner page.

Route Operator What is licensed Payment basis (operator's statement) Training use Date Source
Agent storefront Dappier Inc. publisher-derived APIs sold to AI agents not recorded not recorded schema dateModified 2026-06-03 dappier.com; agents.dappier.com/llms.txt
AI data marketplace Human Native (HUMAN NATIVE LTD), acquired by Cloudflare rights holders' data for AI developers not recorded not recorded acquisition announced 2026-01-15, price not disclosed blog.cloudflare.com
Attribution and Gist ProrataAI, Inc. publisher content for AI answers "Payment depends on the relevant rights, licenses, and commercial agreements"; no split stated "never used for model training" 2026-11-09 prorata.ai
Publisher Content Marketplace (pilot) Microsoft Corporation publisher content for grounding publishers "paid on delivered value"; no rates published not mentioned announced 2026-02-03 about.ads.microsoft.com
Wikimedia Enterprise API Wikimedia, LLC Wikipedia and Wikimedia content via paid API pricing not disclosed not stated 2026-01-15 enterprise.wikimedia.com

The full set of such services is on the page for content licensing platforms for publishers.

Uptake and prices on record for charging crawlers

Public evidence of crawlers actually paying is thin: no intermediary publishes per-request prices, total payouts or the names of paying crawler operators. What is on record is limited to these items:

  • Reach plc's half-year report of 2026-07-22 lists TollBit pay-per-use among its revenue lines, as Reach states.
  • TollBit states that it sent over 2.6 billion bots to its paywall in the first half of 2026.
  • Cloudflare states, on 2026-09-15, that 17 % of sites on its network restrict AI training and fewer than 1 % block search crawlers.

Per-request figures quoted by third parties are not stated here, and prices per work, hour or token for licensed datasets sit elsewhere, among them the public price points under AI training data prices.

A price set by a site owner for a crawl is a list price, not a market price, because no source records a crawler operator that paid it.

Charging works best where the law lets a site refuse unlicensed use, and in the EU a machine-readable reservation takes online content out of the commercial text and data mining exception. robots.txt is a protocol under RFC 9309 and is not legally binding by itself, so a 402 response or a License line is a technical and contractual signal.

Article 4(3) of Directive (EU) 2019/790 (in force since 2019-06-06, transposition deadline 2021-06-07) lets rightsholders reserve use "in an appropriate manner, such as machine-readable means in the case of content made publicly available online".

Article 53(1)(c) of Regulation (EU) 2024/1689, which applies from 2025-08-02, requires providers of general-purpose AI models to keep a copyright policy that identifies and complies with such reservations.

The voluntary GPAI Code of Practice of 2025-07-10 adds, in Measure 1.3, that crawlers read and follow robots.txt and other appropriate machine-readable protocols. How the exception treats a reservation is set out under the EU text and data mining exception and its opt-out.

Contracts allocate the risk too. Under TollBit's agreement of 2025-10-29 the publisher warrants its rights and indemnifies TollBit, so the platform does not carry that risk. No US rule on charging crawlers is stated here.

Charging crawlers through an intermediary: benefits, risks, downsides and alternatives

Charging can earn something from crawling that blocking only stops, but every route depends on crawler operators agreeing to pay and on an intermediary's terms.

Benefits

  • The site owner sets the price or the terms: a price per zone on Cloudflare, rates in the TollBit portal, or the Collective's license.
  • A large network gives reach: Cloudflare states that it manages and protects traffic for 20 % of the web (press release of 2025-07-01).
  • Entry is free for publishers on TollBit ("Free for publishers") and on the RSL Collective ("Join for Free").

Risks

  • User-initiated fetchers and undeclared crawlers may not honor a charge: operators of ChatGPT-User, Perplexity-User, Meta-ExternalFetcher and Amzn-User say robots.txt may not fully apply to them (checked 2026-11-09), and Cloudflare measured 20 to 25 million requests per day from the declared crawler (Perplexity-User/1.0) against 3 to 6 million undeclared (2025-08-04).
  • Intermediary terms change, and Cloudflare's pay per crawl is still a closed beta.
  • Rights warranties and indemnities sit with the publisher under TollBit's agreement.

Downsides

  • Cloudflare's fees and minimum payout threshold are not published.
  • The RSL Collective publishes no rates.
  • Payment happens only if a crawler operator licenses, and no source names one that does.

Alternatives

  • Block training crawlers with robots.txt rules.
  • Sign a direct license with an AI company.
  • Join a collective license.
  • Use a usage-based grounding marketplace.
  • Do nothing and not license at all, which is a valid choice; the decision is covered under whether to sell or license data to AI companies at all.

Which route fits depends on the content, the crawlers that actually visit and the tolerance for intermediary terms, and no route is recommended here.

Charging is one way to handle crawlers; the pages below cover blocking, the bots themselves, the intermediaries' profiles and other licensing routes. This page is information, not legal advice.

Frequently asked questions

Is Cloudflare pay per crawl generally available?

No: Cloudflare's documentation, updated 2026-07-28, describes pay per crawl as a closed beta that sites join through a signup, with Enterprise customers going through their account executive. Cloudflare launched it in private beta on 2025-07-01.

How much does pay per crawl charge per request?

Cloudflare publishes no standard price: the site owner sets a price per zone (documentation, checked 2026-11-16), and Cloudflare's fees and minimum payout threshold were not published on 2026-11-09. The price is therefore each site's own choice.

What is the RSL standard?

RSL (Really Simple Licensing) is an open standard for machine-readable AI licensing terms, with license types from free and attribution to usage-based and collective, and a license token required even when the license is free. The RSL Collective offers licenses on it.

Does TollBit buy content from publishers?

No: TollBit's Publisher Platform Agreement of 2025-10-29 says TollBit is not a party to the content license agreements, which run directly between the publisher and the AI developer at rates the publisher sets. TollBit meters the bots.

Does charging crawlers replace robots.txt rules?

No: a 402 response or an RSL License line works only for crawlers that read and honor it, so robots.txt rules for crawlers a site does not want stay in place, and robots.txt is not legally binding by itself. User-initiated fetchers may ignore both.