DataPacts

PerplexityBot and Perplexity-User: Search Crawling, robots.txt and the Stealth-Crawling Dispute

Checked

At a glance

PerplexityBot is the crawler Perplexity uses to surface and link websites in its search results, and Perplexity states that it "is not used to crawl content for AI foundation models"; Perplexity-User fetches pages when a user asks a question and, by Perplexity's account, "generally ignores robots.txt rules" (checked 2026-11-16).

Independent reference. Every buyer is covered on the same terms, every figure carries its source and check date, and no buyer pays to be listed or placed higher.

Request an independent introduction

Tell us what data your company holds. We reply with the buyers that fit and what their terms leave out. Your details reach a buyer only after you agree to a named introduction.

IndependentYour details stay with us until you agreeNot selling is a valid outcome

These crawler controls sit beside AI training data licensing and acquisition, and this page recommends no rule.

On 2025-08-04, Cloudflare published a post stating that Perplexity used an undeclared crawler when blocked, and Cloudflare de-listed Perplexity as a verified bot. Perplexity published a reply that could not be read on 2026-11-16, so the two sides are kept apart and dated throughout.

Every figure and statement carries its stating party and check date, and how we source, date and check every figure sets out the method.

What PerplexityBot and Perplexity-User do, in Perplexity's words

Perplexity states that it collects web data with crawlers and user agents "operating either automatically or in response to user requests", PerplexityBot being the automatic search crawler and Perplexity-User the fetcher that acts on a user's question (checked 2026-11-16).

Perplexity adds that "Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity".

The list gives each agent's stated purpose and training statement in Perplexity's words.

  • PerplexityBot: "designed to surface and link websites in search results on Perplexity", and "not used to crawl content for AI foundation models".
  • Perplexity-User: "supports user actions within Perplexity"; it "might visit a web page to help provide an accurate answer and include a link to the page in its response", and "is not used for web crawling or to collect content for training AI foundation models".

Perplexity's page says nothing about training models on content obtained by other means and names no model, so this page says neither. The section hub on AI crawlers and how to allow, block or charge them groups PerplexityBot with search crawlers, not training crawlers.

Perplexity's two user agents as Perplexity documents them

Perplexity documents two user agents on its crawler page, each with an example string and its own published IP address file in JSON. The table below compares them in Perplexity's order as of 2026-11-16, and the page is titled "Perplexity Crawlers".

User agent Purpose (Perplexity's words) Example user-agent string IP file (prefixes on 2026-11-16) robots.txt (Perplexity's statement) Training (Perplexity's statement) Checked Source
PerplexityBot "designed to surface and link websites in search results on Perplexity" Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) perplexity.ai/perplexitybot.json (13; creationTime 2026-10-28) Perplexity recommends allowing it "to ensure your site appears in search results" "not used to crawl content for AI foundation models" 2026-11-09, re-checked 2026-11-16 docs.perplexity.ai
Perplexity-User "supports user actions within Perplexity"; may visit a page "to help provide an accurate answer" Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) perplexity.ai/perplexity-user.json (10; creationTime 2026-10-28) "generally ignores robots.txt rules" "not used for web crawling or to collect content for training AI foundation models" 2026-11-09, re-checked 2026-11-16 docs.perplexity.ai

Reading key:

  • Both strings are labeled "Example user-agent string (the version number may change)" by Perplexity.
  • The IP files are the source to match against, not the string alone.
  • The page names no other Perplexity agent, and this is not proof that none exists.

robots.txt lines for PerplexityBot and Perplexity-User

Perplexity presents robots.txt as the control for its crawlers, and each of its two tokens is a separate setting. Perplexity writes that "Each setting works independently, and it may take up to 24 hours for our systems to reflect changes" (checked 2026-11-16).

The table below compares four whole-site goals as of 2026-11-16, each token in its own User-agent group, with what Perplexity's statements say the lines do and what stays open. Perplexity recommends allowing PerplexityBot "to ensure your site appears in search results", and says Perplexity-User "generally ignores robots.txt rules".

Goal Lines What it does (Perplexity's statement) What stays open Source
Stay in Perplexity search User-agent: PerplexityBot then Allow: / PerplexityBot may crawl; Perplexity recommends this "to ensure your site appears in search results" Perplexity-User fetches docs.perplexity.ai
Leave Perplexity search User-agent: PerplexityBot then Disallow: / asks PerplexityBot to stop; up to 24 hours to take effect Perplexity-User fetches; undeclared crawling Cloudflare reports docs.perplexity.ai; blog.cloudflare.com
Ask user-requested fetches to stop User-agent: Perplexity-User then Disallow: / a rule Perplexity says the fetcher "generally ignores" depends on Perplexity; a firewall rule on the published ranges docs.perplexity.ai
Both the two groups above independent settings per token as above docs.perplexity.ai
User-agent: PerplexityBot
Disallow: /

Perplexity's page states no crawl-delay, cache or unreachable-file rule and no page-level meta tag, so none is written here, and robots.txt is not legally binding by itself under RFC 9309 (2022). Syntax, precedence and other operators' tokens are covered on the page about robots.txt for AI crawlers.

Checking that a visitor really is PerplexityBot: strings, published IP ranges and firewall rules

A request is from PerplexityBot only if its user agent carries the PerplexityBot token and its source IP address falls in the ranges Perplexity publishes at perplexity.ai/perplexitybot.json, and Perplexity-User has its own file at perplexity.ai/perplexity-user.json.

On 2026-11-16 the two files held 13 and 10 prefixes, both with creationTime 2026-10-28, and the counts are a dated reading, not a permanent figure.

A string alone proves nothing, because any client can send it. Perplexity tells site owners to "combine both" string matching and IP address verification, and calls the JSON endpoints the source of truth. A check takes three steps:

  1. Match the token in the user-agent string, PerplexityBot or Perplexity-User, against the example Perplexity prints.
  2. Match the source IP address against the JSON file for that agent, since the two files are separate.
  3. Keep the ranges current by reading the endpoints again, as the files carry a creation time.

Perplexity's firewall steps are written to let its bots in: a Cloudflare WAF custom rule per user agent with its own IP ranges, action Skip, placed above blocking rules, and AWS WAF IP sets combined with an AND rule on the User-Agent header.

Perplexity adds that Cloudflare's Bot Fight Mode "can't be skipped with WAF custom rules", and that WAF changes "may take some time to propagate". The same IP files serve a site that wants to verify or block, and Perplexity publishes no reverse-DNS method or Web Bot Auth signature on the page, so none is described.

The stealth-crawling dispute as Cloudflare's post of August 4, 2025 states it

On August 4, 2025, Cloudflare published a post stating that Cloudflare had de-listed Perplexity as a verified bot and that Perplexity, when blocked, used an undeclared crawler with a generic Chrome-on-macOS user agent and IP addresses outside its published ranges.

The post is titled "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives", and the table below compares Cloudflare's statements with Perplexity's documented position as of 2026-11-16.

Point Cloudflare's post (2025-08-04) Perplexity's documentation (checked 2026-11-16) Source
Declared agent and volume Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user), "20-25m daily requests" example string without "AppleWebKit/537.36 (KHTML, like Gecko;"; "not used for web crawling" blog.cloudflare.com; docs.perplexity.ai
Undeclared agent and volume Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36, "3-6m daily requests" no statement blog.cloudflare.com
IP addresses "multiple IPs not listed in Perplexity's official IP range", rotated; requests from different ASNs publishes two IP files and calls them the source of truth blog.cloudflare.com; docs.perplexity.ai
robots.txt declared and undeclared crawlers accessed content "contrary to the web crawling norms as outlined in RFC 9309" PerplexityBot controlled by robots.txt; Perplexity-User "generally ignores robots.txt rules" blog.cloudflare.com; docs.perplexity.ai
Test method new unindexed domains, robots.txt disallowing all, WAF blocks on both declared agents n/a blog.cloudflare.com
When blocked answers drew on "other data sources", "less specific" n/a blog.cloudflare.com
Cloudflare's actions de-listed Perplexity as a verified bot; heuristics added to managed rules; signature in the managed AI-crawler block rule, "available to all customers, including our free customers" reply post unreadable (HTTP 403, 2026-11-16) blog.cloudflare.com; perplexity.ai

Cloudflare states that, after customer complaints, it tested "multiple brand-new domains" that were not indexed, with a robots.txt stopping all respectful bots and WAF rules blocking PerplexityBot and Perplexity-User, and that Perplexity "was still providing detailed information regarding the exact content hosted on each of these restricted domains".

As a control, Cloudflare states that ChatGPT-User fetched the robots file and stopped crawling when it was disallowed. Cloudflare also states that the undeclared crawler was seen "across tens of thousands of domains and millions of requests per day".

Every figure is Cloudflare's measurement on its own network, not a court finding. Cloudflare's post does not say whether PerplexityBot is in the declared count, and it names no agent for the undeclared crawler, so this page does not say PerplexityBot itself evaded a rule.

The declared string Cloudflare recorded differs from Perplexity's documented example, and both are shown without a conclusion. Cloudflare separately reports a crawl-to-refer ratio of 118:1 for Perplexity for 2025-08-01 to 2025-08-07, and its own blocking products are set out on the page about Cloudflare's AI Crawl Control and pay per crawl terms.

Perplexity's published reply

Perplexity answered Cloudflare in a post on its own blog, but the page returned HTTP 403 when checked on November 16, 2026, so this page does not summarize Perplexity's arguments. The post sits at perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web, and Perplexity disputes Cloudflare's account.

Perplexity's crawler documentation, re-read on 2026-11-16, does not address undeclared crawling. Its date and content stay unverified here.

What a PerplexityBot disallow does and does not change

A PerplexityBot disallow asks Perplexity's search crawler to stay away, which by Perplexity's account takes a site out of what PerplexityBot surfaces in Perplexity search, and it does not bind Perplexity-User, which in Perplexity's own words "generally ignores robots.txt rules".

Perplexity states that neither agent is used to collect content for AI foundation model training, so as Perplexity describes it the disallow is not a training opt-out, and there is nothing to opt out of on training in its account (checked 2026-11-16).

A PerplexityBot disallow leaves four things open:

  • Perplexity-User fetches on a user's request.
  • Crawling that Cloudflare reports from undeclared agents.
  • The delay of up to 24 hours Perplexity gives for a setting to take effect.
  • Content Perplexity already holds or obtains from other sources.

Cloudflare states that when the stealth crawler was blocked, Perplexity's answers drew on "other data sources" and were "less specific" and "lacked details from the original content". Perplexity makes no recorded statement on deleting content it collected earlier, so none is described. Licensing agreements a site has signed with anyone are untouched by the signal.

Perplexity's crawling before US courts and under EU rules

The suits that touch how Perplexity obtains content are pending in the Southern District of New York, and no final judgment on Perplexity's crawling is recorded here as of 2026-11-16. Neither suit is recorded here as a training-data case.

Reddit, Inc. v. SerpApi LLC, Oxylabs UAB, AWMProxy and Perplexity AI, Inc., S.D.N.Y. 1:25-cv-08736-PAE, was filed on 2025-10-22 with DMCA section 1201 and related claims over scraping Reddit content from Google search results.

The motions to dismiss of SerpApi and Perplexity were predominantly denied on 2026-07-31 (Dkt. 104), and the case is pending as of 2026-11-06.

Publishers' copyright suits over Perplexity's answers include Dow Jones & Company, Inc. and the publisher of the New York Post v. Perplexity AI, Inc., S.D.N.Y. 1:24-cv-07984-KPF, filed on 2024-10-21, which is in discovery after an order of 2026-10-29. That suit and the other publishers' suits are linked in the border below.

Perplexity is not among the 22 signatories listed on the European Commission's page for the voluntary GPAI Code of Practice (page updated 2026-10-07, checked 2026-11-09).

Measure 1.3 of its Copyright chapter, published 2025-07-10, commits signatories to crawlers that read and follow robots.txt as specified in RFC 9309, so that is not a commitment Perplexity has signed on that list.

DSM Directive Article 4(3), Directive (EU) 2019/790 (in force 2019-06-06), allows the text and data mining exception only where use has not been reserved "in an appropriate manner, such as machine-readable means in the case of content made publicly available online".

No ruling on whether it applies to a search crawler that is not stated to train models is recorded here, and the page on the EU text and data mining exception and its opt-out sets out the exception. No US statute or ruling recorded here makes robots.txt binding by itself.

Blocking PerplexityBot or Perplexity-User: benefits, risks, downsides and alternatives

Disallowing PerplexityBot costs nothing and Perplexity says what the token controls, but it removes a site from what PerplexityBot surfaces in Perplexity search and, by Perplexity's own account, does not reach Perplexity-User. Crawler rules and licensing are separate decisions.

Licensing is covered in the profile of Perplexity as a data buyer, and no term is repeated here.

Benefits:

  • The rule is free and takes effect within up to 24 hours by Perplexity's statement.
  • Perplexity states in its own words what the PerplexityBot token controls.
  • The published IP files allow a firewall rule beyond robots.txt.

Risks:

  • Perplexity-User generally ignores robots.txt, by Perplexity's statement.
  • Cloudflare reports undeclared crawling from addresses outside the published ranges.
  • Spoofed strings pass as Perplexity's bots without an IP check.

Downsides:

  • A PerplexityBot disallow means loss of PerplexityBot search visibility and links, which Perplexity recommends allowing for.
  • A disallow brings no payment.
  • By Perplexity's account neither agent is stated to train, so the rule is not a training opt-out.

Alternatives:

  1. Allow both agents.
  2. Disallow PerplexityBot only.
  3. Add a firewall rule on the published IP ranges, or a network-level block such as the managed rule Cloudflare describes, named here as Cloudflare's statement and not recommended.
  4. License content directly, charge for crawling through intermediaries, or decide on whether to sell or license your data to AI companies at all, where not licensing is a valid outcome.

Blocking, allowing, firewalling and licensing can be combined, and none is recommended here.

Other AI crawlers, opt-out signals and the cases around Perplexity's crawling

PerplexityBot is one crawler among many, and the pages below cover the other crawlers, other ways to signal an opt-out and the cases around Perplexity.

This page is information, not legal advice.

Frequently asked questions

Is PerplexityBot used to train AI models?

Not by Perplexity's account: Perplexity states that PerplexityBot "is not used to crawl content for AI foundation models" and that Perplexity-User is not used "to collect content for training AI foundation models" (checked 2026-11-09 and 2026-11-16). No independent check of this is on record.

Does Perplexity respect robots.txt?

Perplexity presents robots.txt as the control for PerplexityBot, but says Perplexity-User "generally ignores robots.txt rules" because a user requested the fetch. Cloudflare reported on 2025-08-04 that undeclared Perplexity crawling continued on sites that disallowed it.

What is the PerplexityBot user agent string?

Perplexity gives Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) as an example string. It notes that the version number may change (checked 2026-11-16).

What is the Perplexity stealth-crawling controversy?

Cloudflare reported on 2025-08-04 that it de-listed Perplexity as a verified bot and that Perplexity, when blocked, crawled with a generic Chrome user agent from unpublished IP addresses at 3 to 6 million requests a day. Perplexity published a reply that could not be read on 2026-11-16.

How long does a robots.txt change take to reach Perplexity?

Perplexity states that each setting works independently and that "it may take up to 24 hours for our systems to reflect changes". It adds that firewall changes "may take some time to propagate" (checked 2026-11-16).