Cloudflare lists Bytespider as an AI crawler and says it is "reportedly used to gather training data". Cloudflare measured it as the most active AI crawler on its network in the year to July 2024, and as a much smaller one by May 2025, when its share of AI-only crawling stood at 7.2 %.
No operator statement and no named measurement of its robots.txt behavior is on record, so this page states neither compliance nor non-compliance. Blocking works at robots.txt, at a firewall and in Cloudflare, and the page recommends no rule.
Every statement is attributed to the party that made it, with a date, and how we source, date and check every figure sets out the method.
What is on record about Bytespider, and what ByteDance has not published
Cloudflare names ByteDance as Bytespider's operator and lists it as an AI crawler, but no ByteDance page documenting the crawler, its purpose, its user-agent string or its IP addresses was found on 2026-11-09. The table below separates what Cloudflare states from what no source states, as of 2026-11-16, and "not on record" means none found on the check date, not proof of absence.
| Item | On record | Stated by | Checked | Source |
|---|---|---|---|---|
| Operator (brand) | ByteDance | Cloudflare (blog 2024-07-03; bot reference updated 2026-04-23) | 2026-11-16 | blog.cloudflare.com; developers.cloudflare.com/ai-crawl-control/reference/bots/ |
| Operating legal entity | not on record | none | 2026-11-09 | n/a |
| Purpose | "reportedly used to gather training data" (Cloudflare's wording); Cloudflare category "AI Crawler" | Cloudflare | 2026-11-16 | blog.cloudflare.com (2024-07-03); developers.cloudflare.com |
| Operator documentation | none found | none | 2026-11-09 | n/a |
| User agent token | Bytespider |
Cloudflare bot reference | 2026-11-16 | developers.cloudflare.com/ai-crawl-control/reference/bots/ |
| Full user-agent string | not published by the operator | none | 2026-11-09 | n/a |
| IP list or reverse DNS | not published by the operator | none | 2026-11-09 | n/a |
| robots.txt statement | none from the operator | none | 2026-11-09 | n/a |
| EU GPAI Code of Practice signatory | not listed among 22 signatories (list as of 2026-10-07) | European Commission | 2026-11-09 | digital-strategy.ec.europa.eu/en/policies/contents-code-gpai |
"ByteDance" here is the brand Cloudflare names, and the legal entity that runs the crawler is not identified on any primary page found. For operators that do document their bots, the section hub on AI crawlers and how to allow, block or charge them is the better starting point.
How much Bytespider crawls: Cloudflare's 2024 and 2025 measurements
Cloudflare measured Bytespider as the most active AI crawler on its network in the year to July 2024 and as a much smaller one by May 2025. The table below lists four Cloudflare measures, each as of its own period and kept with its population.
| Measure | Value | Period | Stated by | Source |
|---|---|---|---|---|
| Share of websites accessed by Bytespider (Cloudflare network) | 40.40 %, first of the AI bots; also first by request volume | year to July 2024 (post 2024-07-03) | Cloudflare | blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ |
| Share of AI-only crawler traffic | 42 % to 7.2 % | May 2024 to May 2025 (post 2025-07-01) | Cloudflare | blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/ |
| Share of a cohort of over 30 AI and search crawlers | 22.8 % to 2.9 %, rank 2 to rank 8 | May 2024 to May 2025 | Cloudflare | same |
| Change in request volume | minus 85 % | May 2024 to May 2025 | Cloudflare | same |
The two 2025 shares use different populations: 42 % to 7.2 % is a share of AI-only crawling, while 22.8 % to 2.9 % is a share of a cohort of over 30 AI and search crawlers, and the two are never combined.
Cloudflare's 2024 post also says Bytespider leads in the frequency with which it is blocked, without a per-bot percentage.
Cloudflare's published figures state no cause for the decline between 2024 and 2025, and none is given here.
The 2025 post also reports a drop in request volume of 85 % over the same twelve months, so the fall is one of volume as well as of share. No later Bytespider figure is on record on 2026-11-16.
Does Bytespider follow robots.txt? Statements and measurements on record
No operator statement and no named measurement of whether Bytespider follows robots.txt is on record on 2026-11-16, so neither compliance nor non-compliance is stated here. ByteDance has published no robots.txt statement for the crawler.
Cloudflare's 2024 post, which carries the 40.40 % figure, says blocks by robots.txt "are reliant on the bot operator respecting robots.txt and adhering to RFC9309", and that "user agents are trivial for bot operators to change".
Both statements are general cautions about the method, and Cloudflare does not apply them to Bytespider in particular. robots.txt, the Robots Exclusion Protocol of RFC 9309 (2022), is not legally binding by itself.
Leading in how often it is blocked measures what site owners chose, not how the crawler behaves, so it says nothing about compliance. What other operators state about robots.txt, and which bypasses have been measured, is covered on the page about robots.txt for AI crawlers.
robots.txt lines that name Bytespider
A robots.txt group for the token Bytespider asks the crawler to stay away, and it works only if the crawler reads and honors the file. Cloudflare's bot reference gives the user agent as Bytespider (checked 2026-11-16), and the token comes from that reference, not from ByteDance.
User-agent: Bytespider
Disallow: /
Whole-site robots.txt group for the token Cloudflare lists.
These are whole-site lines only. RFC 9309 says a crawler should not use a cached copy of the file for more than 24 hours unless the file is unreachable, so a change may take up to a day to reach a crawler that honors it.
Blocking Bytespider at the firewall and in Cloudflare
A firewall rule that matches Bytespider's user agent refuses its requests whatever the crawler does with robots.txt, and Cloudflare documents three ways to do this on its network. The table below compares five methods as of 2026-11-16 without ranking them.
| Method | What it matches | Who enforces it | Limit | Source |
|---|---|---|---|---|
robots.txt group User-agent: Bytespider |
the token, by the crawler's own reading | the crawler | relies on the operator respecting robots.txt (Cloudflare); cache up to 24 hours under RFC 9309 | rfc-editor.org/rfc/rfc9309.html; blog.cloudflare.com (2024-07-03) |
| Web server or firewall rule on the user agent (any host) | requests whose user agent contains Bytespider | the site | misses requests that change the string | n/a |
| Cloudflare WAF rule with user agent matching | user agent | Cloudflare | Cloudflare: available to "All other plans"; same string limit | developers.cloudflare.com/ai-crawl-control/reference/bots/ |
Cloudflare WAF custom rule on cf.bot_management.detection_ids (Bytespider: 33563853) |
Cloudflare's detection | Cloudflare | Bot Management customers only | developers.cloudflare.com/ai-crawl-control/reference/bots/ |
| Cloudflare AI Crawler and Agent access policies, one action per behavior: Search, Agent, Training. | "Verified bots with that behavior and matching unverified bots" | Cloudflare | documentation names no bot, so Bytespider's behavior class is not stated | developers.cloudflare.com/bots/additional-configurations/block-ai-bots/ (updated 2026-11-02) |
Cloudflare's documentation says Bot Management customers can use cf.bot_management.detection_ids in WAF custom rules, and that "All other plans can use user agent matching in robots.txt or WAF rules" (checked 2026-11-16). A server rule outside Cloudflare, in a web server or another firewall, matches the same string and has the same limit.
Cloudflare's access policies set an action for each behavior, Search, Agent or Training. The documentation does not say which behavior Bytespider falls under, and it names no bot at all.
Cloudflare's controls changed over time: a one-click "AI Scrapers and Crawlers" toggle was available to all customers, including the free tier, from 2024-07-03, and on 2026-09-15 Cloudflare announced separate controls for search, AI training and AI agents.
Its press release of 2025-07-01 and its documentation of 2026-11-02 differ on defaults for new domains, so no current default is stated here, and Cloudflare's AI Crawl Control page covers the controls and program terms.
Telling real Bytespider traffic from impostors without an operator IP list
Because ByteDance publishes no IP list or reverse-DNS name for Bytespider, a site cannot confirm that a request carrying the Bytespider user agent comes from ByteDance.
Cloudflare notes that user agents are trivial to change and that operators use a spoofed user agent to appear as a browser (2024-07-03), so a string match proves a label and not an origin.
Other operators do publish lists: OpenAI publishes GPTBot's IP addresses at openai.com/gptbot.json (checked 2026-11-09). For Bytespider, the only classification on record is Cloudflare's own detection, which Bot Management customers can match by ID, and it reflects Cloudflare's judgment, not a ByteDance statement.
General verification methods stay with the section hub. A site that wants to count Bytespider visits can only count requests that carry the string, and its own logs cannot separate the genuine crawler from a client copying the label.
Bytespider and the EU opt-out: Article 4(3) and the GPAI Code
In the EU, a robots.txt disallow for Bytespider can serve as a machine-readable reservation under Article 4(3) of the DSM Directive, but ByteDance is not among the listed signatories of the GPAI Code of Practice.
Article 4(3) of Directive (EU) 2019/790, in force since 2019-06-06, lets rightsholders reserve their rights "in an appropriate manner, such as machine-readable means in the case of content made publicly available online", and how the exception treats an opt-out is set out on the page about the EU text and data mining exception and its opt-out.
The European Commission's list of signatories to the GPAI Code of Practice names 22 signatories as of 2026-10-07, and ByteDance is not among them.
The Code is voluntary and was published on 2025-07-10, and its Measure 1.3 says signatories employ crawlers that read and follow robots.txt as specified in IETF RFC 9309. The Code's robots.txt commitment therefore does not bind ByteDance by signature.
Article 53(1)(c) of Regulation (EU) 2024/1689, applicable from 2025-08-02, requires providers of general-purpose AI models placed on the EU market to keep a policy to comply with Union copyright law, including Article 4(3) reservations. Whether any ByteDance model falls within that scope is not recorded here.
Blocking or allowing Bytespider: benefits, risks, downsides and alternatives
Blocking Bytespider costs little and a firewall block does not depend on the crawler's cooperation, but it earns nothing, cannot be verified against an operator list and does not reach content collected earlier. No ByteDance data program is on record on this page, so licensing is a separate question from blocking.
Benefits:
- A firewall block on the user agent applies whatever the crawler does with robots.txt.
- A rule costs one configuration line or one Cloudflare setting.
- In the EU, a robots.txt disallow can serve as the machine-readable reservation.
Risks:
- A rule that matches the string also blocks impostors and misses a crawler that changes its string.
- No ByteDance statement says what a block means for its collection.
- A user-agent-only rule has no further layer if the string changes.
Downsides:
- A block is prospective, and no statement on deletion of content already collected is on record.
- A block earns nothing, and no payment is attached to allowing the crawler either.
Alternatives:
- Allow Bytespider and leave the site's settings unchanged.
- Use robots.txt only, accepting that it relies on the crawler.
- Charge crawlers through intermediaries instead of blocking them.
- License content to AI developers directly, or not at all, a choice set out on the page about whether to sell or license your data to AI companies at all.
Each route trades control against effort, and none can be checked against an operator record for Bytespider. A site that blocks at the firewall and also publishes a robots.txt group keeps both a technical refusal and a written signal, which can matter for the EU reservation described above.
Other AI crawlers, blocking methods and the law around crawling
Bytespider is one crawler among many, and the pages below cover operators that document their bots, other ways to block or charge crawlers, and the law around crawling.
- AI crawlers section: every AI crawler and control method in one place.
- GPTBot and OpenAI's other crawlers: a crawler whose operator publishes strings and IP lists.
- Amazonbot: Amazon's crawler and its noarchive training opt-out.
- CCBot: Common Crawl's crawler.
- Meta-ExternalAgent: Meta's AI training crawler.
- Pay-per-crawl and crawler licensing: charging crawlers instead of blocking them.
- Opting out of AI training through platform settings: opt-outs for content hosted on platforms, where robots.txt is not the owner's to edit.
- EU AI Act Article 53 copyright policy: the copyright policy GPAI providers must keep under Article 53.
- Is web scraping for AI training legal: copyright, contract and computer-misuse law on scraping.
This page is information, not legal advice.
Frequently asked questions
Who operates Bytespider?
Cloudflare names ByteDance, the owner of TikTok, as Bytespider's operator. ByteDance has published no crawler documentation, and the legal entity that runs the crawler is not identified on any primary page found (checked 2026-11-16).
What user agent does Bytespider send?
Cloudflare's bot reference gives the user agent as Bytespider, which a robots.txt group or a firewall rule can match. ByteDance publishes no full user-agent string, so longer strings seen in logs are not confirmed here.
Does Bytespider respect robots.txt?
No operator statement or named measurement is on record either way. Cloudflare notes that robots.txt blocking relies on the bot operator respecting the file, which is why sites also use firewall rules.
How do I block Bytespider on Cloudflare?
Cloudflare's documentation says plans without Bot Management can match the user agent in WAF rules, while Bot Management customers can use detection IDs in WAF custom rules. Bytespider's ID is 33563853 (checked 2026-11-16).
Is Bytespider still one of the most active AI crawlers?
Cloudflare measured its share of AI-only crawling falling from 42 % in May 2024 to 7.2 % in May 2025. No later figure is on record here.