The token is one of the crawler controls within AI training data licensing and acquisition, and it governs training only, not Apple's search use of the same pages.
Every statement about Apple's controls here is Apple's, from its support page "About Applebot" published 2026-09-04 and re-read on 2026-11-17, and this page recommends no rule. Apple names three uses of Applebot's data and gives each its own control, and a disallow of the token leaves Applebot crawling and Search inclusion running.
Every figure and statement carries its stating party and check date, and how we source, date and check every figure sets out the method.
What Applebot-Extended is: Apple's training control over Applebot's data, not a crawler
Apple describes Applebot-Extended as a secondary user agent that gives web publishers additional controls over how their content can be used by Apple, and states that it "does not crawl webpages" and "is only used to determine how to use the data crawled by the Applebot user agent" (page published 2026-09-04, checked 2026-11-17).
The two names play different roles.
- Applebot: the crawler that fetches pages, whose data powers the search technology in Spotlight, Siri and Safari and, Apple says, "may also be used to help train" Apple foundation models.
- Applebot-Extended: the secondary user agent that only decides how that crawled data is used for training.
The scope Apple states is wider than the H1 product name. A disallow opts content out of training Apple's "general purpose foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools".
Because the token never fetches anything, no server log shows an Applebot-Extended visit; Apple reads the token when Applebot's data is used.
Two errors recur in search results, a description of the token as a training crawler and a claim that it does not exist, and Apple's page supports neither. Applebot-Extended is one entry in the list of AI crawlers and how to allow, block or charge them.
Three uses of Applebot-crawled content and the control Apple gives for each
Apple names three uses of Applebot-crawled content, search, foundation model training and context for AI answers, and gives each its own control. The table below compares them in Apple's order, as of 2026-11-17, and ranks none.
| Use (Apple's words) | Products Apple names | Control | What the control does (Apple) | Source |
|---|---|---|---|---|
| search technology | Spotlight, Siri, Safari | robots.txt rules for Applebot; noindex meta tag |
Applebot does not crawl the paths; noindex keeps the page out of Spotlight and Siri Suggestions | support.apple.com/en-us/119829 |
| "help train Apple foundation models" | Apple Intelligence, Services, Developer Tools | robots.txt group for Applebot-Extended |
opts content out of training Apple's general purpose foundation models | support.apple.com/en-us/119829 |
| "additional context and up-to-date content" for AI output, for example broad world knowledge answers in Siri and Search | Siri, Search | nosnippet (meta tag or X-Robots-Tag: applebot: nosnippet); isAccessibleForFree: false for paywalled pages |
content not used as that context; paywalled pages stay eligible for search results | support.apple.com/en-us/119829 |
The controls stack and do not replace one another. Apple says that even with Applebot-Extended disallowed and nosnippet set, a site's instructions may still allow Applebot to crawl, and the content remains discoverable through Spotlight, Siri and Safari.
The isAccessibleForFree signal applies at the page level only, and Apple adds that "Section-level markup using hasPart is not supported".
Writing the Applebot-Extended rule next to Applebot rules
The rule is a separate User-agent group naming Applebot-Extended; Apple's own example is User-agent: Applebot-Extended followed by Disallow: /private/, which withholds one path from training, and Disallow: / withholds the whole site. The table below shows whole-site and path rules as of 2026-11-17, with Apple's statement of what each does.
| Goal | Lines | What it does (Apple's statement) | What keeps running | Source |
|---|---|---|---|---|
| Keep one path out of training | User-agent: Applebot-ExtendedDisallow: /private/ (Apple's example) |
content under the path not used to train foundation models | Applebot crawling, search inclusion | support.apple.com/en-us/119829 |
| Keep the whole site out of training | User-agent: Applebot-ExtendedDisallow: / |
same, for every path | Applebot crawling, search inclusion, ranking unaffected | support.apple.com/en-us/119829 |
| Also keep pages out of AI answers | the row above plus <meta name="applebot" content="nosnippet"> per page |
no description or web answer; not used as AI-output context | search inclusion with title only | support.apple.com/en-us/119829 |
| Leave Apple search too | User-agent: ApplebotDisallow: / |
Applebot does not crawl | iTMS for registered Podcasts URLs | support.apple.com/en-us/119829 |
| Rules written only for Googlebot | no Applebot group | Applebot follows the Googlebot instructions | Applebot-Extended handling not stated | support.apple.com/en-us/119829 |
Apple states that Applebot "respects standard robots.txt directives in general search crawls that are targeted at Applebot". It also states that when robots instructions do not mention Applebot but mention Googlebot, Applebot follows the Googlebot instructions.
Apple does not say whether that fallback, a User-agent: * group or a Google-Extended rule is read as an Applebot-Extended rule, so no advice on those cases is given here. Wildcard groups, fallbacks and other operators' tokens are set out under robots.txt for AI crawlers.
Rendering is a separate matter. Apple writes that if JavaScript, CSS and other resources are blocked in robots.txt, Applebot may not render the content properly, so a training opt-out belongs in the Applebot-Extended group and not in resource blocks for Applebot.
What an Applebot-Extended disallow changes and what keeps running
A disallow tells Apple not to use the content to train its foundation models; Apple states that the pages "can still be included in search results" and that "Site rules for Applebot-Extended are not considered in ranking for Search" (page published 2026-09-04, checked 2026-11-17).
Five things keep running after a disallow, by Apple's statements.
- Applebot crawling, which the token does not control.
- Inclusion in search results.
- Search ranking, which ignores Applebot-Extended rules.
- Use as AI-answer context, unless nosnippet or isAccessibleForFree false is set on the page.
- The iTMS agent for Apple Podcasts, which "does not follow robots.txt, as it is not a general search crawler" and "only crawls URLs associated with registered content on Apple Podcasts".
Apple's page says nothing about content used before a disallow was added or about how fast a change applies. Robots.txt is a convention defined in RFC 9309 and is not legally binding by itself.
Apple gives its own argument for the opposite choice, "Allowing Applebot-Extended will help improve the capabilities and quality of Apple's generative AI models over time", which is Apple's claim and is quoted without comment.
Checking that a visitor really is Applebot: reverse DNS, the IP file and the user-agent format
Apple's own tests for Applebot traffic are a reverse DNS lookup in the *.applebot.apple.com domain or a match of the IP address against the CIDR prefixes in Apple's JSON file; the user-agent string alone proves nothing, because any client can send it.
Apple writes that traffic is "generally identified by using reverse DNS in the *.applebot.apple.com domain".
A check takes three steps.
- Run a reverse lookup on the source IP and confirm the host sits in *.applebot.apple.com, then confirm by a forward lookup that the host points back to the same address; Apple's example is 17-58-101-179.applebot.apple.com with address 17.58.101.179.
- Match the IP against the CIDR prefixes in search.developer.apple.com/applebot.json.
- Read the user-agent for "Applebot" and the link +http://www.apple.com/go/applebot, which is only a hint.
The file carries a creation time of 2026-09-15 and 24 IPv4 prefixes at the 2026-11-17 reading, with no IPv6 prefixes.
Apple gives the general user-agent format as ending in "(Applebot/Applebot_version; +http://www.apple.com/go/applebot)" and notes that "Occasionally, Applebot will update the browser version that it advertises". No Applebot-Extended string exists, because the token does not crawl. Apple asks for questions or concerns at applebot@apple.com.
When an Applebot-Extended disallow counts as a legal reservation
In the EU, DSM Directive Article 4(3) lets rightsholders reserve text and data mining rights by machine-readable means, and the AI Act requires providers of general-purpose AI models to keep a policy that identifies and complies with such reservations; no court ruling on record here decides whether an Applebot-Extended disallow is such a reservation.
Article 4(3) of Directive (EU) 2019/790, in force since 2019-06-06, lets rightsholders reserve their rights "in an appropriate manner, such as machine-readable means in the case of content made publicly available online", and the EU text and data mining exception and its opt-out are set out on their own page.
Article 53(1)(c) of Regulation (EU) 2024/1689 sets that policy duty, and the obligations for general-purpose AI model providers apply from 2025-08-02. The voluntary General-Purpose AI Code of Practice, published 2025-07-10, says in Measure 1.3 of its copyright chapter that signatories' crawlers follow robots.txt as specified in RFC 9309.
Apple's own EU status is covered on its buyer profile and is not repeated here.
Outside such a reservation, robots.txt is a technical convention. RFC 9309 defines the protocol and does not make it binding, and this page is information, not a statement of any court's view.
Disallowing Applebot-Extended: benefits, risks, downsides and alternatives
Disallowing Applebot-Extended costs nothing and, by Apple's statement, leaves Search inclusion and ranking untouched, but it earns nothing, covers training only and relies on Apple honoring its own page.
Licensing content to Apple is a separate question, and Apple publishes no program, terms or payouts for data holders as of 2026-11-09, as the profile of Apple as a data buyer records.
Benefits:
- The rule is a two-line edit to robots.txt.
- Search inclusion and ranking stay as they are, by Apple's statement.
- In the EU the disallow is a machine-readable robots.txt signal of the kind Article 4(3) and the Code of Practice refer to.
Risks:
- Spoofed user agents need a reverse-DNS or IP check before any log line is trusted.
- A rule that sits only in a
User-agent: *group or under Googlebot may not reach Applebot-Extended, and Apple is silent on it. - Blocking Applebot itself to stop training also removes search visibility in Spotlight, Siri and Safari.
Downsides:
- No payment: the support page mentions none.
- Training only: use as AI-answer context needs nosnippet or isAccessibleForFree.
- Earlier use of the content is not addressed by Apple.
Alternatives:
- Allow everything, which is the course Apple's page gives its reason for.
- Disallow Applebot-Extended only for chosen paths, as in Apple's example.
- Add nosnippet on pages as well.
- License the content directly, or decide on whether to sell or license data to AI companies at all.
Other crawler controls, Apple's acquisition record and the law around Applebot's data
Applebot-Extended is one training control among many, and the pages below cover the others, the other ways to signal an opt-out and the law around crawling.
- AI crawlers and how to allow, block or charge them: every AI crawler and control method in one place.
- Google-Extended: Google's parallel training token, which also does not crawl.
- GPTBot and OpenAI's other crawlers: OpenAI's training crawler and its search and user bots.
- ClaudeBot: Anthropic's training crawler and its two other robots.
- Amazonbot: Amazon's crawler and its page-level training opt-out.
- Meta-ExternalAgent: Meta's AI training crawler.
- CCBot: Common Crawl's crawler, whose archive other developers train on.
- PerplexityBot and Perplexity-User: Perplexity's search crawler and user fetcher.
- Bytespider: ByteDance's undocumented crawler, by contrast with Apple's documented one.
- the shared AI training opt-out: the shared training signal, on which Apple has no statement on record.
- machine-readable TDM reservations and meta tags: reservation signals beyond robots.txt.
- pay-per-crawl and crawler licensing: charging crawlers instead of blocking them.
- opting out of AI training through platform settings: opt-outs for content hosted on platforms, where robots.txt is not the holder's to edit.
- EU AI Act Article 53 copyright policy: the copyright policy general-purpose AI providers must keep under Article 53.
- is web scraping for AI training legal: copyright, contract and computer-misuse law on scraping.
- Authors v. Apple: the book claims against Apple, which concern books, not crawling.
This page is information, not legal advice.
Frequently asked questions
Does blocking Applebot-Extended remove a site from Siri, Spotlight or Safari?
No, by Apple's statement: Applebot-Extended does not crawl, pages that disallow it can still be included in search results, and its rules are not considered in Search ranking. The page was published 2026-09-04 and checked 2026-11-17.
Is there an Applebot-Extended user-agent string in server logs?
No: Apple states Applebot-Extended does not crawl webpages, so it sends no requests. The visits come from Applebot, whose user-agent contains "Applebot" and Apple's link apple.com/go/applebot (checked 2026-11-17).
Can Applebot be slowed with Crawl-delay?
No. Apple states that "Applebot does not follow crawl-delay" and that its crawl rate adjusts automatically when a site slows down or returns errors. The page was published 2026-09-04 and checked 2026-11-17.
Does an Applebot-Extended disallow remove content Apple already used?
Apple does not say. Its page describes how Applebot-Extended decides the use of crawled data and says nothing about content used for training before a disallow was added (checked 2026-11-17).