
OpenAI documents four user agents, and only GPTBot is tied to training by OpenAI's statement.
A GPTBot disallow in robots.txt signals that content should not train its models, while OAI-SearchBot is a separate setting for ChatGPT search, ChatGPT-User is a user-initiated fetcher for which OpenAI says robots.txt rules may not apply, and OAI-AdsBot only visits pages submitted as ads.
Every OpenAI statement below comes from its bot documentation, re-checked on 2026-11-10, and the page recommends no rule.
Each user agent has its own IP list. Cloudflare measured an OpenAI crawl-to-refer ratio of 887:1 for 2025-08-01 to 2025-08-07. In the EU a GPTBot disallow can act as the machine-readable reservation of DSM Directive Article 4(3).
Every figure and statement carries its stating party and check date, and how we source, date and check every figure sets out the method.
What GPTBot collects, and how OpenAI's three other bots differ
OpenAI states that GPTBot crawls content that may be used in training its generative AI foundation models, and it describes OAI-SearchBot, ChatGPT-User and OAI-AdsBot as separate bots with other purposes. Its own words are that GPTBot "is used to make our generative AI foundation models more useful and safe" (checked 2026-11-10).
The list below gives each bot's stated purpose, and the section hub on AI crawlers and how to allow, block or charge them places OpenAI's bots among those of other operators.
- GPTBot: crawls content that may be used in training generative AI foundation models.
- OAI-SearchBot: surfaces websites in search results in ChatGPT's search features.
- ChatGPT-User: visits pages for certain user actions in ChatGPT and Custom GPTs, "not used for crawling the web in an automatic fashion".
- OAI-AdsBot: validates the safety of web pages submitted as ads on ChatGPT, and its data is "not used to train generative AI foundation models".
OpenAI states nothing about training use of ChatGPT-User data, so none is described here.
OpenAI's four user agents as OpenAI documents them
OpenAI documents exactly four user agents, each with its own string and its own published IP list. The table below compares them in the order OpenAI lists them, as of 2026-11-10, with the labels OpenAI gives each string.
| User agent | Purpose (OpenAI's words) | User-agent string | String label | IP list | robots.txt (OpenAI's statement) | Training (OpenAI's statement) | Checked | Source |
|---|---|---|---|---|---|---|---|---|
| OAI-SearchBot | "used to surface websites in search results in ChatGPT's search features" | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot |
example, version may change | openai.com/searchbot.json | robots.txt tag; changes take "~24 hours" | not stated as training | 2026-11-10 | developers.openai.com/api/docs/bots |
| OAI-AdsBot | "used to validate the safety of web pages submitted as ads on ChatGPT"; "only visits pages submitted as ads" | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot |
full | openai.com/adsbot.json | not stated | data "not used to train generative AI foundation models" | 2026-11-10 | developers.openai.com/api/docs/bots |
| GPTBot | "used to crawl content that may be used in training our generative AI foundation models" | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot |
example, version may change | openai.com/gptbot.json | robots.txt tag; disallow means content "should not be used in training" | yes, by OpenAI's statement | 2026-11-09, re-checked 2026-11-10 | developers.openai.com/api/docs/bots |
| ChatGPT-User | "certain user actions in ChatGPT and Custom GPTs", GPT Actions; "not used for crawling the web in an automatic fashion" | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot |
full | openai.com/chatgpt-user.json | "robots.txt rules may not apply" | not stated | 2026-11-10 | developers.openai.com/api/docs/bots |
Reading key:
- "Example" strings may change version number, according to OpenAI, while "full" strings are given as complete.
- OAI-SearchBot's string is a Chrome-style string, unlike the other three.
- The page names no other bot, such as an agent or Operator user agent, and this is not proof that none exists.
robots.txt lines for GPTBot, OAI-SearchBot and ChatGPT-User
Each OpenAI token is its own robots.txt setting, so a site can disallow GPTBot for training while allowing OAI-SearchBot for ChatGPT search. OpenAI writes that "each setting is independent of the others" and calls OAI-SearchBot and GPTBot its "robots.txt tags" (checked 2026-11-10).
The table below compares six whole-site goals as of 2026-11-10, each token in its own User-agent group, with what OpenAI's statements say the lines do and what keeps running. A ChatGPT-User group is a request that OpenAI says "may not apply", and OAI-AdsBot needs no line unless a site submits ChatGPT ads.
| Goal | Lines | What it does (OpenAI's statement) | What keeps running | Source |
|---|---|---|---|---|
| Opt out of training, stay in ChatGPT search | User-agent: GPTBot then Disallow: / |
signals content should not train OpenAI's foundation models | OAI-SearchBot crawling; ChatGPT-User fetches; OAI-AdsBot on ad pages | developers.openai.com/api/docs/bots |
| Same, stated explicitly | User-agent: OAI-SearchBot then Allow: /, then User-agent: GPTBot then Disallow: / |
OpenAI's own example of independent settings | same as above | developers.openai.com/api/docs/bots |
| Allow both | no OpenAI group, or Allow: / for both |
search and training crawling allowed | one crawl may serve both uses | developers.openai.com/api/docs/bots |
| Opt out of ChatGPT search too | the first row plus User-agent: OAI-SearchBot then Disallow: / |
not shown in ChatGPT search answers | navigational links may still appear; ChatGPT-User fetches | developers.openai.com/api/docs/bots |
| Ask user-initiated visits to stop | User-agent: ChatGPT-User then Disallow: / |
a request OpenAI says "may not apply" | depends on OpenAI; Cloudflare observed compliance on 2025-08-04 | developers.openai.com/api/docs/bots |
| Ad landing-page checks | no line for sites that submit no ChatGPT ads | OAI-AdsBot visits only pages submitted as ads | n/a | developers.openai.com/api/docs/bots |
User-agent: GPTBot
Disallow: /
These are whole-site lines only. Syntax, caching and other operators' tokens are covered on the page about robots.txt for AI crawlers.
Checking that a visitor really is GPTBot: IP lists and the robots.txt marker
A request is from GPTBot only if its user agent matches OpenAI's string and its source IP address appears in openai.com/gptbot.json, and each of the other three bots has its own list. OpenAI publishes openai.com/searchbot.json, openai.com/chatgpt-user.json and openai.com/adsbot.json (checked 2026-11-10).
A string alone proves nothing, because any client can send it, and OpenAI publishes the lists so that site owners can make this comparison themselves. A check therefore takes three steps:
- Match the user-agent string against the one OpenAI prints for the bot, remembering that GPTBot's and OAI-SearchBot's strings are examples whose version may change.
- Match the source IP address against the list for that bot, since the four lists are separate.
- On requests for robots.txt, look for the marker: OpenAI says it "may use a user-agent string with an additional robots.txt marker", and for OAI-SearchBot prints
...compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot.
For GPTBot, OpenAI says it "may add a robots.txt marker" but prints no string. OpenAI states the marker helps tell robots.txt requests apart when logs lack paths. General spoofing and multi-operator checks stay with the section hub.
What a GPTBot block does and does not change
A GPTBot disallow signals that a site's content should not be used to train OpenAI's generative AI foundation models, and it does not stop OpenAI's search crawling, user-initiated visits or ad checks. That is the meaning OpenAI gives it (checked 2026-11-10), and robots.txt is not legally binding by itself under RFC 9309.
A GPTBot disallow leaves four things running:
- OAI-SearchBot crawling for ChatGPT search.
- ChatGPT-User fetches, for which OpenAI says robots.txt rules may not apply.
- OAI-AdsBot visits to pages submitted as ads.
- Content collected before the rule.
OpenAI states that search changes can take about 24 hours and gives no timing for GPTBot. The documentation describes a signal for future use, and no OpenAI statement about removing content collected before a disallow is recorded here.
Cloudflare reported on 2025-08-04 that it saw ChatGPT-User stop when a site disallowed it, which is Cloudflare's observation and not a commitment by OpenAI.
An OAI-SearchBot opt-out has a different effect: OpenAI says such sites "will not be shown in ChatGPT search answers, though can still appear as navigational links". If both bots are allowed, OpenAI says it "may use the results from just one crawl for both use cases".
How much GPTBot crawls: Cloudflare's measurements
Cloudflare, not OpenAI, has published the only GPTBot volume figures recorded here, and each covers a different period and measure. The table below lists three figures as stated by Cloudflare, each as of its own period.
| Measure | Value | Period | Stated by | Source |
|---|---|---|---|---|
| Share of websites accessed by GPTBot | 35.46 % | year to July 2024 (post of 2024-07-03) | Cloudflare | blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ |
| GPTBot share of Verified Bot traffic | about 7.5 %, second after Googlebot | 2025 | Cloudflare | blog.cloudflare.com/radar-2025-year-in-review/ |
| OpenAI crawl-to-refer ratio | 887:1 | 2025-08-01 to 2025-08-07 | Cloudflare | blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/ |
The three measures are not comparable: one counts websites reached, one counts a share of a traffic class, and one compares crawls with visitors referred back.
No OpenAI volume figure is recorded here, so a site's own server logs, matched against the IP lists above, are the only direct count of GPTBot visits to that site.
OpenAI's commitments behind a GPTBot disallow in the EU
In the EU, a GPTBot disallow can act as the machine-readable reservation of DSM Directive Article 4(3), and OpenAI is a listed signatory of the GPAI Code of Practice, which commits signatories to crawlers that follow robots.txt.
Article 4(3) of Directive (EU) 2019/790 (in force 2019-06-06) allows the general text and data mining exception only where use has not been reserved "in an appropriate manner, such as machine-readable means in the case of content made publicly available online", as set out on the page about the EU text and data mining exception and its opt-out.
OpenAI is among the 22 signatories on the European Commission's Code page, last updated 2026-10-07 (checked 2026-11-09). The Code is voluntary and was published on 2025-07-10, and its Copyright chapter, Measure 1.3, says signatories employ crawlers that read and follow robots.txt as specified in RFC 9309.
Regulation (EU) 2024/1689, Article 53(1)(c), applies from 2025-08-02 and requires general-purpose AI providers to keep a policy that identifies and complies with Article 4(3) reservations.
No US statute or ruling that makes robots.txt binding is recorded here.
Blocking or allowing OpenAI's bots: benefits, risks, downsides and alternatives
Disallowing GPTBot costs nothing and OpenAI states what it signals, but it earns nothing and leaves OpenAI's other bots and earlier collection untouched. Licensing content to OpenAI is a separate decision from crawler rules, and the profile of OpenAI as a data buyer covers that side.
Benefits:
- The rule is free, and a plain text file.
- OpenAI states in its own words what a GPTBot disallow means.
- Search stays if OAI-SearchBot stays allowed, which OpenAI itself recommends for appearing in ChatGPT search.
- In the EU, the rule can act as a machine-readable reservation under DSM Directive Article 4(3).
Risks:
- ChatGPT-User may still fetch pages, because OpenAI says robots.txt rules may not apply.
- Spoofed strings pass as OpenAI's bots without an IP check.
- Allowing both OAI-SearchBot and GPTBot means one crawl can serve both uses.
Downsides:
- Blocking OAI-SearchBot too removes the site from ChatGPT search answers.
- A disallow brings no payment.
- The rule is prospective only.
Alternatives:
- Allow everything.
- Allow search and disallow training, as in OpenAI's own example.
- Charge or license crawling through intermediaries, as covered under the pages below.
- License content directly, or decide on whether to sell or license your data to AI companies at all, where not licensing is a valid outcome.
Blocking, allowing and licensing can be combined, and none is recommended here.
Other AI crawlers, opt-out signals and the law around OpenAI's crawling
GPTBot is one crawler among many, and the pages below cover the others, the other ways to signal an opt-out and the law around crawling.
- AI crawlers and how to allow, block or charge them: every AI crawler and control method in one place.
- Google-Extended: opting out of Gemini training.
- ClaudeBot: Anthropic's ClaudeBot, Claude-User and Claude-SearchBot.
- Pay-per-crawl and crawler licensing: charging crawlers instead of blocking them.
- The shared AI training opt-out: the draft shared AI training opt-out.
- Opting out of AI training through platform settings: opt-outs for content hosted on platforms, where robots.txt is not the owner's to edit.
- EU AI Act Article 53 copyright policy: the copyright policy GPAI providers such as OpenAI must keep under Article 53.
- Is web scraping for AI training legal: copyright, contract and computer-misuse law on scraping.
- OpenAI copyright litigation (MDL 3143): the US cases over OpenAI's training data.
This page is information, not legal advice.
Frequently asked questions
Does blocking GPTBot remove a site from ChatGPT search?
No: ChatGPT search uses OAI-SearchBot, a separate robots.txt setting, so a GPTBot disallow leaves search untouched. Only an OAI-SearchBot opt-out keeps a site out of ChatGPT search answers, and it may still appear as a navigational link (checked 2026-11-10).
Does ChatGPT-User follow robots.txt?
Not reliably: OpenAI states that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Cloudflare reported on 2025-08-04 that it observed ChatGPT-User stopping when disallowed.
What is the GPTBot user agent string?
OpenAI gives Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot as an example string. OpenAI says the version number may change (checked 2026-11-10).
Where does OpenAI publish GPTBot IP ranges?
OpenAI publishes GPTBot's IP addresses at openai.com/gptbot.json. It lists OAI-SearchBot at searchbot.json, ChatGPT-User at chatgpt-user.json and OAI-AdsBot at adsbot.json.
How long does a robots.txt change take to reach OpenAI's bots?
For search, OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust. It states no timing for GPTBot, and RFC 9309 says a cached robots.txt should not be used for more than 24 hours.