AI crawlers are bots that fetch web pages for AI products. They fall into three jobs: training a model, indexing pages for AI search, and fetching a page when a user asks. For a tool site, the safe default is to allow the search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and decide separately on training bots (GPTBot, ClaudeBot, Google-Extended).
This guide is part of our Tool Owner Resources. Below is the full list of AI crawlers, what each one does, whether it follows robots.txt, and a robots.txt you can copy.
Key takeaways
- Blocking a training bot does not remove you from AI search. Search and training use different bots.
- Blocking OAI-SearchBot keeps your site out of ChatGPT search answers.
- User-triggered bots (ChatGPT-User, Perplexity-User) may ignore robots.txt.
- Most tool sites already allow the search bots. Check yours with our free AI search readiness checker.
What do tool sites actually block?
We checked the robots.txt of 541 AI and SEO tool websites in our AI tool visibility study. Only 34.6% name any AI bot at all. Blocking is rare and aimed at training bots.
Share of 541 tool sites that block each bot in robots.txt. Source: AI SEO Gazette study, 2026. About 6.7% block training bots while leaving search bots open, which is the setup most owners want.
What is the list of AI crawlers?
Here are the bots that matter for a tool site, with the vendor’s own wording on what each does.
| Bot | Company | Purpose | Follows robots.txt? | For a tool site |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Yes | Allow |
| GPTBot | OpenAI | Crawls content that may train OpenAI models | Yes | Your choice |
| ChatGPT-User | OpenAI | Visits a page when a user asks in ChatGPT | May not apply | Allow |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search | Yes | Allow |
| Perplexity-User | Perplexity | Fetches a page when a user asks | Generally ignores it | Allow |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude search results | Yes | Allow |
| Claude-User | Anthropic | Fetches a page when a user asks Claude | Yes | Allow |
| ClaudeBot | Anthropic | Collects web content that may train Claude | Yes | Your choice |
| Google-Extended | Control token for Gemini training and grounding | Yes (token) | Your choice | |
| Googlebot | Crawls for Google Search | Yes | Allow |
Sources: OpenAI bots documentation, Perplexity bot guide, Anthropic crawler help article, Google common crawlers.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI’s crawler for ChatGPT search. OpenAI says it is used to surface websites in search results in ChatGPT’s search features. Sites that opt out in robots.txt will not appear in ChatGPT search answers, though they can still show as plain navigational links, and a robots.txt change can take about 24 hours to take effect.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot gathers content that may be used to train OpenAI’s models. OAI-SearchBot decides whether you appear in ChatGPT search. They are controlled separately, so you can disallow GPTBot and still allow OAI-SearchBot.
Should I block GPTBot?
Blocking GPTBot tells OpenAI not to use your content for training. It does not affect ChatGPT search. For a tool site that wants to be recommended, there is no search penalty either way, so this is a content-rights choice, not an SEO one. If you want models to learn what your tool does, leave it open.
Does Perplexity respect robots.txt?
PerplexityBot follows robots.txt, and Perplexity says to allow it to appear in search results. Perplexity-User, which fetches pages when someone asks a question, generally ignores robots.txt because a user requested the fetch. In August 2025 Cloudflare accused Perplexity of using undeclared crawlers to get past no-crawl rules. Treat robots.txt as a request, not a lock. To truly block a bot you need a firewall rule.
What is Google-Extended?
Google-Extended is a robots.txt token that controls whether content Google crawls may train future Gemini models. Google states it does not affect inclusion in Google Search and is not a ranking signal. It is not a separate crawler. Googlebot does the fetching.
What are ClaudeBot, Claude-SearchBot and Claude-User?
Anthropic uses three bots and says blocking one does not block the others. ClaudeBot collects content that may train Claude. Claude-SearchBot indexes content to improve search results, and disabling it may reduce your visibility there. Claude-User fetches a page when a user asks Claude a question that needs it. Anthropic says its bots honor robots.txt.
Copy-paste robots.txt for a tool site
This version allows every search and user-triggered bot and leaves the training bots open. To opt out of training, uncomment the last three blocks.
# Allow AI search and user-triggered bots
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
# Optional: opt out of model training
# User-agent: GPTBot
# Disallow: /
#
# User-agent: ClaudeBot
# Disallow: /
#
# User-agent: Google-Extended
# Disallow: /We keep this and eight other copy-paste files on our templates page. Check the result with the AI Crawler Access Checker listing, or run the AI search readiness checker on your homepage.
Decision table: which bots should you block?
| Your goal | Allow | Block |
|---|---|---|
| Be found and recommended in AI answers | All search and user bots | Nothing |
| Be found, but keep content out of training | All search and user bots | GPTBot, ClaudeBot, Google-Extended |
| Stay out of AI answers entirely | Googlebot only | All AI bots, plus a firewall rule |
Most tool owners want the first or second row. Our guide on what llms.txt does covers the other file people ask about. For more on how publishers control these bots, see AI crawling controls for publishers and MCP servers explained. Browse more site-audit options in our SEO tools category, or see SagBot, which shows how AI bots view a site.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT search?
No. ChatGPT search uses OAI-SearchBot. GPTBot is for training, and OpenAI says ChatGPT-User does not decide search eligibility either.
How long does a robots.txt change take to work?
OpenAI says its systems can take about 24 hours to adjust after a robots.txt update. Other vendors do not publish a delay.
Do I need to list every AI bot in robots.txt?
No. Bots not named follow your wildcard rules, and most tool sites name none. In our study only 34.6% of tool sites name any AI bot.
Can a bot ignore my robots.txt?
Yes. robots.txt is a request. User-triggered fetchers such as ChatGPT-User and Perplexity-User may ignore it, so use firewall rules if you need a hard block.
Will allowing AI bots hurt my Google rankings?
No. Googlebot is separate, and Google says Google-Extended does not affect inclusion in Google Search.
Sources
- OpenAI, Overview of OpenAI crawlers
- Perplexity, Perplexity crawlers
- Anthropic, Does Anthropic crawl data from the web?
- Google, Google’s common crawlers
- Cloudflare, Perplexity is using stealth, undeclared crawlers (Aug 4, 2025)
Part of our Tool Owner Resources. Last updated 21 Nov 2026.
Related guide: Perplexity SEO.
Part of our SaaS SEO guide.
