Home / Resources / robots.txt for AI crawlers
GEO · Technical Guiderobots.txt for AI crawlers: GPTBot, PerplexityBot, ClaudeBot and Google-Extended explained
Many sites block AI crawlers without realising what they have switched off. Here is how the main crawlers differ and how to set robots.txt rules on purpose rather than by accident.
Key takeaway: AI assistants use different crawlers for training, for search and for fetching a page when a user asks. Blocking all of them with one broad rule can remove you from AI answers; allowing search and user-fetch bots while opting out of training is a common middle path. robots.txt is a request, not a security control, and vendors change their crawlers, so always check each vendor’s current documentation.
Three kinds of AI crawler
- Training crawlers collect content to train models. Blocking them opts your content out of future training, but does not by itself remove you from search-based answers.
- Search or index crawlers build the index an assistant draws on when it answers with citations. Blocking these can stop you being cited.
- User-triggered fetchers load a page when a person asks the assistant about it. Many vendors say these act on a user’s behalf and may not follow robots.txt in the same way, so check their documentation.
The main crawlers
| Crawler (user-agent) | Operator | Typical purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection |
| OAI-SearchBot | OpenAI | Search results inside ChatGPT |
| ChatGPT-User | OpenAI | Fetching a page when a user asks |
| ClaudeBot | Anthropic | Training data collection |
| PerplexityBot | Perplexity | Building Perplexity’s search index |
| Google-Extended | A control token for whether content is used for Gemini models; it does not affect Google Search | |
| Googlebot | Search crawling, which also feeds AI Overviews | |
| Applebot-Extended | Apple | A control for use of content in Apple’s AI features |
| CCBot | Common Crawl | An open web archive that many models draw on |
This list reflects publicly documented purposes at the time of writing. Vendors add, rename and change crawlers, so confirm against their own pages before editing production rules.
Setting rules on purpose
Decide your goal first.
- Be visible in AI answers: allow search and user-fetch crawlers, and do not block Googlebot.
- Visible but opted out of training: allow the search crawlers and disallow the training ones, such as GPTBot and ClaudeBot, if that matches your policy.
- Not in AI at all: disallow the lot, knowing you will probably not be cited.
An example that allows OpenAI’s search and user bots while opting out of its training bot:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /
Common mistakes
- A blanket
User-agent: * Disallow: /left over from a staging site. - A CDN or firewall bot-protection setting that blocks AI crawlers even though robots.txt allows them.
- Key content that only appears after JavaScript runs, so a crawler that does not execute scripts sees an empty page.
- Assuming robots.txt protects private content. It is a public request, not access control; use authentication for anything sensitive.
How to check your setup
- Open
yourdomain.com/robots.txtand read every rule. - Check your CDN or security tool for bot rules.
- View the page source and confirm your main content is in the HTML.
- Test real buyer questions in the assistants before and after a change. See how to track AI search visibility.
You can also help assistants find your key pages with a llms.txt file, though support is not universal.
Frequently asked questions
Should I block GPTBot?
It depends on your goal. Blocking GPTBot opts your content out of OpenAI model training, but it is separate from OAI-SearchBot, which relates to search results in ChatGPT. Decide each deliberately.
Does blocking Google-Extended remove me from Google Search or AI Overviews?
Google documents Google-Extended as a control for use of content by Gemini models, separate from Google Search. AI Overviews are fed by Google Search crawling, so check Google’s current documentation for how each control applies.
Is robots.txt enough to keep content private?
No. It is a voluntary request that well-behaved crawlers follow. Use authentication or access controls for anything sensitive.
Can I be cited by AI if I block all AI crawlers?
It is unlikely, because assistants cannot cite what their crawlers cannot read, though some answers draw on third-party pages that mention you.
Want this done for your brand?
Start with a free audit: a written scope and a baseline of how AI assistants describe your brand today.
Get my free audit →