GPTBot, ClaudeBot, PerplexityBot: a robots.txt for AI crawlers

A lot of sites added "block all AI bots" rules to their robots.txt in the last two years. Many of them blocked the wrong thing. AI companies run more than one crawler, and they do very different jobs.

Three kinds of AI crawler

Training crawlers collect text that may be used to train future models. They do not send you visitors. Examples are GPTBot from OpenAI, ClaudeBot from Anthropic and CCBot from Common Crawl.

Search crawlers index pages so an assistant can find and cite them when answering a question. These are the ones that put your link in front of a user. Examples are OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot.

User agents acting for a person fetch a page because someone asked the assistant to read it right now. Examples are ChatGPT-User, Claude-User and Perplexity-User.

Google is a special case. Google-Extended is not a separate crawler but a token that controls whether content Googlebot already fetched may be used for Gemini. Blocking it does not affect Google Search.

Decide each group on purpose

Whether you allow training is a business decision, and either answer is reasonable. Whether you allow search crawlers is usually not: blocking them means AI assistants cannot cite you.

A robots.txt that opts out of training but stays visible in AI search looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Check the layer below robots.txt

robots.txt is a request, and well-behaved crawlers follow it. But a CDN or firewall bot rule can block a crawler before it ever reads the file. If your host has a one-click "block AI bots" setting, check what it covers, because some versions block search crawlers too.

Build yours

The robots.txt AI crawler generator lets you choose each group separately and checks the result with the same rules as the audit. After you deploy it, run a free audit to confirm the crawlers you meant to allow can actually reach the page.