MarketingPatcher

robots.txt AI crawler generator

Choose which AI crawlers may read your site. AI search crawlers decide whether ChatGPT, Claude and Perplexity can cite you; training crawlers only collect text for models. The file is checked with the same rules as the audit.

Start from
AI search and answer crawlers

Block these and AI assistants cannot find or cite you.

AI training crawlers

Only collect text for model training. Blocking them is up to you.

Every other crawler (User-agent: *)
Sitemap URL
Content-Signal optional

Lets in 6 of 6 AI search crawlers and 0 of 7 training crawlers.

robots.txt 
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

faq

What is the difference between AI search and AI training crawlers?
Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages to answer a question right now and cite the source. Training crawlers such as GPTBot, ClaudeBot and CCBot collect text to train future models. Blocking training crawlers keeps your content out of training data without hiding you from AI answers.
Does robots.txt actually stop AI crawlers?
The major AI companies document that their crawlers follow robots.txt. It is a request, not a lock, and it does not stop a firewall from blocking crawlers you allowed. The MarketingPatcher audit sends real requests as GPTBot, ClaudeBot and PerplexityBot to catch that case.
What is Content-Signal?
A robots.txt line proposed by Cloudflare that states how content may be used after it is fetched: for search, as input to AI answers, or for training. It is optional and not every crawler reads it yet.
Where does the file go?
At the root of your domain, served as plain text at /robots.txt. In most frameworks that means the public or static folder. If your robots.txt is generated (for example by app/robots.ts in Next.js), put these rules into the generator instead.

more free tools