robots.txt and AI crawlers: which to allow, which to block, and the exact lines

How robots.txt works

robots.txt is a plain-text file at the root of your site (yourdomain.com/robots.txt). It is a request, not a lock. Well-behaved crawlers read it before they fetch anything and follow it. Nothing technically forces them to, and a firewall is a separate thing.

The file is made of groups. Each group starts with one or more User-agent lines that name a crawler, followed by Allow and Disallow rules for paths. A crawler uses the group that matches its own name most specifically, and ignores the rest. This detail causes more mistakes than anything else, so we come back to it below.

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

The 15 AI crawlers we test

These are the crawlers our free check looks for. The names are the tokens their operators publish for robots.txt.

Answer and search crawlers (blocking these can keep you out of AI answers):

• OAI-SearchBot (OpenAI): builds the search index behind ChatGPT search.
• ChatGPT-User (OpenAI): fetches a page when a ChatGPT user asks about it.
• PerplexityBot (Perplexity): builds the search index behind Perplexity answers.
• Perplexity-User (Perplexity): fetches a page when a Perplexity user asks about it.
• Claude-SearchBot (Anthropic): builds the search index behind Claude's search results.
• Claude-User (Anthropic): fetches a page when a Claude user asks about it.

Training crawlers and opt-out tokens

Blocking these is a common policy choice. It does not decide whether you appear in answers.

• GPTBot (OpenAI): collects public pages to train OpenAI models.
• ClaudeBot (Anthropic): collects public pages to train Anthropic models.
• anthropic-ai (Anthropic): an older name still seen in robots.txt files.
• Google-Extended (Google): lets you opt out of Google using your pages for Gemini. It does not affect Google Search.
• Applebot-Extended (Apple): lets you opt out of Apple using your pages to train its models.
• CCBot (Common Crawl): builds the open Common Crawl archive that many AI models learn from.
• Bytespider (ByteDance): collects public pages for ByteDance's AI models.
• Meta-ExternalAgent (Meta): collects public pages to train Meta's AI models.
• Amazonbot (Amazon): collects public pages for Amazon's services and AI models.

Policy 1: allow everything

If you want to be found everywhere and have no objection to training, you need no AI-specific rules. A robots.txt that only keeps crawlers out of private areas is enough.

User-agent: *
Disallow: /admin/
Disallow: /cart/

Policy 2: appear in AI answers, opt out of training

This is the most common deliberate choice. It keeps the answer and search crawlers out of your block list and asks the training crawlers to stay away.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
Disallow: /

User-agent: *
Disallow: /admin/

Policy 3: block every AI crawler

Choose this only if you are sure. It also blocks the crawlers that power AI search, so you should expect not to be cited by those assistants. Add the six answer crawlers to the list above.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
Disallow: /

The mistakes that block AI search by accident

Most invisible sites did not choose to be invisible. These are the causes we see most.

• A leftover staging rule. User-agent: * with Disallow: / blocks every crawler, including the ones you want. It is easy to forget after launch.
• A copied "block all AI bots" snippet. Many include the search crawlers along with the training ones. If your goal was only to stop training, you have blocked more than you meant to.
• A group that overrides the wildcard. If a crawler has its own group, it follows that group only. A bot named in a group with no rules for your private paths will ignore the Disallow lines you wrote under User-agent: *.
• A security or hosting plugin. Some add bot-blocking rules at the server or firewall level that robots.txt never shows. The file can say allowed while the server answers with an error.
• Assuming Google-Extended affects search. It does not. It only controls use for Gemini.

What robots.txt cannot do

It cannot remove content that a crawler has already collected. It cannot stop a crawler that ignores it, and it is not a security measure: anyone can read the file, so never list a private URL in it to hide it.

It also applies per host. A subdomain such as blog.yourdomain.com needs its own file.

Two real examples

When we ran our check on nytimes.com on 20 October 2026, its robots.txt blocked 14 of the 15 crawlers we test: all six answer crawlers and eight of the nine training crawlers. That is a deliberate policy by a large publisher, and it shows what a block list looks like in practice.

Our own site allows all 15. Our robots.txt keeps crawlers out of /api/ only, and our check reports each crawler as allowed with that one exception. Robots.txt files change, so run the check on your own domain rather than relying on a snapshot like these.