afterclick

Is your website blocking AI crawlers?

Check whether ChatGPT, Claude, Perplexity and other AI tools are allowed to read your site. Free, no sign-up.

Check a website

Enter an address. We read only its public robots.txt and llms.txt files, nothing else.

For example example.com or https://www.example.com

Free, no sign-up. See the privacy notice.

What this check does

Every website can publish a file called robots.txt that tells crawlers which pages they may read. AI companies run crawlers too, each with its own name. This tool reads your robots.txt, compares it with the 15 AI crawlers listed below, and shows which are allowed and which are blocked, including the exact line that blocks them. It also looks for an llms.txt file. It reads nothing else on your site.

Answer crawlers and training crawlers

The two kinds do different jobs, so the sensible choice is often different too. Answer crawlers fetch pages so an AI tool can search or quote them. Blocking them can keep your site out of AI answers. Training crawlers collect pages to build models. Many sites block those on purpose, and it does not change how you appear in answers.

Answer crawlers

  • OAI-SearchBotOpenAI

    Builds the search index behind ChatGPT search.

  • ChatGPT-UserOpenAI

    Fetches a page when a ChatGPT user asks about it.

  • PerplexityBotPerplexity

    Builds the search index behind Perplexity answers.

  • Perplexity-UserPerplexity

    Fetches a page when a Perplexity user asks about it.

  • Claude-SearchBotAnthropic

    Builds the search index behind Claude's search results.

  • Claude-UserAnthropic

    Fetches a page when a Claude user asks about it.

Training crawlers

  • GPTBotOpenAI

    Collects public pages to train OpenAI models.

  • ClaudeBotAnthropic

    Collects public pages to train Anthropic models.

  • anthropic-aiAnthropic

    An older Anthropic crawler name, still seen in robots.txt files.

  • Google-ExtendedGoogle

    Lets you opt out of Google using your pages for Gemini. It does not affect Google Search.

  • Applebot-ExtendedApple

    Lets you opt out of Apple using your pages to train its models.

  • CCBotCommon Crawl

    Builds the open Common Crawl archive that many AI models learn from.

  • BytespiderByteDance

    Collects public pages for ByteDance's AI models.

  • Meta-ExternalAgentMeta

    Collects public pages to train Meta's AI models.

  • AmazonbotAmazon

    Collects public pages for Amazon's services and AI models.

How to allow or block a crawler

Edit the robots.txt file at the root of your site, for example example.com/robots.txt. To let every crawler in:

User-agent: *
Allow: /

To block one training crawler and leave everything else as it is:

User-agent: GPTBot
Disallow: /

Crawlers read the file again from time to time, so a change can take days to have an effect. Run the check again after you edit it.

Mistakes that block AI crawlers by accident

  • A blanket Disallow: / under User-agent: * blocks every crawler, including the ones that feed AI answers.
  • A robots.txt written for a test site that was copied to the live site.
  • A firewall or bot-protection setting that blocks automated visitors, even though robots.txt allows them.
  • Blocking an answer crawler when you meant to block a training crawler, because the names look alike.

Questions people ask

Does blocking Google-Extended remove me from Google Search?

No. Google-Extended only controls whether Google may use your pages for its AI models. Google Search uses a different crawler.

Will allowing every crawler make AI tools mention my site?

Not by itself. Allowing a crawler only lets it read your pages. Whether an AI tool uses them depends on the quality and clarity of the content and on that tool.

Do I need an llms.txt file?

It is optional, and many AI tools do not read it yet. It costs little to add, but do not expect it to change results on its own.

Want the whole picture?

The free SEO audit checks your titles, headings, links, structured data and AI readiness across your pages and lists what to fix first.

Run the free audit