AI readiness: what it means for a website, and how to check yours

What people mean by AI readiness

More and more people put a question to an AI assistant (ChatGPT, Claude, Perplexity, Gemini) instead of typing it into a search engine. Those assistants answer from what they already know and, increasingly, from pages they fetch on the spot. If your pages cannot be fetched or cannot be understood, you cannot be part of the answer.

AI readiness is our short name for that: how easily an AI system can reach your public pages, read them and work out what your business is. It is a set of ordinary technical conditions, not a trick and not a ranking switch.

One thing to say plainly before anything else. Nobody can promise that an AI assistant will mention or recommend your site. Which sources a model uses, and how it weighs them, is not published and changes often. What you can do is remove the avoidable reasons you would be skipped.

Two kinds of AI crawler, and why the difference matters

AI companies send automated visitors to websites, and they send them for different jobs. We split the ones we test into two groups.

Answer crawlers fetch pages so an assistant can answer a question or build a search index. Examples are OAI-SearchBot and ChatGPT-User from OpenAI, PerplexityBot and Perplexity-User from Perplexity, and Claude-SearchBot and Claude-User from Anthropic. Blocking these can keep a site out of AI answers.

Training crawlers collect pages to train models. Examples are GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent and Amazonbot. Blocking them is a common and legitimate policy choice. It does not decide whether you appear in answers.

The two groups use different names, so a site can allow one and block the other. Many sites have blocked both without meaning to, which is the usual way a business ends up invisible to AI search by accident. The robots.txt guide goes through the exact lines.

The eight checks we run

The AI readiness part of our audit looks at eight conditions. Each one describes something that may help an AI system use your site. None of them is a guarantee.

• AI crawler accessibility: whether your robots.txt and server responses let AI crawlers fetch your public pages.
• HTTP accessibility: error pages, redirect loops and login walls in front of public content.
• Semantic structure: sensible page titles, a clear heading order, meaningful link text and proper HTML elements.
• Structured data: schema markup, such as Organization, that states facts about your business in a machine-readable way.
• Content extractability: whether the text is present in the HTML the server sends. Pages that only show text after JavaScript runs can look empty to a crawler.
• Entity consistency: whether your business name, services and contact details match across pages and structured data.
• Freshness signals: visible or structured publication and update dates. Missing dates are informational, not a failure.
• llms.txt: whether the optional llms.txt file exists. Its absence is not treated as a failure.

What llms.txt is, and how much weight to give it

llms.txt is a proposal for a plain-text file at the root of a site that gives AI tools a short, curated summary: a title, a one-line description and a list of the pages that matter. It is not a standard that every AI company has adopted, and many AI tools do not read it yet.

It is cheap to add and harmless if ignored, so a short, accurate file is reasonable. Do not expect traffic from it, and be wary of anyone who sells it as a growth lever. Our llms.txt checker tells you whether you have one and whether it follows the proposed format, and gives a starter file if you do not.

A fifteen-minute checklist

You can cover most of the avoidable problems in one sitting.

• Run the AI crawler check on your domain and read which crawlers are blocked and by which line.
• Open yourdomain.com/robots.txt yourself. If you see a blanket Disallow: / under User-agent: *, find out why it is there.
• View the source of your home page and confirm that your key text (what you do, who for, where) is in the HTML, not added later by script.
• Check that every important page has its own clear title and a single H1 heading.
• Add Organization structured data to the home page with your real name, URL and contact details, and make sure they match the page.
• Check that public pages do not sit behind a login, a cookie wall that blocks bots, or a firewall rule that blocks unknown visitors.
• Optional: publish a short llms.txt that links to your most useful pages.

What no tool can tell you

A clean AI readiness result means the common technical obstacles are out of the way. It does not mean you will be cited. Assistants differ from each other, change over time and sometimes answer differently to the same question twice.

Treat the checks as hygiene, in the same way a working sitemap is hygiene for search. The things that decide whether you are worth citing are older: pages that answer real questions clearly, accurate facts about your business, and a site people link to and trust.