Before an AI system can recommend your business, it has to do something less glamorous: read your website.

That sounds obvious until you start checking real sites. Menus live in images. Product details render only after JavaScript runs. Hours appear in a booking widget but not on the page. A robots.txt file blocks the wrong crawler. The sitemap is stale. The “About” page has brand poetry but no clear sentence saying what the company does.

For humans, the site looks fine. For an answer engine, it is fog.

This is why AI crawlability is the first gate in AEO. If the machine cannot access the page, parse the facts, and connect them to the entity, the rest of your content strategy is downstream of a blocked pipe.

The first gate is access.

Every answer engine has to source information somehow. Some knowledge comes from model training. Some comes from search indexes. Some comes from live user-triggered browsing. Some comes from partnerships, feeds, and APIs. The exact mix changes by product and by query.

But the operational question for a site owner is simple: when AI systems or their search layers try to reach your public content, do you let them in?

OpenAI documents multiple crawlers and user agents, including GPTBot and OAI-SearchBot, and says site owners can use robots.txt controls to manage how their content works with OpenAI products. Anthropic documents separate Claude crawlers, including ClaudeBot, Claude-User, and Claude-SearchBot, and warns that blocking search or user-fetch agents may reduce visibility in Claude search results or user-directed retrieval. Perplexity documents PerplexityBot and recommends allowing it in robots.txt if you want your site to appear in Perplexity search results.

Google’s situation is more nuanced. Google-Extended is a robots.txt token used to control whether content helps improve Gemini Apps and Vertex AI; it does not use a separate HTTP user agent string. The fetch happens through existing Google user agents, while Google-Extended is used as a control token.

That is the new crawlability reality: “allow Googlebot” is no longer the whole conversation.

Do not copy someone else’s robots.txt strategy.

Some publishers block AI training crawlers for business reasons. Some brands allow search-oriented AI crawlers but block training crawlers. Some block everything until licensing terms are clearer. Some allow everything because discovery matters more to them than training control.

There is no universal answer. There is only a decision that should be made intentionally.

The bad version is accidental blocking. That happens when an old security rule, plugin, CDN setting, or copied robots.txt snippet blocks agents you actually want. It also happens when teams block a broad class of bots without separating training, search indexing, and user-requested retrieval.

The practical AEO question is not “should every AI crawler be allowed?” It is:

Which AI surfaces do we want to appear in?
Which bots feed those surfaces?
Which bots are for training versus search versus live user fetches?
What does our policy say?
Does our robots.txt actually match that policy?

If nobody can answer those questions, the site has a governance problem, not just a technical SEO problem.

The second gate is readable HTML.

Access is not the same as readability.

An AI crawler may be allowed to fetch your homepage and still fail to extract the facts that matter. The most common reason is that the page was built for visual impact rather than machine legibility.

Here are the patterns that cause trouble:

Important text is embedded inside images.
Menus or spec sheets are PDF-only.
Product price, stock, shipping, or returns load after client-side JavaScript.
The page uses vague headlines with no plain explanation nearby.
The business name, address, and phone number differ across pages.
Key pages are blocked behind popups, login walls, app-like shells, or broken redirects.
The sitemap omits pages that actually answer buyer questions.
The content is split across widgets crawlers may not execute.

Search engines have become good at rendering modern websites, but not every AI retrieval path behaves like a full browser. If your facts only exist after a heavy app loads, you are making every downstream system work harder.

The AEO-friendly version is simple: put the critical facts in stable, crawlable HTML. That does not mean your site has to look plain. It means the machine-readable layer should not depend on a human seeing the design.

The third gate is entity clarity.

Once a system can read the page, it has to understand what the page is about and who is responsible for it.

This is where a surprising number of sites get vague. A SaaS company says it is “building the future of customer intelligence” but never says it is a call analytics platform for franchises. A restaurant says “seasonal, soulful, unforgettable” but hides cuisine, neighborhood, hours, booking links, private dining details, and dietary options. An ecommerce brand has gorgeous product photography but inconsistent category names and thin product copy.

Humans can infer. Machines should not have to.

A clear entity layer answers:

What is the organization called?
What does it sell or provide?
Who is it for?
Where is it located or available?
What pages prove those facts?
Which social, marketplace, review, and knowledge profiles refer to the same entity?
What should the model call this business in an answer?

Structured data helps, but it is not a substitute for visible clarity. Google’s structured data guidelines say structured data should represent the main content of the page and should not mark up hidden or misleading content. That principle is even more important for AI answers: if the JSON-LD says one thing and the page says another, the system has to decide which source to trust.

The fourth gate is answer readiness.

Readable content is not automatically answer-ready.

Answer-ready content is organized around the questions people ask. It defines terms. It compares options. It gives direct explanations. It includes specifics. It makes the useful part easy to extract.

For a restaurant, answer-ready content might include:

What kind of food do you serve?
Are you good for groups?
Do you take reservations?
Do you have vegetarian, gluten-free, or kid-friendly options?
What neighborhood are you in?
Are you open late?
Is there parking?
What are the best dishes for a first visit?

For ecommerce, it might include:

What problem does the product solve?
What size, material, compatibility, and warranty details matter?
What is in stock?
What are shipping and return rules?
How does this compare with the common alternatives?
Who should not buy it?

For SaaS, it might include:

Who is the product for?
What integrations does it support?
What workflows does it replace?
What is the pricing model?
What are the limits?
How does it compare with named alternatives?
What evidence supports the claims?

This is not filler content. It is source material for answers.

The fifth gate is citation confidence.

An answer engine does not only need facts. It needs confidence that the facts are safe to repeat.

Citation confidence comes from consistency, specificity, corroboration, and source quality. If your own site says one thing, your Google Business Profile says another, your menu platform says a third, and a review site has an old address, the model has a reconciliation problem. If your product page claims broad superiority without evidence, the model may prefer a third-party review or documentation page. If your article makes statistical claims without links, it is less useful as a source.

The GEO research paper on generative engine optimization found that content changes such as adding citations, statistics, and authoritative language could improve visibility in generative engine responses, with gains up to 40% across diverse queries in the authors’ benchmark. The exact number should not be treated as a guarantee for any one site, but the direction is instructive: answer engines need evidence-rich passages.

Good AEO content does not just say “we are trusted.” It shows the facts, links the sources, and names the context.

A practical AI crawlability checklist.

Five gates, in order — a failure at any one of them makes the next irrelevant.

The five gates

FIG. 01

Access
When AI systems or their search layers try to reach your public content, do you let them in?
Readable HTML
Do the important facts exist in stable markup, or only after a heavy app finishes loading?
Entity clarity
Once a system can read the page, does it know what the page is about and who is responsible for it?
Answer readiness
Does the content resolve a real question specifically enough to be quoted?
Citation confidence
Do your own sources, listings and third-party profiles agree well enough to be trusted?

The rest of this checklist works through them one file at a time.

Start with the boring files.

Check robots.txt.

Open https://yourdomain.com/robots.txt. Look for rules that block GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, or Google-Extended. Decide which rules are intentional. Document the reason.

Check sitemap.xml.

Open your sitemap. Make sure the pages that explain your business, products, locations, menus, pricing, docs, FAQs, and comparisons are included. A sitemap does not guarantee discovery, but it reduces guesswork.

Check llms.txt if you use it.

llms.txt is not yet a universal standard, but many technical teams use it as a machine-readable orientation file for AI agents. If you publish one, keep it accurate. Do not let it become a stale brochure — and do not expect it to do a job robots.txt or the sitemap already does. llms.txt is a map, not a magic wand covers what the convention does and does not grant.

Fetch the page without a browser.

Use a simple HTTP fetch or “view source” check. Can you see the important text in the initial HTML? If the page is empty until JavaScript runs, decide whether the critical facts should be rendered server-side or duplicated in crawlable sections.

This is a diagnostic, not a model of how every product reads the web. Some systems render like a full browser, some read a search index built elsewhere, and some issue a plain fetch at the moment a user asks. A raw fetch simply shows you the least capable path — which is the one worth designing for.

Check the facts a model would need.

For each important page, write down the five facts an AI answer would need to mention you accurately. Then verify those facts appear in visible text, not only in design assets.

Validate structured data.

Use JSON-LD where it fits. Google recommends JSON-LD for structured data when possible because it is easier to implement and maintain at scale. Make sure it matches the visible page.

Run prompts across models.

After the technical layer is clean, ask the models buyer-like questions. Do not judge from one answer. Look for repeated failure patterns: wrong facts, missing facts, competitor citations, or no citation at all.

Re-test after changes.

If you fix crawlability or content, re-run the same prompts and compare. One-off checks are noisy. Repeatable measurement is where the signal appears.

What Omni Impact checks first.

The free Omni Impact AEO Readiness Check exists because most teams do not know where to start. It runs a baseline crawl in about 60 seconds and scores a site from 0 to 100 across five categories:

AI crawlability
Entity understanding
Structured data
Answer readiness
Citation confidence

Under the hood, it checks 14 signals. It is not a full AEO audit, and it does not pretend to know every model’s private retrieval logic. It answers the first practical question: can answer engines read enough of this site to understand and cite it confidently?

That is the correct first question. You cannot optimize the answer layer if the answer layer cannot read the source.

The bottom line.

AI visibility starts before the prompt. It starts with access, readable HTML, clear facts, consistent structured data, and evidence-rich pages that a system can safely use.

Most crawlability problems are not mysterious. They are the same small failures repeated across modern websites: blocked agents, hidden content, thin pages, vague copy, stale sitemaps, contradictory facts, and structured data nobody maintains.

Fixing those issues will not guarantee a recommendation. Nothing can. But it gives answer engines something real to work with.

That is the first job of AEO: make the truth about your business easy for machines to find, parse, trust, and repeat.

Can AI Even Read Your Website? The Crawlability Checklist for Answer Engines