llms.txt Is a Map, Not a Magic Wand

llms.txt may be the most overpromised small text file on the web.

The useful version is straightforward: it is a proposed convention for giving AI agents a concise, curated map of a website’s important information and machine-friendly resources.

The inflated version says adding the file makes a site rank in ChatGPT, grants crawler access, feeds every model, or fixes weak content.

It does none of those things by itself.

That does not make llms.txt pointless. It makes the implementation decision easier. Publish one when a clear orientation file would help an agent navigate a documentation set or information-heavy site. Keep it accurate. Test it. Do not treat it as an AI ranking directive.

What the proposal actually says.

Jeremy Howard published the original llms.txt proposal in September 2024. Version 2, updated in August 2026, describes a Markdown file placed at /llms.txt or within a subpath such as /docs/llms.txt.

The file has a small required core:

• One H1 naming the project or site
• An optional blockquote summary
• Optional explanatory text
• H2 sections containing curated lists of links, each with a useful description

The proposal also recommends clean Markdown versions of important pages and standard link relationships that point from HTML to the Markdown alternative and to the applicable llms.txt file.

The design goal is context efficiency. Instead of asking an agent to process a navigation-heavy website or guess which of thousands of URLs matter, the site provides a short orientation document and links to clean detail pages.

That is especially sensible for software documentation, where an agent may need to find one API reference, tutorial, compatibility note, or migration guide.

It is not robots.txt for AI.

The names make confusion almost inevitable.

If your policy is to allow a search crawler, configure that crawler correctly. If your goal is discovery, maintain the sitemap and internal links. If your goal is to give agents a concise map, add llms.txt. The AI crawlability checklist covers the first two jobs, including which agent types a policy decision actually affects.

It is not a guaranteed inclusion signal.

The llms.txt proposal has real adoption. Documentation platforms generate it, AI companies publish it for their own developer documentation, and tools can choose to read it.

Adoption is not the same as a universal ranking guarantee.

Google’s Search Central guidance is explicit: sites do not need new machine-readable files, AI text files, or special schema to appear in AI Overviews or AI Mode. Eligibility still depends on the normal Search foundation, and no compliant page is guaranteed to be crawled, indexed, or served.

OpenAI’s developer documentation now points visitors to an llms.txt index and offers Markdown page variants. That demonstrates a practical use for documentation consumption. It does not mean ChatGPT Search promises to rank every website with the file.

The defensible claim is narrower: some agents and tools use the convention, it can reduce navigation and context cost, and it gives site owners a clean place to curate machine-friendly resources.

It cannot rescue inaccessible or weak pages.

Suppose an llms.txt file links to a pricing page that renders no meaningful content without client-side JavaScript, a policy page returning a 403 to the relevant fetch agent, or documentation that contradicts the product.

The same problem appears when teams put every claim in the orientation file but do not maintain the linked source pages. An agent may read the summary, but users and retrieval systems still need durable evidence behind it. Which system reads what, and when, is the subject of where AI answers actually come from — worth reading before assuming one file reaches every path.

A good file is selective.

The first implementation mistake is turning llms.txt into another sitemap.

An agent does not need 4,000 undifferentiated links. It needs orientation and a path to the most useful evidence.

Keep optional or secondary material in the proposal’s Optional section so an agent working under a tight context budget can skip it.

Markdown alternatives can be more valuable than the index.

The most interesting part of the version 2 proposal is not the root file. It is the recommendation to expose clean Markdown alternatives for important pages and advertise them using rel="alternate" with type="text/markdown".

The orientation file helps an agent choose a page. The Markdown alternative helps it consume that page without spending tokens on navigation, scripts, and presentation markup.

This is most compelling for large technical sites. A normal marketing site with clean server-rendered HTML may gain little from duplicating every page as Markdown. A documentation platform with hundreds of dense references may gain much more.

Use the complexity where it solves a retrieval problem. Do not build a second publishing system only to check an AEO box.

Maintenance is the real cost.

A stale llms.txt file is worse than a missing one when it sends agents toward retired docs, old prices, deprecated APIs, or dead URLs.

Assign an owner. Generate the file from the same content source when possible. Check links in continuous integration. Review the summary when positioning changes. Remove deprecated pages. If Markdown alternatives are generated, ensure their canonical facts stay synchronized with the HTML.

The file should change because the site changed, not because an SEO calendar says it is time to “refresh AI content.”

Test it like an agent would use it.

The proposal recommends a practical test: give an agent only the llms.txt file as a starting point and ask questions about the site.

Record failures. If the agent chooses the wrong page, improve the section or description. If the linked page is hard to parse, fix the destination. If the question should not be answered from public information, do not add sensitive material to make the test pass.

When to prioritize it.

Prioritize llms.txt when:

• You operate substantial public documentation.
• Important content already has clean Markdown versions.
• Agents regularly use the site to complete technical tasks.
• The information architecture is difficult to infer from navigation alone.
• You can generate and maintain the file from a reliable source.

Deprioritize it when:

• Core pages are blocked, thin, contradictory, or unreadable.
• The sitemap and internal links are broken.
• The team cannot keep another surface current.
• The business expects the file to compensate for missing evidence or authority.

Fix the source layer first.

The bottom line.

llms.txt is a sensible, evolving convention for helping agents orient themselves on a website. Its strongest use case is curated access to clean, detailed content — especially documentation.

It is not access control. It is not a sitemap replacement. It is not special schema. It is not a promise of inclusion in Google AI features or a guaranteed ChatGPT citation.

Publish it as a map. Keep the destinations good. Measure whether agents can use it. Ignore anyone selling the file as a magic wand.

Source notes

1. 01 The llms.txt version 2 proposal llmstxt.org (opens in a new tab)
2. 02 OpenAI — Crawler documentation and documentation index developers.openai.com (opens in a new tab)
3. 03 Google Search Central — AI features and your website developers.google.com (opens in a new tab)
4. 04 RFC 9309 — Robots Exclusion Protocol rfc-editor.org (opens in a new tab)
5. 05 Sitemaps protocol sitemaps.org (opens in a new tab)