What is llms.txt?

Definition

llms.txt is a Markdown file published at a website root or a specific path. It gives compatible agents a short, curated map of useful resources under that scope. The familiar root URL is https://example.com/llms.txt.

It was proposed by Jeremy Howard at Answer.AI in September 2024 as a community convention, not a formal standard. The canonical specification lives at llmstxt.org.

Origin

Howard’s original argument: large language models have small context windows compared to a typical website. Asking a model to crawl an entire site every time someone wants to ask a question about your product is wasteful and lossy. Sites already publish robots.txt for crawlers and sitemap.xml for search engines, llms.txt proposes a parallel convention specifically for LLMs.

The format is intentionally minimal:

• Markdown (so a human can read and edit it).
• Predictable structure (so a parser can reliably extract sections and links).
• Curated, not exhaustive (the goal is signal, not coverage).

What problem it solves

1. Context window limits. Even with multi-million-token models, loading a full documentation site costs latency and money. A focused llms.txt can reduce unnecessary retrieval when a compatible client needs only a small set of canonical resources.
2. Discovery noise. A site’s most valuable content for an LLM is rarely the same as its most visited content. llms.txt lets you tell models explicitly: “these are the pages worth reading.”
3. Reviewable contracts. When the file is version-controlled and checked in CI, its URLs become an explicit surface that can be tested when pages move.
4. Companion file llms-full.txt. A second file (sibling convention developed by Mintlify and Anthropic) inlines the actual content of the linked pages as concatenated Markdown, so an LLM client can load a single URL and get the full, ready-to-feed corpus.

What llms.txt is not

Plenty of confusion exists in the wild. To be clear:

• It is not a W3C, IETF, or otherwise standardized protocol. It is a community proposal.
• It is not enforced by any major LLM provider. No vendor (OpenAI, Anthropic, Google, Meta) has publicly committed to consistently fetching it. Adoption today is mostly opportunistic.
• It is not a ranking signal. Publishing llms.txt does not make your site rank higher in classic search nor in AI Overviews. There is no public evidence to that effect as of August 2026.
• It does not replace robots.txt, sitemap.xml, or schema.org. See the dedicated comparison page.
• It is not a security mechanism. Anything you list in llms.txt is, by definition, public.

Adoption status (2026)

Adoption is real but uneven. Most early adopters are documentation platforms and developer-tools companies: Anthropic, Cloudflare, Vercel, Stripe, Mintlify, Perplexity. Outside the dev-tools world, coverage is still thin.

Our fixed technology-oriented panel found 113 valid root files among 218 reachable hosts on 2026-08-03. This is a transparent panel measurement, not a Web-wide adoption estimate.

Skepticism is also documented. Google’s John Mueller has been publicly cautious, and Mintlify themselves have acknowledged the skepticism head-on: the file is useful as a stable, machine-readable contract, but its real-world consumption by LLMs depends on each vendor’s evolving choices.

Typical use cases

• API documentation. Map an LLM directly to your endpoint reference pages and tutorials.
• SaaS marketing site. Help an assistant answer questions about your product, pricing, and integrations from a small set of canonical pages.
• Open-source project. Point clients to the README, contribution guide, examples, and changelog.
• Knowledge base. Surface the small, high-quality answers rather than every help-center article.

Quick FAQ

Is llms.txt required? No. It is voluntary.

Will it improve my SEO? Not directly. Read our benefits and limitations page for a careful answer.

How do I create one? See the step-by-step guide, or use the generator to scaffold a baseline.

Continue reading

• How llms.txt works, file structure and parsing rules.
• Comparison with robots.txt and sitemap.xml.
• Best practices, ten rules and common mistakes.