What is llms.txt?

Definition

llms.txt is a Markdown file placed at the root of a website (https://example.com/llms.txt) that provides large language models with a short, curated map of the most useful pages and resources on that site.

It was proposed by Jeremy Howard at Answer.AI in September 2024 as a community convention, not a formal standard. The canonical specification lives at llmstxt.org.

Origin

Howard’s original argument: large language models have small context windows compared to a typical website. Asking a model to crawl an entire site every time someone wants to ask a question about your product is wasteful and lossy. Sites already publish robots.txt for crawlers and sitemap.xml for search engines — llms.txt proposes a parallel convention specifically for LLMs.

The format is intentionally minimal:

• Markdown (so a human can read and edit it).
• Predictable structure (so a parser can reliably extract sections and links).
• Curated, not exhaustive (the goal is signal, not coverage).

What problem it solves

1. Context window limits. Even with multi-million-token models, loading a full documentation site costs latency and money. A 50 KB llms.txt can replace a 50 MB crawl for many tasks.
2. Discovery noise. A site’s most valuable content for an LLM is rarely the same as its most visited content. llms.txt lets you tell models explicitly: “these are the pages worth reading.”
3. Stable contracts. URLs in llms.txt become an explicit, version-controlled surface — if you rename a page, you remember to update the file.
4. Companion file llms-full.txt. A second file (sibling convention developed by Mintlify and Anthropic) inlines the actual content of the linked pages as concatenated Markdown, so an LLM client can load a single URL and get the full, ready-to-feed corpus.

What llms.txt is not

Plenty of confusion exists in the wild. To be clear:

• It is not a W3C, IETF, or otherwise standardized protocol. It is a community proposal.
• It is not enforced by any major LLM provider. No vendor (OpenAI, Anthropic, Google, Meta) has publicly committed to consistently fetching it. Adoption today is mostly opportunistic.
• It is not a ranking signal. Publishing llms.txt does not make your site rank higher in classic search nor in AI Overviews. There is no public evidence to that effect as of April 2026.
• It does not replace robots.txt, sitemap.xml, or schema.org. See the dedicated comparison page.
• It is not a security mechanism. Anything you list in llms.txt is, by definition, public.

Adoption status (2026)

Adoption is real but uneven. Most early adopters are documentation platforms and developer-tools companies: Anthropic, Cloudflare, Vercel, Stripe, Mintlify, Perplexity. Outside the dev-tools world, coverage is still thin.

A 2025–2026 industry survey by SE Ranking on roughly 300,000 domains measured an adoption rate of about 10 %, concentrated in tech. Marketing sites and traditional publishers lag.

Skepticism is also documented. Google’s John Mueller has been publicly cautious, and Mintlify themselves have acknowledged the skepticism head-on: the file is useful as a stable, machine-readable contract, but its real-world consumption by LLMs depends on each vendor’s evolving choices.

Typical use cases

• API documentation. Map an LLM directly to your endpoint reference pages and tutorials.
• SaaS marketing site. Help an assistant answer questions about your product, pricing, and integrations from a small set of canonical pages.
• Open-source project. Point clients to the README, contribution guide, examples, and changelog.
• Knowledge base. Surface the small, high-quality answers rather than every help-center article.

Quick FAQ

Is llms.txt required? No. It is voluntary.

Will it improve my SEO? Not directly. Read our benefits and limitations page for a careful answer.

How do I create one? See the step-by-step guide, or use the generator to scaffold a baseline.

Continue reading

• How llms.txt works — file structure and parsing rules.
• Comparison with robots.txt and sitemap.xml.
• Best practices — ten rules and common mistakes.