llms.txt vs robots.txt, sitemap.xml, and llms-full.txt

TL;DR

They are not interchangeable. robots.txt publishes crawl rules that compliant clients may apply. sitemap.xml tells search engines what exists. llms.txt offers compatible clients a curated resource map. llms-full.txt hands them the actual content.

Side-by-side matrix

llms.txt vs robots.txt

robots.txt is a crawler-instruction file standardized as IETF RFC 9309. It uses the Robots Exclusion Protocol grammar (User-agent, Disallow, Allow, Sitemap) to express which paths compliant crawlers should or should not fetch. It does not enforce access and is not an authorization layer.

llms.txt is the opposite intent: a positive recommendation list. It does not block anyone, it does not grant access, and it has no effect on whether a crawler fetches anything else on your site. It just says: if you are an LLM client, here is the high-quality subset.

Practical implication: continue to use robots.txt for crawl preferences and sitemap discovery. Use authentication or network controls for actual access restrictions. Add llms.txt as a complement only when it has a maintained audience and purpose.

llms.txt vs sitemap.xml

sitemap.xml aims for completeness: it lists every URL you want a search engine to know about, plus metadata (lastmod, priority, alternate languages). It is XML, machine-only, often auto-generated.

llms.txt aims for curation: a small Markdown list of the pages an LLM should read first. It does not replace your sitemap. It rarely includes more than a few dozen URLs, while a sitemap on a content-heavy site can list hundreds of thousands.

Think of sitemap.xml as a directory and llms.txt as a recommended-reading shelf curated by a librarian.

llms.txt vs llms-full.txt

Same family, different role:

• llms.txt is the map: a list of titled links.
• llms-full.txt is the territory: the actual content of those (and other) pages, concatenated as Markdown into one file.

The llms-full.txt convention was popularized by Mintlify in collaboration with Anthropic. It can give a developer or tool one URL for a consolidated documentation corpus. That convenience must be weighed against freshness, size and prompt-injection exposure.

Publish llms.txt when a curated map solves a real task. Add llms-full.txt only when bulk textual ingestion is an explicit requirement.

llms.txt vs schema.org / JSON-LD

Schema.org is a vocabulary for marking up the meaning of individual pages in JSON-LD or microdata. Search engines and other software use it to extract structured facts: a product’s price, a recipe’s ingredients, a FAQ’s questions and answers.

llms.txt operates one level above: it’s a site-wide map, not a per-page enrichment. The two are complementary. Schema.org tells an LLM what a page is; llms.txt tells it which pages to look at first.

How to combine them

1. Publish both robots.txt and sitemap.xml as you already do for SEO.
2. Add a root or path-scoped llms.txt for a defined compatible client.
3. Optionally add llms-full.txt if your site is documentation- or knowledge-heavy.
4. In robots.txt, leave /llms.txt and /llms-full.txt accessible (do not Disallow them).
5. Keep schema.org markup on individual pages where it makes sense (Product, FAQ, Article…).

Next

• How llms.txt works, the spec in detail.
• Best practices.
• FAQ.