llms.txt vs robots.txt, what's the difference?
Two files, two different jobs. One publishes advisory crawl rules; the other offers a curated map. Neither is an access-control mechanism.
Last updated:
The short answer
robots.txt publishes crawl rules for compliant clients such as Googlebot or Bingbot. It communicates which URLs they should or should not fetch, but it is not authorization or technical access control.
llms.txt is a context layer. It tells AI language models which pages on your site are most useful for understanding what you do. It is not an access control mechanism at all, it does not grant or deny crawl permission. It is a curated reading list written in Markdown, proposed in September 2024 by Jeremy Howard at Answer.AI.
Neither file replaces the other. They operate at different layers of the web stack and serve different audiences.
What robots.txt does
The robots.txt file lives at the root of a domain (/robots.txt) and
uses the Robots Exclusion Protocol (REP) to communicate crawl permissions. A typical entry looks
like this:
User-agent: Googlebot
Disallow: /private/
Allow: /The key characteristics of robots.txt:
- Audience: Web crawlers of all kinds, search engine bots, AI training crawlers, link checkers, archiving bots.
- Format: Plain text. Key-value pairs using a defined syntax (User-agent, Allow, Disallow, Crawl-delay, Sitemap).
- Purpose: Crawl policy. It tells compliant crawlers what they should and should not fetch.
- Effect: Compliant crawlers will not request Disallowed URLs. Non-compliant crawlers may ignore it.
- Standard status: Widely adopted industry convention; an IETF informational RFC (RFC 9309) was published in 2022.
- What it does NOT do: robots.txt does not tell crawlers what your content means, what your most important pages are, or what context they should use when answering questions about you.
What llms.txt does
A llms.txt file can live at the domain root or a more specific path such as
/docs/llms.txt. It is a Markdown document, not a key-value configuration file. A
minimal valid file looks like:
# My Product
> A short, factual description of what this site is about.
## Documentation
- [Getting started](https://example.com/docs/start/): first steps for new users.
- [API reference](https://example.com/docs/api/): complete endpoint documentation.
## Optional
- [Blog](https://example.com/blog/): articles and updates.The key characteristics of llms.txt:
- Audience: Compatible agents, RAG pipelines and developer tools with explicit support for the convention.
- Format: Markdown. A required title (H1), an optional blockquote description, and H2 sections containing Markdown link lists.
- Purpose: Context and curation. It points compatible clients to selected resources; it does not guarantee accurate answers or downstream use.
- Effect: Tools that read llms.txt use it as a starting point for fetching your content. It does not grant or restrict access.
- Standard status: A community proposal. Not an IETF or W3C standard. Maintained at llmstxt.org by Jeremy Howard.
- What it does NOT do: llms.txt does not control who can crawl your site. It does not improve your Google rankings. It is not a robots.txt replacement.
Side-by-side comparison
| Attribute | robots.txt | llms.txt |
|---|---|---|
| Location | /robots.txt | /llms.txt or a scoped path |
| Format | Plain text, key-value pairs | Markdown |
| Primary audience | All web crawlers | AI/LLM clients and agent frameworks |
| Function | Advisory crawl rules | Context and curation (what to read) |
| Enforces crawl access? | No | No |
| Affects Google ranking? | Indirectly (blocking crawl prevents indexing) | No |
| Formal standard? | IETF RFC 9309 | Community proposal (llmstxt.org) |
| Year introduced | 1994 | 2024 |
| Required sections | User-agent + Disallow/Allow | H1 title (everything else optional) |
Why they are complementary
The crawl-policy layer and the optional resource-map layer can coexist without conflict. A careful deployment looks like this:
- Use robots.txt to express crawl preferences. If you want compatible AI systems to read your content, make sure you have not accidentally blocked AI crawler user-agents in robots.txt. Many sites added blanket bot blocks during the 2023–2024 AI training data controversy; review your rules to ensure legitimate retrieval crawlers (as opposed to training crawlers, if you wish to distinguish) can access your public pages.
- Use llms.txt for a defined compatible client. The map can offer a curated shortcut, but publication does not prove the client discovers, reads or follows it.
- Make sure llms.txt itself is not blocked. If your robots.txt disallows bots from
your root, they may not be able to fetch
/llms.txteither. Verify that your robots.txt does not prevent access to the file you want AI systems to read.
Frequently asked questions
Does llms.txt replace robots.txt?
No. robots.txt publishes advisory crawl rules for compliant clients. llms.txt offers a curated map to compatible agents. Use either file only for the problem it actually solves.
Can I use llms.txt to block AI crawlers?
No. llms.txt has no access-control function. To block specific AI crawlers, add their user-agent strings to your robots.txt Disallow rules. For example, to block GPTBot:
User-agent: GPTBot
Disallow: /llms.txt is not read as a permission document. Publishing it does not grant crawlers any additional access they did not already have.
Do AI crawlers respect robots.txt?
Providers publish crawler-specific robots.txt guidance, but REP remains advisory. A
Disallow rule communicates intent to compliant clients. If content must not be fetched,
protect it with authentication, network policy or server-side authorization.
Continue reading
- llms.txt vs robots.txt vs sitemap.xml, full three-way comparison.
- AI crawlers explained, how GPTBot, ClaudeBot, and PerplexityBot work.
- How to create llms.txt, step-by-step with templates.
- Validator, check your existing llms.txt file.