Best practices

Ten rules

1. Curate. A short list of high-signal pages beats a long list of mediocre ones. Ten to thirty links can be a useful starting heuristic, but the intended tasks should determine the count.
2. Use absolute URLs. Always https://yourdomain.com/.... Relative URLs are technically allowed but fragile.
3. Group by product surface. Sections like Product, Pricing, Developers reflect how a user (and an LLM) thinks. Avoid blog/doc/guide buckets unless they map onto your real navigation.
4. Keep the summary factual. The blockquote after the H1 should read like a Wikipedia opener, not like a landing-page hero.
5. One sentence per item. The colon-prefixed note is for disambiguation, not for marketing.
6. Use the Optional label sparingly. It is a clear editorial home for press, brand assets and archives, but v2 gives it no special machine semantics.
7. Mirror your stable URLs. If a page in llms.txt moves, update or redirect it. Stale URLs poison the file’s reputation.
8. Publish llms-full.txt only for a defined ingestion need. Consolidated documentation can help a known consumer, but it also increases size, freshness and security costs.
9. Run the validator in CI. A content migration that breaks your file should fail the build.
10. Date your file. A short note like “Last reviewed 2026-04-01” in the body is helpful for both humans and crawlers.

Common mistakes

• No H1. The H1 is the only required element. Without it, the file is invalid.
• Multiple H1s. Use H2s for sections. There must be exactly one H1.
• Custom front matter. YAML and JSON headers sit outside the published grammar and can make conforming parsers reject or misread the file.
• Pasted Markdown tables or images. Keep the preamble focused and use link lists inside H2 sections. Extra structures add parsing ambiguity and often duplicate linked pages.
• Including auth-gated URLs. If a page requires login, do not list it, the LLM will hit a wall.
• Overlong descriptions. “The world’s most advanced AI-powered platform for next-gen synergistic transformation” helps no one. Keep each note only as long as needed to distinguish the destination.
• Listing 500 URLs without scope. Recheck the tasks, split real content boundaries into path-level files, or provide a separate full-content resource for a known consumer.
• Blocking the resource unintentionally. Make sure each declared root or path-level file is reachable under the crawl policy you intend.

Multilingual sites

The proposal does not prescribe one internationalization architecture. Two design choices are common:

1. One default-language file at the root. The simplest option when the listed resources and intended consumers share a language.
2. Per-locale variants. Serve /llms.txt (default), /fr/llms.txt, /es/llms.txt. Link to them from your root file’s body or under an Optional section, or declare the applicable file with rel="describedby" on localized pages.

Whichever pattern you pick, do not duplicate URL sets across locales: each variant should point to the localized version of each page.

Security and privacy

• Everything in llms.txt is public. Treat the file as broadcast.
• Never list staging or preview URLs. Any client that retrieves the public file can see them.
• Do not list URLs with secrets in query strings. This sounds obvious; we have seen it happen.
• If the page exposes user data behind auth, it does not belong here.
• Audit the file at every release. A leaked draft URL is the most common security mistake.

Treat the file like external configuration, without assuming agents obey it. In the Ahrefs analysis of 137,210 domains, the largest user-agent string in the research category was prompt-injection-survey/1.0. That label proves neither an attack nor its operator, but it is a useful reminder to keep content factual, review changes and constrain any consuming agent.

Automation in CI

Treat llms.txt like any other artifact: generate, validate, and gate releases on it.

• Generate it from your content source (CMS, MDX collection, database).
• Run the validator in CI; fail the build on any error.
• Diff the file across releases; alert the docs owner on large deletions.
• Smoke-test the production URL after deploy: curl -fsS https://yourdomain.com/llms.txt | head -1.

Next

• Benefits and limitations, what to expect, what not to.
• Real-world examples, copy what works.
• Validator.