llms.txt blog
In-depth articles on the llms.txt spec, AI crawlers, generative engine optimization, and practical implementation guides for developers and site owners.
-
Adoption is not usage: reading the llms.txt studies properly
Why llms.txt adoption measures range from 10.13% to 51.8%: each study samples a different population, and publication is not consumption.
Read article → -
Threat-modeling llms.txt prompt injection
Ahrefs observed a research user agent named prompt-injection-survey. The label does not prove an attack, but external agent context deserves supply-chain controls.
Read article → -
Who fetches llms.txt? Agents, not AI search crawlers
In Ahrefs logs for 137,210 domains, AI retrieval bots made 1.1% of llms.txt requests and agentic tools 10.5%. Fetching still does not prove use.
Read article → -
What Chrome's Lighthouse llms.txt audit actually checks
Lighthouse treats a 4xx response as Not Applicable. For a successful response it also checks minimum length, one H1 and one Markdown link, but not full v2 conformance.
Read article → -
AI bots did not probe missing llms.txt files in one study
In Ahrefs' May 2026 sample, no AI bot requested a missing /llms.txt. The result is not universal proof, but it shows why publishing alone is not discovery.
Read article → -
State of llms.txt adoption, June 2026: measured data
We fetch and parse /llms.txt on a fixed panel of 219 notable hosts every week. Around half serve the file, with sharp differences by sector. Raw data published.
Read article → -
llms.txt vs robots.txt, what's the difference?
robots.txt publishes crawl rules; llms.txt offers compatible agents a curated map. Learn how the files differ in format, purpose and enforcement.
Read article → -
GEO: Generative Engine Optimization, what it is and why it matters
Generative Engine Optimization (GEO) is the practice of optimizing content to appear in AI-generated answers. Learn how it differs from traditional SEO and where llms.txt fits in.
Read article → -
llms.txt best practices, how to write a file AI systems actually use
A practical DO/DON'T guide: use absolute URLs, write factual descriptions, keep it concise, and update regularly. Everything you need to write a file that works.
Read article → -
AI crawlers explained, how Perplexity, ChatGPT, and Claude read your site
AI crawlers differ from Googlebot: they build context, not just indexes. Understand what GPTBot, ClaudeBot, and PerplexityBot do and how llms.txt helps.
Read article → -
llms.txt for API-first products
Developer tools and API products benefit especially from llms.txt because developers ask AI assistants about APIs. A guide with a full example template.
Read article → -
llms-full.txt, what it is, when to use it, and how to create one
llms-full.txt is the companion file that inlines your full page content. Learn when it makes sense, how it differs from llms.txt, and who uses it.
Read article → -
How to measure the impact of your llms.txt file
No direct analytics dashboard exists yet, but you can triangulate. Track AI referral traffic, monitor bot fetch logs, and test citation accuracy manually.
Read article → -
llms.txt for documentation sites
Docs sites are the ideal use case for llms.txt. Learn what to include, what to exclude, and how to structure your file so AI assistants give accurate answers.
Read article →