Who fetches llms.txt? Agents, not AI search crawlers
Many llms.txt files are published in the hope of appearing in ChatGPT or Perplexity answers. The measured requests point elsewhere, and that changes what belongs in the file.
Last updated:
The measured breakdown
Ahrefs classified every user agent that requested /llms.txt across 137,210 domains during May 2026, sorting them into twelve categories. Only 3% of the roughly 38,000 accepted files received any request at all, about 1,100 files in this technical panel. The breakdown below describes those requests, not the full population and not what a model ultimately used.
Four categories sit above every individual AI category:
| Category | Share of requests |
|---|---|
| SEO audit tools | 21.7% |
| Other and unidentified | 14.9% |
| General web crawlers | 13.1% |
| Tech profiling tools | 11.6% |
Combined, named AI bots account for 19.5%. Split by what they are for, the picture sharpens considerably:
| AI bot type | Share |
|---|---|
| AI agents and agentic infrastructure | 10.5% |
| AI training crawlers | 5.3% |
| AI assistants | 2.5% |
| AI retrieval bots | 1.1% |
A further 12% comes from an industry studying the convention rather than consuming it: GEO and AEO scoring tools at 5.8%, dedicated llms.txt scanners and directories at 3.6%, research crawlers at 2.7%.
One comparison makes the scale legible: Slackbot, a link-preview bot, fetched llms.txt files more often than PerplexityBot did.
Agents lead among the measured AI fetches
The bottom row of that second table is the one to sit with. AI retrieval bots, the crawlers that fetch pages to answer a live user query in an AI search product, such as OAI-SearchBot and PerplexityBot, made 1.1% of all measured requests. Ahrefs report that OAI-SearchBot, PerplexityBot and Claude’s search crawler made only a couple of hundred fetches combined.
Meanwhile AI agents and the infrastructure serving them made 10.5%, and Claude-Code out-fetched every retrieval bot, every assistant and every training crawler in the dataset.
This matches, rather than contradicts, what Google has said. As we cover on does Google use llms.txt, John Mueller’s public position is that no AI service has said it uses the file, and he compared it to the keywords meta tag: a claim by a site owner about itself, which a system can verify more cheaply by reading the site directly.
Training crawlers are the second AI category at 5.3%, fetching llms.txt nearly five times more often than retrieval bots do. If the file influences anything, the data points upstream, at corpus construction, rather than at the moment an answer is assembled.
What that means for what you write
If an intended consumer is a coding agent working on behalf of a developer, several of the usual instincts are wrong.
Marketing pages are the wrong entries. An agent asked “how do I authenticate against this API” needs the reference page, the error codes and a working example. Your pricing page does not help it. Our best practices make the general case; the readership data makes it concrete.
Structure beats breadth. The specification is a curated index, not a sitemap. A file that lists forty pages saves an agent nothing, because it still has to choose. Twelve well-described entries do the work.
Descriptions carry the signal. Each link’s description is the only thing an agent sees before deciding whether to spend a fetch. “API reference” is close to useless. “REST endpoints, auth headers, rate limits and error codes” is a routing decision.
Consider llms-full.txt if your docs are the product. Where the whole point is to hand an agent the content rather than a map, the companion file exists for that; see what llms-full.txt is.
And, per the finding we covered in no AI agent goes looking for your llms.txt, none of this matters until something tells the agent the file exists.
Every number here is a ceiling
Ahrefs are explicit on this, and it should be repeated rather than buried: a request in a log proves that a client retrieved the bytes. It proves nothing about whether the model read them, used them, or discarded them. Their 19.5% AI share is therefore the most generous possible reading of AI consumption, not an estimate of it.
Their population also skews technical, being drawn from their own analytics customers, which is why they present their 28% adoption figure as an upper bound. Our own adoption panel measures a different population entirely and is not comparable.