llms.txt: what actually moves AI citations

The pitch now arrives as cold outreach and agency line items: add llms.txt, get cited by AI assistants. It sounds plausible, because a file exists, robots.txt exists, and AI crawlers exist. The evidence caught up in 2026, and it mostly went the other way. Ahrefs measured server logs across 137,210 domains in May 2026 and found that almost nobody fetches the file. Google’s own guidance says you don’t need it for AI search. This post sorts the file’s real job from the pitch, and the fix list costs nothing.

What llms.txt actually is

llms.txt is a small markdown file that lives at your site root. It states what the site is and links the pages that matter, while the agent reads it rather than crawling. Jeremy Howard, who co-founded Answer.AI and fast.ai, proposed the format in September 2024. The spec has since reached version 2 (August 10, 2026), which is worth reading because it narrowed the claim. The spec now says the files are used most heavily for software documentation. Coding agents follow them to find API references and tutorials. It also recommends markdown versions of pages, plus a Link header that points agents at the index.

Two things it isn’t. It isn’t a directive, because the spec is explicit that the file controls nothing and blocks nothing, unlike robots.txt. And it isn’t a documented ranking input. No AI platform has ever said its search or assistant product reads the file. That gap is exactly what the numbers below measure.

Plenty of the AI-native web publishes one anyway, and the split is instructive. On September 7, 2026, I fetched eight. The docs domains for Claude, Anthropic, Mintlify, Next.js, Svelte and Mistral all return a live file. Anthropic’s main marketing homepage returns a 404. That pattern matches the spec’s own framing, because it is a documentation tool that docs platforms generate automatically.

The log data: 97% of the files are never read

Ahrefs published the first large measurement on June 15, 2026. The team analyzed 137,210 domains that received traffic in May 2026, then checked who ever requested the file. Adoption is real, since 28% of those domains publish the file. The authors flag that number as an upper bound, because their analytics customers skew technical. Readers are not. Ninety-seven percent of those files received zero requests in May, from anything: no bots, no humans, nothing. Files that don’t exist don’t get probed either. The 404 traffic was 98% human, mostly SEOs checking competitors by hand.

The 3% of files that do get fetched tell the sharper story, because their readers are mostly not AI systems.

Among the named AI tools, GPTBot leads and Claude-Code, the coding agent, is second. Both outrank every AI search and assistant bot, and Slackbot fetched llms.txt files more often than PerplexityBot did. That gap is a fair proxy for how much AI search actually cares. Twelve percent of fetches came from GEO and checker tools. The industry studying the file out-read the search engines it was written for, which nobody predicted. Chrome’s Lighthouse llms.txt audit produced roughly one fetch per thousand sites.

Google managed to take both sides in a single week in late May 2026. Its guidance and its tooling disagreed, and the logs are where the question gets settled. Its guide on optimizing for generative AI features has a section titled mythbusting. That section says machine-readable files like llms.txt aren’t needed to appear in AI search. Days later, Chrome shipped an llms.txt check inside Lighthouse’s experimental agentic-browsing audits. SEO Lily Ray pressed Google’s John Mueller on the contradiction. He called the file “not done for search” and “a temporary crutch” for coding tools parsing documentation.

The cats.txt problem: the pitch’s proof proves nothing

Here’s why the pitch survives despite all that, and it’s the part worth remembering at contract time. Mark Williams-Cook, an SEO consultant, invented a standard called cats.txt. It is a root-level text file declaring your office cats, their jobs, and a PurrLevel score out of ten. Then he ran it against the exact four proofs the llms.txt pitch cites, and it cleared all of them. PerplexityBot, GPTBot and ClaudeBot fetched the file about his cats. Google indexed it. A Google AI Overview confidently cited Odd, a cat who does not exist, complete with a PurrLevel rating. ChatGPT endorsed cats.txt as a ranking tactic within two weeks. It reversed itself once the discourse caught up, while the file never changed.

His diagnosis explains every GEO pitch you’ll see this year. When you ask a model whether a tactic works, it isn’t running an experiment. It’s returning the average of what the web already says. Confidence is the product, not proof.

The practical move follows directly: when a proposal or a tool claims llms.txt results, ask for two things. Ask for log lines showing AI agents fetching the file. Ask for citations that appeared after publication which didn’t exist before. If the answer is “the bots crawled it” or “ChatGPT said it helps”, that’s the bar a cat file cleared.

What I checked on this site, and what I can’t see

This site ships an llms.txt, and I’m keeping it, which needs some explaining after two debunking sections. On September 7, 2026, /llms.txt here returns 200 as plain text, while robots.txt allows every user agent, AI crawlers included. The six docs domains above returned the same. What I can’t do is count who fetches my file, because this publishing setup has no raw access-log view. The Ahrefs numbers stand in for that, and I’m not going to invent first-party counts.

So why keep it? Because the honest case is the spec’s own case. The file costs nothing, it misleads no one, and if a provider ever documents support, the work is already done. What I won’t do is sell it as a citation lever. When an agency pitches it as one, the questions above are what I’d bring to the call.

The plumbing that actually gets you read

Skip the file debate and look at what AI systems demonstrably fetch: your pages. The audits agree on which layers decide whether those fetches produce anything usable, and none of them cost money. A vendor-run audit of 1,500 sites, published in January 2026, found 30% blocking AI bots in robots.txt by accident. The usual causes are stale staging rules or security plugins. The same audit found 70% with zero schema markup and 40% hiding core content behind client-side JavaScript. Sixty percent broke heading hierarchy for styling. Each failure is checkable in an afternoon, which is more than anyone can say for the file everyone is selling.

• robots.txt allows GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and the rules get audited quarterly.
• Core content renders in the raw HTML, because many retrieval agents skip JavaScript entirely.
• Headings follow one clean hierarchy with a single H1, since models chunk pages by outline.
• Schema marks up the organization and the content, so your brand resolves to an entity.
• Pages answer real questions with checkable claims, because citations attach to content, not plumbing.

The citation layer has its own economy, and most of it is content-side rather than plumbing-side. I covered the citation-farm economy in the Perplexity post, and what AI Overviews reward in the content workflow post. The short version: engines cite pages that answer questions cleanly and can be verified. That is the same thing Google rewarded before the chatbots arrived. The same discipline argument runs through programmatic SEO done with discipline: fewer pages, each one worth fetching.

Filed under SEO & Content. More in the SEO & Content category and about this blog.