For the past year, llms.txt has been sold as the new robots.txt for the AI era: add this one file to your root directory and AI search tools will finally "understand" your site. It's a tidy story. It's also not what the evidence shows.
Here's what llms.txt actually is, what's been tested, and where it genuinely earns a place on a website.
What llms.txt is, and where it came from
llms.txt was proposed in September 2024 by Jeremy Howard of Answer.AI, as a way to help large language models work within limited context windows. The idea: a plain-Markdown file at /llms.txt giving an LLM a concise map of a site's key content, so a coding assistant or documentation tool doesn't have to wade through navigation menus, cookie banners and JavaScript to find the page it actually needs.
That's a real, specific problem. It was never proposed as a search ranking signal or an "AI SEO" mechanism, and the standard's own documentation doesn't claim otherwise.
The claim that took over: "you need this to be visible in AI search"
Somewhere between the original proposal and the current SEO content cycle, llms.txt got reframed as essential for AI search visibility, showing up in ChatGPT, Gemini and AI Overviews. That claim is where the evidence falls apart.
Google has directly rejected it. In June 2026, Google's John Mueller called llms.txt "purely speculative for now," noting the format has existed for roughly two years with no confirmation that any AI system actually reads it. He went further, comparing it to the deprecated keywords meta tag: a file entirely controlled by the site owner, with no verification, and therefore too easy to manipulate for any search or AI system to trust.
The usage data backs him up. Ahrefs analysed 137,210 domains in May 2026. Roughly 28% had published a valid llms.txt file. Of those, 97% received zero requests. Of the small remainder that did get hit, the traffic mostly wasn't AI retrieval at all - it broke down as SEO audit tools, general web crawlers and tech-profiling bots checking whether the file existed. Genuine AI retrieval bots accounted for about 1.1% of requests. Perhaps the clearest single data point: Slackbot fetched llms.txt files more often than PerplexityBot did.
No major AI platform - OpenAI, Anthropic, Google or Perplexity - has confirmed that its production search or citation systems read llms.txt.
Where it does hold up
There's one use case the data supports: AI coding assistants working against developer documentation. Tools like Cursor and Claude Code can be pointed at an llms.txt file to pull clean, ad-free technical documentation into a task, saving tokens compared to scraping a rendered HTML page.
The important caveat: this isn't automatic. Neither Cursor nor Claude Code auto-discovers an llms.txt file sitting on a website. A developer has to manually add the URL to their own tooling (for example, a rules file the coding agent consults on request). So the realistic pitch for a SaaS or API product isn't "AI will find this and cite you," it's "developers integrating with you can point their own coding agent at this if they choose to."
For a standard blog, e-commerce store or local business site, there's no evidence llms.txt does anything measurable.
What actually drives AI visibility
If the file itself isn't the lever, here's where the real work is:
Crawl access. Check robots.txt actually permits the bots that matter - GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot - if you want them there. This is a genuine yes/no gate that llms.txt cannot substitute for. See our AI crawler accessibility checklist.
Renderable content. Many AI crawlers don't execute JavaScript. Content locked behind client-side rendering is invisible to them regardless of quality.
Structured, extractable answers. Direct-answer content near the top of a page, genuine FAQ and HowTo schema, and headings that match how people actually phrase questions to an AI - rather than how they'd type a Google query.
Third-party corroboration. AI systems weight cross-referenced mentions - reviews, directories, forums, press coverage - more heavily than a single on-site claim. This matters more here than in classic search ranking.
Parametric versus dynamic visibility. Whether a brand is baked into a model's training data (parametric) is a different question from whether it turns up in a live search the model performs (dynamic). No on-page change fixes the first retroactively; that only shifts at retraining cycles, roughly annual given the cost involved. The lever you actually control is the second: being retrievable and citable when a live search happens.
The bottom line
llms.txt costs nothing to add and won't hurt anything. But treating it as a visibility requirement isn't supported by the data, and telling a client it's "vital" is a claim that won't survive scrutiny. Add it once if you run developer documentation or a public API, skip the theatre around it otherwise, and spend the actual budget on crawl access, renderable content, structured answers and off-site corroboration - the things with evidence behind them.
Run our free llms.txt checker to validate the file if you already have one, or a full page scan to see what actually affects AI and search visibility on your site.
References
- Answer.AI, /llms.txt - a proposal to provide information to help LLMs use websites (September 2024)
- Search Engine Journal, Google Confirms LLMs.txt Has No Current Implementation
- Search Engine Journal, Google Says LLMs.Txt Comparable To Keywords Meta Tag
- Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read
- Search Engine Journal, 97% Of llms.txt Files Got No Requests, Ahrefs Data Shows
- dev.to, Using llms.txt with Cursor and Claude Code: a concrete playbook