llms.txt is a proposed plain-Markdown file, placed at yoursite.com/llms.txt, that gives AI assistants a short, curated summary of your site and links to the pages that matter most. It sits alongside robots.txt and sitemap.xml but solves a different problem: robots.txt controls whether crawlers can access your content, a sitemap lists every URL, and llms.txt tells an AI system which pages are actually worth reading.
It has been widely sold as essential for AI search visibility. The evidence doesn't support that. But for a specific type of site it's a genuinely useful, cheap file to have. This guide covers what it is, what has been tested, who should bother, and exactly how to build one properly.
Where llms.txt came from
llms.txt was proposed in September 2024 by Jeremy Howard of Answer.AI, as a way to help large language models work within limited context windows. A long, JavaScript-heavy page full of navigation, cookie banners and marketing copy is expensive and unreliable for an AI tool to parse for the handful of facts it needs. llms.txt gives it a shortcut: a clean index of your most important pages, what they're about and where to find them.
That's a real, specific problem. It was never proposed as a search ranking signal or an "AI SEO" mechanism, and the proposal itself doesn't claim otherwise.
Does llms.txt help you show up in AI search?
Somewhere between the original proposal and the current SEO content cycle, llms.txt got reframed as the key to showing up in ChatGPT, Gemini and AI Overviews. That's where the evidence falls apart.
Google has directly rejected it. In June 2026, Google's John Mueller called llms.txt "purely speculative for now", noting the format has existed for roughly two years with no confirmation that any AI system actually reads it. He compared it to the deprecated keywords meta tag: a file entirely controlled by the site owner, with no verification, and therefore too easy to manipulate for any search or AI system to trust.
The usage data backs him up. Ahrefs analysed 137,210 domains in May 2026. Roughly 28% had published a valid llms.txt file, and 97% of those received zero requests. Of the small remainder that did get hit, most traffic wasn't AI retrieval at all but SEO audit tools, general crawlers and tech-profiling bots checking whether the file existed. Genuine AI retrieval bots accounted for about 1.1% of requests. Slackbot fetched llms.txt files more often than PerplexityBot did.
No major AI platform (OpenAI, Anthropic, Google or Perplexity) has confirmed that its production search or citation systems read llms.txt. It costs nothing to add and won't hurt anything, but telling a client it's "vital" is a claim that won't survive scrutiny.
Who should actually do this
If you run developer documentation, a public API, or a SaaS product with technical docs, llms.txt is worth 30 minutes of your time. The real audience isn't AI search - it's developers who use AI coding assistants (Cursor, Claude Code, GitHub Copilot) to build against your product. Those tools don't discover llms.txt automatically, but a developer can manually point their coding agent at your /llms.txt URL, and when they do, it saves the assistant from wading through your site's navigation, cookie banner and marketing copy to find the documentation it actually needs.
The important caveat: none of these tools auto-discovers an llms.txt file sitting on a website. So the realistic pitch for a SaaS or API product isn't "AI will find this and cite you", it's "developers integrating with you can point their own coding agent at this if they choose to".
If you run a blog, an e-commerce store or a local business site, you can skip this. There's no evidence it does anything for you, and your time is better spent on the things below.
What actually drives AI visibility
If the file itself isn't the lever, here's where the real work is:
- Crawl access. Check robots.txt actually permits the bots that matter (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot) if you want them there. That's a genuine yes/no gate llms.txt can't substitute for. See our AI crawler accessibility checklist.
- Renderable content. Many AI crawlers don't execute JavaScript, so content locked behind client-side rendering is invisible to them regardless of quality.
- Structured, extractable answers. Direct answers near the top of a page, genuine FAQ and HowTo schema, and headings that match how people phrase questions to an AI.
- Third-party corroboration. AI systems weight cross-referenced mentions (reviews, directories, forums, press) more heavily than a single on-site claim.
Whether a brand is baked into a model's training data is a different question from whether it turns up when the model runs a live search. No on-page change fixes the first retroactively. The lever you control is the second: being retrievable and citable when that live search happens. Our guide to why you're not showing up in AI Overviews covers it in full.
The two files
/llms.txt is a short index - a map of your site's key resources, in root-level, plain Markdown.
/llms-full.txt is the expanded version: your entire documentation set stitched into one continuous Markdown file, for tools that want to ingest everything in one pass rather than following links.
Not every site needs the second file. Add it once your documentation is substantial enough that following individual links becomes inefficient.
Structure of /llms.txt
The format is deliberately simple: an H1 for your site or project name, a blockquote for a one-line summary, H2s for sections, and plain Markdown bullet links for navigation.
# Website Title or Project Name
> A concise, one-sentence description of what this website or software project does. Keep this under 150 characters to save AI tokens.
## Main Overview
A brief paragraph providing critical context about the organization, project, or primary documentation. This helps the LLM understand the core purpose before it reads further.
## Core Resources
* Getting Started Guide (/docs/getting-started): Quick introduction to installation and initial setup.
* API Reference (/docs/api): Complete documentation for our public REST endpoints.
* Configuration Options (/docs/config): Detailed breakdown of env variables and settings.
## Secondary Resources
* Troubleshooting FAQ (/docs/faq): Solutions to common errors and deployment bugs.
* Changelog (/docs/changelog): Latest version updates and breaking changes.
## Full Content
* Full Documentation Bundle (/llms-full.txt): A single text file containing all documentation for easier ingestion.
Place this at your site root (https://example.com/llms.txt) so it's found the same way robots.txt and sitemap.xml are.
Structure of /llms-full.txt
Consolidate your documentation into one file, using H1s to mark where each original page begins. Each page section can include tagged code fences (for example, a bash block for install commands and a typescript block for API examples).
# Website Title - Complete Content
> This file consolidates all documentation into a single file for complete context window ingestion.
---
# Page 1: Getting Started
## Installation
Run the following command to install the package:
(npm install example-package)
## Quick Start
Initialize the library in your code...
---
# Page 2: API Reference
## Authentication
All requests must include the Bearer token header...
Rules that matter
Use relative URLs. Links inside both files should be relative (/docs/api, not https://example.com/docs/api), so the file resolves correctly regardless of environment or domain mapping.
Declare the language on every code fence. Always tag code blocks (```typescript, ```bash) so a coding assistant parses syntax correctly rather than guessing.
Cut the structural fluff. No headers, footers, cookie banners, or sidebar navigation. These files exist to save tokens - padding them with boilerplate defeats the point.
Keep the summary line under ~150 characters. It's the first thing consumed and the cheapest to get wrong by making it too long.
Automating it, so it doesn't go stale
A hand-written llms.txt drifts out of date the moment your documentation changes, and a stale index is arguably worse than none - it actively misdirects. Wire generation into your build rather than maintaining it by hand:
- VitePress: the
vitepress-plugin-llms(orvitepress-plugin-llmstxt) packages generatellms.txtandllms-full.txtautomatically from your existing docs source at build time. - Next.js:
next-plugin-llmsgenerates llms-standard output from your Next.js documentation routes. - Mintlify-hosted docs: Mintlify generates and serves
llms.txtautomatically for sites built on it, no separate build step required.
If your framework isn't covered by an existing plugin, a simple build script that walks your docs directory and emits both files following the structure above is a half-day job at most - treat it the same as your sitemap generation: automated, run on every deploy, never hand-maintained.
Quick checklist
- Confirm this is worth doing for your site type (dev docs / API / SaaS, yes; blog / e-commerce / local business, skip)
/llms.txtat site root, H1 + blockquote summary + H2 sections + relative links/llms-full.txtif your docs are substantial enough to warrant it- Code fences tagged with language
- No structural fluff
- Generation wired into your build pipeline, not maintained by hand
- Set a reminder to spot-check it after any major documentation restructure
Validate what you publish with our free llms.txt checker - it flags missing structure, broken formatting and HTML masquerading as a text file.
References
- Answer.AI, /llms.txt - a proposal to provide information to help LLMs use websites (September 2024)
- Search Engine Journal, Google Confirms LLMs.txt Has No Current Implementation
- Search Engine Journal, Google Says LLMs.Txt Comparable To Keywords Meta Tag
- Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read
- GitHub, vitepress-plugin-llms
- GitHub, next-plugin-llms
- Mintlify, How to generate llms.txt
- dev.to, Using llms.txt with Cursor and Claude Code: a concrete playbook
