The Robots.txt Guide: Syntax, Common Mistakes, and What It Can't Do
Ben Foord3 min readA plain-English guide to robots.txt syntax and the handful of mistakes that accidentally block search engines from your entire site.
Robots.txt is one of the smallest files on your site and one of the easiest to get catastrophically wrong. It's a plain text file at yoursite.com/robots.txt that tells well-behaved crawlers which parts of your site they're allowed to request. Get one line wrong and you can accidentally deindex your entire site; get it right and it's a useful, low-effort tool for managing crawl behaviour.
The Basic Syntax
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
User-agentspecifies which crawler the following rules apply to.*means "all crawlers"; you can also target a specific one by name (e.g.User-agent: Googlebot).Disallowtells that crawler not to request matching paths.Disallow: /admin/blocks everything under/admin/; an emptyDisallow:blocks nothing.Allowcarves out an exception within a broader disallowed path - useful when you want to block a folder but allow one file inside it.Sitemappoints crawlers to your XML sitemap. This line can go anywhere in the file and isn't tied to a specificUser-agentblock.
The Mistake That Deindexes an Entire Site
By far the most damaging robots.txt error is a stray:
User-agent: *
Disallow: /
This blocks every crawler from every page on the entire site. It's a common leftover from a staging environment - many CMSs and hosting platforms automatically add this rule while a site is in development, specifically so search engines don't index an unfinished build. If that rule survives the move to production, your live site quietly stops being crawled, with no error message anywhere to flag it - traffic just declines and nothing in the UI tells you why.
Always check your live robots.txt after a launch, migration, or hosting change. It takes ten seconds to visit yoursite.com/robots.txt and confirm there's no blanket Disallow: / left over from staging.
Other Common Robots.txt Mistakes
- Blocking CSS or JavaScript files. Google renders pages to evaluate them properly, and blocking the resources it needs to render your page correctly can hurt how it's understood and ranked - even though the page's HTML itself isn't blocked.
- Using robots.txt to try to remove a page from search results. Disallowing a URL stops it being crawled, but a page already indexed can still appear in search results (often with no description, since Google can't read the blocked content) until it's properly removed via a
noindextag or Search Console request. Robots.txt andnoindexsolve different problems and aren't interchangeable. - Case sensitivity errors. Paths in robots.txt are case-sensitive -
/Admin/and/admin/are different rules to a crawler, even if your server treats URLs as case-insensitive. - Forgetting the sitemap line entirely. It's optional but free - a one-line pointer that helps crawlers discover your sitemap without depending solely on Search Console submission.
Robots.txt vs. Meta Robots vs. Noindex
These three get conflated constantly:
- Robots.txt controls crawling - whether a bot is allowed to request a URL at all.
- A
noindexmeta tag (in a page's<head>) controls indexing - the page can still be crawled, but shouldn't appear in search results. - Blocking a page in robots.txt prevents a crawler from ever seeing a
noindextag on it, since it never fetches the page to find one. If you actually want a page removed from search results, usenoindexand make sure robots.txt isn't blocking it - otherwise the instruction never gets read.
Robots.txt and AI Crawlers
Robots.txt also governs whether AI crawlers like GPTBot, ClaudeBot, and PerplexityBot can reach your content - a growing consideration alongside classic search engine access. See our dedicated AI crawler accessibility checklist for the specific user-agents to check for and a ready-to-use example allowing them. And once AI crawlers are allowed in, an llms.txt file points them at your key content - validate yours with our free llms.txt checker.
The Fast Way to Check Yours
AuditCrow's free scan checks your robots.txt as part of its crawlability review, flagging blanket disallow rules, blocked assets, and conflicts with your sitemap automatically. For the rest of the fundamentals worth checking alongside it, see our technical SEO checklist.