Free robots.txt checker
See exactly which crawlers your robots.txt lets in: Google and Bing, the AI search crawlers behind ChatGPT, Claude and Perplexity, and the AI training bots. Instant, no signup.
Enter a page URL to test that specific path; a bare domain tests the homepage.
What this checker tests
robots.txt is a small text file at the root of your site that tells crawlers which pages they may read. It is also one of the easiest places to break a site by accident: a leftover Disallow: / from a staging build can take a whole site out of Google. This tool fetches your file, parses it the way Google does (longest matching rule wins, and a tie goes to Allow), and tests the page you entered against three groups of crawlers, showing the exact line that decided each result.
The three groups, and why they're treated differently
Search engines (Googlebot and Bingbot) are the ones that matter most: blocking them means the page can't appear in search results, so this shows as red. AI search crawlers (OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot) fetch pages so AI answers can read and cite them; blocking them shows as amber, because you disappear from those answers. AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended) collect content for model training. Blocking those is a legitimate choice that doesn't affect search or AI answers, so it's reported, never marked as a fault.
Other things it catches
A robots.txt that returns a server error (which can make Google pause crawling the whole site), a web page served where the text file should be, a file over Google's 500 KiB limit, and a missing Sitemap line. For the rules that most often go wrong, read our robots.txt guide and the AI crawler accessibility checklist. For the wider picture of how AI tools see your page, run the AI visibility check.
This is one check. Your site needs dozens.
The full AuditCrow audit runs this alongside speed, SEO, accessibility, trust and AI readiness, then tells you what to fix first, in plain English.
Common questions
Straight answers about what this check does, and doesn't do.
What's the difference between AI search crawlers and AI training crawlers?
Search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages so AI answers can read and cite them. Training crawlers like GPTBot, ClaudeBot and Google-Extended collect content to train models. You can block training and still allow search, and many sites do.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google Search crawls with Googlebot, which is separate.
Is having no robots.txt a problem?
No. With no file (a 404), every crawler may read every page. The problems come from a file that errors (server errors can make Google pause crawling), serves a web page instead of text, or blocks something you didn't mean to block.
Does robots.txt stop a page appearing in Google?
Not reliably. It stops crawling, but a blocked page can still be indexed from links pointing to it, without a description. To keep a page out of search results, allow crawling and use a noindex tag instead.
From the blog
All postsThe Robots.txt Guide: Syntax, Common Mistakes, and What It Can't Do
A plain-English guide to robots.txt syntax and the handful of mistakes that accidentally block search engines from your entire site.
Read moreIs Your Site Blocking GPTBot, ClaudeBot & PerplexityBot? An AI Crawler Accessibility Checklist
AI answer engines can only cite your content if their crawlers can reach it. Here's how to check whether GPTBot, ClaudeBot, and PerplexityBot are actually allowed in.
Read moreThe Noindex Tag: What It Does and How to Avoid Locking Google Out by Accident
A noindex tag tells Google not to show a page in search results. Used deliberately, it's a useful tool. Left on by accident, it's the single most common way a site launch goes wrong.
Read moreOther free tools
Quick checks for the things that most often go quietly wrong.