Is Your Site Blocking GPTBot, ClaudeBot & PerplexityBot? An AI Crawler Accessibility Checklist

Search engines aren't the only crawlers reading your site anymore. When someone asks ChatGPT, Claude, or Perplexity a question, those tools often reach back out to the web to find and cite sources - but only if their crawlers are allowed to fetch your pages in the first place. A surprising number of sites block them without realising it, sometimes on purpose during a security lockdown, sometimes by accident. This is a quick, practical checklist to find out where you stand.

Why this matters now

Being technically correct for Google but invisible to AI crawlers is a growing blind spot. Your site can have perfect on-page SEO and still never appear in an AI-generated answer if the bot that would have read it was turned away at the door. This sits in an odd gap: it's a technical SEO concern (robots.txt, server rules, crawl access), but the payoff is about AI visibility rather than classic rankings - so it tends to fall between the two disciplines and get missed by both.

The crawlers to check for

• GPTBot - OpenAI's crawler, used to gather content that may inform ChatGPT's web-search results and training.
• ClaudeBot (and anthropic-ai) - Anthropic's crawlers for Claude.
• PerplexityBot - Perplexity's crawler for its cited, source-linked answers.
• Google-Extended - controls whether Google's AI features (separate from classic Googlebot indexing) can use your content.
• CCBot - Common Crawl's crawler; not an AI company itself, but its dataset feeds many AI models' training data, so blocking it has knock-on effects.

How to check your own site

1. Open your robots.txt - visit yoursite.com/robots.txt in a browser. Look for a User-agent: line matching any of the bots above, followed by Disallow: /. That's a full block.
2. Check for a blanket Disallow: / under User-agent: * with no specific allowance for AI bots - by default this blocks everyone, AI crawlers included.
3. Look past robots.txt, too. Some hosting/CDN setups (Cloudflare "Block AI bots" toggles, some WAF rules, aggressive rate-limiting) block these crawlers at the server level even when robots.txt says nothing about them. If you use Cloudflare, check Security settings for an AI-bot-blocking toggle that may have been switched on by default.
4. Confirm the crawler can actually render what you want cited. If your key content depends on client-side JavaScript, a crawler that doesn't execute JS may only see an empty shell - the AI equivalent of an SEO crawler seeing a blank page.

A minimal robots.txt example that allows the major AI crawlers

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

If you'd rather keep AI crawlers out of specific sections (say, gated content or an internal search results page) while still allowing the rest of your site, scope the Disallow rule to that path only rather than blocking the whole domain.

Should you actually allow them?

That's a real business decision, not just a technical one. Allowing AI crawlers means your content can be cited and summarised - which can bring you visibility and referral traffic when an AI answer links back to you, but it also means your content can be used in ways you don't directly control. There's no universally right answer; the point of this checklist is to make sure the choice is deliberate rather than accidental.

Where this fits with a wider audit

Robots.txt and crawl-access checks are one part of a much broader technical health picture that also includes indexability, sitemap correctness, page speed, and content quality - the areas AuditCrow's free website audit tool already covers on the pages you scan. Run a free scan alongside the manual robots.txt check above for a fuller picture of what both search engines and AI crawlers can see on your site. For more on the technical side, see our technical SEO audit guide.