How do I check if AI crawlers can access my site?

Resolve your robots.txt per user-agent token rather than reading it as a whole. Each AI crawler obeys only the most specific group matching its own name and ignores every other group, including the wildcard. So the answer differs for GPTBot, ClaudeBot and Google-Extended even when they appear to be covered by the same rules.

Does blocking AI crawlers affect Google rankings?

No. Google-Extended governs Gemini and AI Overviews only and has no effect on ordinary Search indexing or ranking, which Googlebot controls. Blocking GPTBot, ClaudeBot or PerplexityBot has no effect on Google at all — they are different companies.

Is there a free AI crawler checker?

Yes. Ours is free and needs no account for your first scan. It resolves your robots.txt per token following RFC 9309 — group specificity, longest-match paths and Allow overrides — so you learn what your file actually permits rather than what it appears to say.

AI Visibility27 July 20267 min read

Free AI crawler checker: what your robots.txt actually permits

Reading robots.txt top to bottom gives the wrong answer surprisingly often. Group specificity and Allow precedence decide what a crawler may fetch — not the order of the lines.

By Sapun Lamichhane · Arcetis

robots.txt looks like the simplest file on a website and is resolved by rules most people have never read. Reading it top to bottom gives the wrong answer often enough that the mistake has become the most common cause of a site being invisible to AI.

Groups are not additive

This is the rule that catches everyone. A crawler finds the single most specific group matching its user-agent and obeys that group alone. Every other group in the file, including the wildcard, is ignored entirely.

User-agent: *
Disallow: /admin/
Disallow: /private/

User-agent: GPTBot
Allow: /

# GPTBot may now fetch /admin/ and /private/.
# It never reads the wildcard group at all.

The author of that file was granting GPTBot access to the public site. What they actually did was exempt it from every restriction they had.

Longest match wins, not first match

Within a group, the most specific path rule applies regardless of order — and an Allow can override a Disallow that appears above it. A file that reads as a deny followed by exceptions may resolve to something quite different from the sequence it seems to describe.

One operator, several crawlers

Blocking "OpenAI" is not a thing you can do. OpenAI operates GPTBot for training, OAI-SearchBot for search indexing and ChatGPT-User for live retrieval, and each is governed separately. Anthropic and Perplexity split the same way.

That granularity is useful once you know about it: you can decline to be training data while remaining fully citable in live answers. But it has to be expressed as several decisions, and a blanket rule makes all of them at once, in the same direction.

The one everybody gets wrong

Google-Extended controls whether your content may appear in Gemini and AI Overviews. It has no effect whatsoever on Google Search ranking. Allowing Googlebot does not allow it, and disallowing it costs you nothing in ordinary search.

Robots.txt is not a privacy control

A disallowed page can still be indexed if something links to it — the crawler will not fetch the content but may list the URL. To keep a page out of results you need a noindex directive, which requires the page to be crawlable. Disallowing it in robots.txt prevents the crawler from ever seeing the noindex.

What free means here

Our AI crawler checker is free, with no account needed for your first scan. It resolves your file per token the way a crawler does — specificity, longest match, Allow precedence — and reports the outcome per crawler rather than printing your file back at you.