Free AI crawler checker: what your robots.txt actually permits
Reading robots.txt top to bottom gives the wrong answer surprisingly often. Group specificity and Allow precedence decide what a crawler may fetch — not the order of the lines.
By Sapun Lamichhane · Arcetis
robots.txt looks like the simplest file on a website and is resolved by rules most people have never read. Reading it top to bottom gives the wrong answer often enough that the mistake has become the most common cause of a site being invisible to AI.
Groups are not additive
This is the rule that catches everyone. A crawler finds the single most specific group matching its user-agent and obeys that group alone. Every other group in the file, including the wildcard, is ignored entirely.
User-agent: *
Disallow: /admin/
Disallow: /private/
User-agent: GPTBot
Allow: /
# GPTBot may now fetch /admin/ and /private/.
# It never reads the wildcard group at all.The author of that file was granting GPTBot access to the public site. What they actually did was exempt it from every restriction they had.
Longest match wins, not first match
Within a group, the most specific path rule applies regardless of order — and an Allow can override a Disallow that appears above it. A file that reads as a deny followed by exceptions may resolve to something quite different from the sequence it seems to describe.
One operator, several crawlers
Blocking "OpenAI" is not a thing you can do. OpenAI operates GPTBot for training, OAI-SearchBot for search indexing and ChatGPT-User for live retrieval, and each is governed separately. Anthropic and Perplexity split the same way.
That granularity is useful once you know about it: you can decline to be training data while remaining fully citable in live answers. But it has to be expressed as several decisions, and a blanket rule makes all of them at once, in the same direction.
The one everybody gets wrong
Google-Extended controls whether your content may appear in Gemini and AI Overviews. It has no effect whatsoever on Google Search ranking. Allowing Googlebot does not allow it, and disallowing it costs you nothing in ordinary search.
Robots.txt is not a privacy control
A disallowed page can still be indexed if something links to it — the crawler will not fetch the content but may list the URL. To keep a page out of results you need a noindex directive, which requires the page to be crawlable. Disallowing it in robots.txt prevents the crawler from ever seeing the noindex.
What free means here
Our AI crawler checker is free, with no account needed for your first scan. It resolves your file per token the way a crawler does — specificity, longest match, Allow precedence — and reports the outcome per crawler rather than printing your file back at you.