Skip to content
WebsiteValidator

AI & security

AI Crawler Checker

Which AI crawlers your robots.txt actually lets in

Every AI crawler is controlled by its own robots.txt token, and blocking one does nothing to the others. OpenAI alone operates three - GPTBot for training, OAI-SearchBot for search retrieval, ChatGPT-User for live browsing on a user's behalf - and blocking GPTBot while leaving the other two open is extremely common and almost always unintentional.

Quick runs this page's high-signal checks in about 3 seconds, no browser. Full adds Core Web Vitals, the accessibility engines and a screenshot, and takes about a minute.

Try:
  • Free, no signupNo account, no card
  • No AI in the score986 deterministic rules
  • Nothing publishedYour scans stay yours
  • Answers in secondsQuick scan, no browser

So each token is checked individually rather than collapsed into one 'AI crawlers' verdict: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, CCBot and dozens more, alongside llms.txt.

The result tells you what you have actually decided, which is frequently not what you meant to decide. Blocking AI crawlers is a legitimate choice; blocking the training crawler while leaving the citation crawler open, or the reverse, usually is not.

Why this matters

You do not know what your robots.txt lets in any more

Most robots.txt files are a stack of pasted snippets. One from an article about blocking AI training, one from a plugin, one from whoever built the site three years ago. Meanwhile the token list keeps moving: OpenAI alone runs separate agents for training, for search and for live browsing, and blocking one leaves the other two open. The file takes ten seconds to read and still nobody can say what it currently permits.

What it costs you when this is wrong

  • You disappear from answers, not from Google

    Disallow OAI-SearchBot or Claude-SearchBot and your pages stop being retrievable as answer sources, while Googlebot carries on and your traffic graph looks normal. You find out when someone else is cited in an answer about your own product.

  • The opt-out you think you made is partial

    Blocking GPTBot does nothing to ClaudeBot, CCBot, Bytespider or meta-externalagent. If you blocked one token and stopped, the others are still crawling, and nothing tells you that the decision you made was a fraction of the one you intended.

  • A customer's assistant is turned away

    Tokens such as ChatGPT-Agent, Operator and NovaAct are the ones acting for a real person asking about you right now. If they are disallowed, the assistant reports it cannot reach your site, and that lost visit never appears in your analytics at all.

Why run it here

  • 81 crawlers, 81 separate verdicts

    Every token is its own rule carrying its own weight: GPTBot and ClaudeBot are critical, ChatGPT-Agent and Claude-SearchBot are errors, CCBot and Bytespider are warnings. A single collapsed "AI crawlers" result cannot tell you which of those you have shut out.

  • Read the way a crawler reads it

    Groups, specificity and Allow overrides are resolved by a conformance parser, so a wildcard Disallow followed by an Allow for GPTBot reports as allowed - the conclusion a crawler actually reaches. A fixture pins that exact case so it cannot regress; access is evaluated at the site root.

  • Allowed on purpose, or allowed by silence

    Each verdict records whether robots.txt says anything at all about that token, or whether it is open only because nothing mentions it. That is the difference between a position you chose and one you inherited.

This check runs on its own here, and as part of the full audit alongside the other 985 rules. Either way the findings carry their evidence and their citation, and the report is never published or kept.

How to allow AI crawlers

Each operator obeys its own token, and robots.txt groups are not additive — a crawler follows the single most specific group matching its name and ignores every other.

  1. List which AI crawlers you want to permit, deciding separately about training, search indexing and live retrieval. They are three different trades.

  2. Add or remove the specific tokens: GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI; ClaudeBot, Claude-User and Claude-SearchBot for Claude; Google-Extended, PerplexityBot, Applebot-Extended and CCBot for the rest.

  3. Repeat your site-wide Disallow rules inside any group you create. A group containing only Allow releases that crawler from the wildcard rules entirely, which is almost never intended.

  4. Remember that Google-Extended governs Gemini and AI Overviews only and has no effect whatsoever on ordinary Google ranking.

  5. Re-check after every robots.txt edit. This is the file where a one-line change goes unnoticed for months.

What happens when you run it

  1. Enter your address

    Just the domain is enough — example.com, with or without the https. No account, no card, no crawl of your whole site.

  2. We render it like a visitor

    Your page opens in a real browser and JavaScript runs to completion, so what gets audited is the page a person actually sees — not the raw HTML your server sent. A screenshot comes back with the report as evidence of exactly what we measured.

  3. You get findings you can act on

    Up to 986 checks run against that rendered page. Each finding names the rule, quotes the evidence found on your page, cites the public specification behind it, and says what to change. Nothing is a judgement call — run it twice on an unchanged page and the score is identical.

What runs this check

  • RFC 9309 robots parsing
  • Real browser rendering

Named because provenance is the product. Every finding in your report says which of these produced it, so you can check the reasoning rather than take a score on faith.

Questions

Should I block AI crawlers?
That is a business decision, not a technical one, and this tool does not take a side. It tells you which are currently allowed so the decision you have made is the one you intended. Blocking training while allowing retrieval is a coherent position; doing it by accident is not.
Does blocking GPTBot remove me from ChatGPT?
Not by itself. GPTBot is the training crawler. Search retrieval uses OAI-SearchBot and live browsing uses ChatGPT-User - different tokens, checked separately here.
Is llms.txt a standard?
No. It is a proposal, with no established support from any major model provider. It is checked and reported at EMERGING confidence, which means it appears in your report and is deliberately near-incapable of moving your score.

Also checks

This page answers these searches too — they are the same job, so they share one page rather than being split across near-identical ones:

  • gptbot robots.txt
  • block ai crawlers
  • claudebot checker
  • llms.txt checker

Other checkers

Run the ai crawler checker now

No signup, no credit card. Quick answers in seconds; a full audit takes about a minute.

Run a free scan

No signup. No credit card. Nothing stored but the result.