Which AI crawlers can actually read your site?
GPTBot, OAI-SearchBot, ClaudeBot and Google-Extended are separate tokens with separate consequences. Blocking one does not block the others, and allowing Googlebot does not allow any of them.
By Sapun Lamichhane · Arcetis
Most robots.txt files were written before generative search existed. They allow Googlebot, block a handful of scrapers, and say nothing at all about the crawlers that now decide whether your content appears in an AI answer. That silence is usually fine — but the sites that added a blanket AI block in 2023 are frequently unaware of what it now costs them.
The tokens are not interchangeable
The most common mistake is assuming one rule covers a vendor. It does not. OpenAI alone publishes three distinct user agents, and each governs something different:
- GPTBot — collects data used for model training. Blocking it has no effect on whether ChatGPT can cite you today.
- OAI-SearchBot — indexes pages for ChatGPT Search. This is the one that controls whether you appear in ChatGPT results.
- ChatGPT-User — fetches a page when a user explicitly pastes or asks about a URL. Blocking it means ChatGPT cannot open your link even when a user asks it to.
Anthropic splits the same way: anthropic-ai for collection, Claude-User for user-initiated fetches, Claude-SearchBot for indexing. Google-Extended controls Gemini and AI Overviews and has no effect whatsoever on ordinary Search — Googlebot governs that, separately.
The rule that catches most people
A blanket disallow under the wildcard agent removes you from everything at once:
User-agent: *
Disallow: /This is almost always left over from a staging deployment. It is also worth being precise about what does not count: a Disallow line targeting a subdirectory is not a site-wide block, and any tool that tells you otherwise is doing a substring match rather than parsing the file. Only an exact `/` path under `User-agent: *` blocks everything.
A reasonable default
If you want to be discoverable in AI answers, be explicit rather than silent — silence works today but leaves you exposed to a future default change:
User-agent: *
Allow: /
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /
Sitemap: https://example.com/sitemap.xmlIf you would rather not be used for training but still want to be cited, allow the search and user-fetch tokens and disallow only the collection ones. That distinction is the whole reason the vendors publish separate agents.
How to check
Our AI Visibility checker tests twenty crawler tokens individually against your live robots.txt and tells you which products can and cannot reach the page. It takes about forty seconds and needs no signup.