robots.txt does not block the whole site
GEO_ROBOTS_NO_BLANKET_DISALLOW
A `Disallow: /` under the wildcard user-agent removes the entire site from every compliant crawler. This is the single most destructive line that can appear in a robots.txt, and it is usually left behind from a staging deployment.
- Free, no signupNo account, no card
- No AI in the score986 deterministic rules
- Nothing publishedYour scans stay yours
- Answers in secondsQuick scan, no browser
How this rule is weighted
- Importance
- 10 / 10
- How much this matters relative to other rules.
- Confidence
- SPECIFICATION
- A standard says so: HTML, WCAG, an RFC or Google's own documentation.
- Severity
- critical
- How the finding is presented when it fails.
- Scoring
- Counts
- A failure costs points in its category.
Impact is importance multiplied by the confidence tier’s weight — SPECIFICATION 1.0, RESEARCH 0.7, EMERGING 0.3, EXPERIMENTAL 0.1. Two rules backed by the same class of evidence therefore always carry the same weight, which is what makes the score reproducible rather than hand-tuned.
Questions about this rule
- Does GEO_ROBOTS_NO_BLANKET_DISALLOW affect my score?
- Yes. It is an importance 10 rule at SPECIFICATION confidence, so a failure costs 10 × 1 points of impact in the AI Visibility category before that category is normalised to 100.
- What does SPECIFICATION confidence mean?
- A standard says so: HTML, WCAG, an RFC or Google's own documentation. Confidence is a declared tier rather than a per-rule number, so every rule backed by the same class of evidence carries the same weight - which is what makes the score reproducible instead of hand-tuned.
- How do I check GEO_ROBOTS_NO_BLANKET_DISALLOW on my own site?
- Paste your URL into the box above and it runs only this rule, usually in a couple of seconds. It is also included in the AI Visibility checker and in a full scan.
Tags
- geo
- robots
- crawlability
Related AI Visibility rules
- Googlebot is not blocked in robots.txtIf robots.txt disallows Googlebot, the page is excluded from Google Search entirely - the single most consequential crawler block there is.GEO_GOOGLEBOT_ALLOWED
- ClaudeBot is not blocked in robots.txtIf robots.txt disallows ClaudeBot, Claude's browsing/citation features cannot access this page at all.GEO_CLAUDEBOT_ALLOWED
- GPTBot is not blocked in robots.txtIf robots.txt disallows GPTBot, ChatGPT's live browsing/citation features cannot access this page at all - every other AEO/GEO signal becomes moot.GEO_GPTBOT_ALLOWED
- bingbot is not blocked in robots.txtIf robots.txt disallows bingbot, the page is absent from Bing, and from Microsoft Copilot, which grounds on Bing's index.GEO_BINGBOT_ALLOWED
- Google-Extended is not blocked in robots.txtIf robots.txt disallows Google-Extended, the page is excluded from Gemini grounding and AI Overviews. Note this token does not affect ordinary Google Search indexing, which Googlebot controls separately.GEO_GOOGLE_EXTENDED_ALLOWED
- OAI-SearchBot is not blocked in robots.txtIf robots.txt disallows OAI-SearchBot, the page cannot appear in ChatGPT's search results. This is a separate token from GPTBot, which controls training-data collection.GEO_OAI_SEARCHBOT_ALLOWED