Why is my robots.txt Disallow being ignored?
Almost always group specificity. A crawler obeys exactly one group - the most specific one that matches its name - and ignores every other rule in the file.
By Sapun Lamichhane · Arcetis
Almost always group specificity. A crawler does not read your robots.txt top to bottom and apply everything it finds - it selects exactly one group, the most specific one matching its own name, and ignores every other rule in the file.
So if you have a `User-agent: *` group with your real rules and a `User-agent: Googlebot` group added later for one exception, Googlebot obeys only the second group. Everything in the wildcard group stops applying to it.
How does a crawler choose its group?
- 1It looks for a group whose user-agent line matches its own name, case-insensitively.
- 2If several match, the most specific wins - `Googlebot-Image` beats `Googlebot`, which beats `*`.
- 3It obeys that one group completely and ignores all others, including the wildcard.
- 4If no group matches, it is unrestricted - the absence of a rule is permission, not denial.
That last point catches people out in the other direction: a file full of specific `Disallow` rules and no wildcard group restricts nothing for any crawler you did not name.
What else silently fails?
- `Allow` overriding `Disallow`. The longest matching path wins, not the last one written. `Disallow: /admin` plus `Allow: /admin/public` permits the second - which is usually intended, and occasionally a leak.
- A `Crawl-delay` directive. Google has never supported it and ignores it entirely. Bing and Yandex do honour it.
- A `Sitemap:` line pointing at a 404. It is independent of groups and applies file-wide, and nothing tells you when it breaks.
- Blocking a page you want de-indexed. The crawler cannot fetch the page, so it never sees the noindex, so the page stays in the index indefinitely.
How should I check it?
Not by reading it. A checker has to resolve groups, specificity and Allow overrides the way a crawler does, then report what each agent can actually reach.
That resolution is not optional. A naive substring check once reported this product's own robots.txt as blocking everything, because the string `Disallow: /scan/` contains `Disallow: /`. A different naive check reported a missing wildcard group on a file that had one, because it compared `User-agent: *` case-sensitively against our `User-Agent: *`. Both were bugs in the checker, not the file.
How do I test mine?
The robots.txt checker on this site parses your file group by group, applies specificity and Allow overrides, and reports which crawlers can reach which paths - including every AI crawler token separately.