Free robots.txt checker: the file that is easiest to get wrong
One line in robots.txt can remove a site from search entirely, and the resolution rules mean reading the file rarely tells you what it does.
By Sapun Lamichhane · Arcetis
robots.txt has the worst consequence-to-complexity ratio on the web. It is a few lines of text, and one of them can remove your entire site from search results with no error and no notification.
The rules that decide what it means
- 1A crawler obeys the single most specific group matching its user-agent, and ignores every other group including the wildcard.
- 2Within that group, the longest matching path rule wins, regardless of the order it appears in.
- 3An Allow can override a Disallow, including one written above it.
- 4The file must be at the domain root, served with a 200 status and a text/plain content type.
Rules one and two together are why reading the file gives the wrong answer so often. Its behaviour is resolved, not sequential.
The mistakes that cost the most
- A staging Disallow: / shipped to production. The most expensive two lines in web development, and it happens regularly.
- Disallowing CSS or JavaScript directories, which prevents the page being rendered properly and therefore judged properly.
- Using robots.txt to hide something. A disallowed URL can still be listed, and blocking it prevents any noindex from being read.
- A group containing only Allow rules, which exempts that crawler from every site-wide restriction rather than adding to them.
The crawlers your file does not mention
Most robots.txt files were written before generative search existed. They allow Googlebot, block a few scrapers, and say nothing about the crawlers that now decide whether you appear in an AI answer.
Silence is usually fine — it means permitted. The problem is the sites that added a blanket AI block in 2023 and have not revisited it since, and which are now absent from products that did not exist when the rule was written.
What free means here
Our robots.txt checker is free, with no account needed for your first scan. It resolves your file the way a crawler does and reports the outcome per user-agent — including the AI crawlers — rather than printing your own file back at you and leaving the interpretation to you.