Google retired the robots.txt Tester in Search Console at the end of 2023 — the replacement
report shows the file Google fetched, but it won't tell you whether a specific URL is blocked
for a specific crawler. This does. Paste your robots.txt, choose a crawler, and list the URLs
you want to check: each one comes back allowed or blocked, with the exact rule that decided it
highlighted in your file and the user-agent group that applied. Matching follows
RFC 9309
— longest-match-wins, allow beats disallow on ties, * and $ wildcards,
and percent-encoded paths normalized so escaped and literal URLs compare the same way Google's
parser compares them. Everything runs in your browser; the file is never uploaded and no URL is
ever fetched.
Need to write the rules rather than check them? Use the
robots.txt Generator. Then build the file you point at
with the Sitemap Generator, run a broader crawl check with
the SEO Analyzer, and confirm what your server actually sends
with the HTTP Header Analyzer.
How a crawler decides: group first, then rule
Matching happens in two stages, and mixing them up is where most robots.txt bugs come from.
Stage one picks exactly one group. A crawler compares its product token
(Googlebot) against every User-agent line, case-insensitively, and
uses the most specific match. Everything after the token is ignored, so
googlebot/1.2 and googlebot* both mean googlebot.
Several User-agent lines stacked before the same rules head one shared group, and
two separate groups naming the same token are merged. User-agent: * is consulted
only when nothing named matches — and if no group matches at all, every URL is allowed.
Stage two picks one rule inside that group. Every Allow and
Disallow in the group is tested against the URL path plus its query string; the
longest matching pattern wins, and an Allow beats a Disallow of equal
length. If nothing matches, the URL is allowed — robots.txt is allow-by-default. An empty
Disallow: places no restriction on anything, which is the idiomatic way to say
"crawl everything".
Mistakes this tester catches
- A named group silently disabling your
* rules. Add a User-agent: Googlebot group for one special case and Googlebot stops obeying every rule in your * group. - Rules with no leading slash.
Disallow: admin never matches anything, because matching starts at the first character of the path and every path starts with /. - A stray
$. $ only anchors when it is the last character of the pattern; anywhere else it is a literal dollar sign. - Rules above the first
User-agent line. They belong to no group and are ignored entirely. Noindex: in robots.txt. Google dropped support for that unofficial directive on 1 September 2019 — it does nothing.- An accidental
Disallow: / under User-agent: *. Usually a staging file that shipped to production.
Crawling is not indexing
A Disallow stops a crawler fetching a URL. It does not remove that URL from search
results, and it never makes a page private — robots.txt is a public file that only well-behaved
crawlers choose to obey. If a page is already indexed, blocking it in robots.txt can actually
keep it there, because the crawler can no longer fetch the page to see your
noindex tag. Remove pages with noindex on a crawlable URL, and protect
private ones with authentication. The
on-page SEO checklist covers where robots.txt fits
alongside canonicals, sitemaps and metadata.
Frequently asked questions
If my robots.txt has both a Googlebot group and a * group, which one applies?
Only the Googlebot group. A crawler obeys the single group whose user-agent token matches it most specifically, and the * group is a fallback used only when no named group matches. The named group replaces the * group — it does not add to it. So if * disallows /admin/ and your Googlebot group does not, Googlebot may crawl /admin/. This is the most common robots.txt mistake, and the tester shows which group applied to every result.
When an Allow and a Disallow both match, which one wins?
The most specific rule wins, and "most specific" means the longest rule pattern measured in characters. RFC 9309 puts it as "the match that has the most octets". If an allow and a disallow are exactly the same length, the allow wins — Google describes this as using the least restrictive rule. For example, for /page.htm, "Disallow: /*.htm" (6 characters) beats "Allow: /page" (5), but for /page.php5, "Allow: /page" and "Disallow: /*.ph" are both 5 characters, so the allow wins and the URL is crawlable.
Do wildcards work in robots.txt?
Yes, two of them. An asterisk (*) matches zero or more of any character, and a dollar sign ($) at the end of a pattern anchors the match to the end of the URL. So "Disallow: /*.pdf$" blocks /report.pdf but not /report.pdf?download=1, and "Allow: /$" matches only the homepage. A $ anywhere other than the end of the pattern is treated as a literal dollar sign. Matching always starts at the first character of the path and is case-sensitive, so "Disallow: /fish" does not block /Fish.asp.
Is robots.txt a way to keep pages private?
No. robots.txt is a crawling instruction, not access control. It is a public file that anyone can read, well-behaved crawlers choose to obey it, and malicious ones ignore it. It also does not remove a page from search results: a disallowed URL can still be indexed if other pages link to it, because Google can list a URL it has never fetched. To keep a page out of the index, use a noindex meta tag or X-Robots-Tag header on a page that is crawlable — if you block the URL in robots.txt, the crawler can never see the noindex. To keep it genuinely private, put it behind authentication.
Does this tester match what Google actually does?
It implements RFC 9309, the standardized Robots Exclusion Protocol that Google co-authored and its crawlers follow, plus the leniencies Google documents: case-insensitive field names, tolerated misspellings like "Dissallow", ignored unknown directives, a skipped byte order mark, and CR, LF or CRLF line endings. The test suite encodes the worked examples from both RFC 9309 and Google's robots.txt documentation, including their rule-precedence and path-matching tables. Where the two differ, this tool matches Google, since Google is usually the crawler you care about: Crawl-delay is shown for reference but never changes a verdict, and percent-encoding is normalized the way Google's parser does it — hex digits are uppercased and escapes of unreserved characters are decoded (%62%61%7A and baz match each other), but %2A and %24 are compared literally rather than being treated as an escaped * or $ wildcard.
Does my robots.txt get uploaded anywhere?
No. Parsing and matching run entirely in your browser with JavaScript. Nothing is sent to a server, this tool never fetches a URL for you, and there is no signup — paste the file and test.