Disallow Doesn't Mean Deindex: What robots.txt Actually Controls

You add Disallow: /internal/ to robots.txt, wait a few weeks, and the page is still sitting in Google’s results — now with a gray “no information is available for this page” snippet where the description used to be. Search Console labels it Indexed, though blocked by robots.txt, which reads like a contradiction until you learn the one fact most robots.txt advice skips: robots.txt controls crawling, not indexing — and blocking the crawl can actually prevent a page from ever being deindexed. The file is a keep-out sign for fetchers, not an eraser for search results, and the two behaviors are governed by entirely different mechanisms.

Once you separate those two mechanisms, the whole topic gets simple: one decision table, one matching algorithm, and one famous footgun.

What RFC 9309 actually standardizes

The Robots Exclusion Protocol spent 25 years as a de-facto convention before it became a real standard in 2022, as RFC 9309. What the RFC defines is narrow and precise:

• Fields: User-agent, Allow, Disallow. That’s the whole vocabulary. Sitemap and everything else ride along as “other records” that crawlers may ignore but must not treat as group boundaries (§2.2.4).
• Special characters: # starts a comment, * matches zero or more characters, $ anchors the end of a pattern — and $ is only special as the last character of a pattern.
• Group selection: the crawler obeys the group whose user-agent token is the most specific match for its own name; * is the fallback; no match at all means everything is allowed.
• Rule precedence: among matching rules, “the most specific match” wins — defined as the one with the most octets, not the one that appears first. On an exact length tie between an Allow and a Disallow, the Allow should win.
• One freebie: /robots.txt itself is implicitly always allowed, so a crawler can never be locked out of reading the rules.

Notice what’s absent. The RFC describes how content “may be accessed” by crawlers, explicitly notes the protocol “is not a form of access authorization,” and says nothing about search results. Indexing isn’t out of scope by accident — it was never in scope. A search engine’s index is built from everything it knows about a URL, and crawling the page is only one source of that knowledge.

Why a Disallowed URL still ends up indexed

The other source is links. If external pages link to /internal/pricing-draft, Google knows the URL exists, knows the anchor text people use for it, and can index the URL on that evidence alone — without ever fetching it. Google’s own documentation is blunt about this:

“A page that’s disallowed in robots.txt can still be indexed if linked to from other sites.”

That’s the URL-only listing: no snippet, no cached content, just the address and whatever the surrounding web says about it. And Google’s stated fix is not a stricter robots.txt — it’s to “password-protect the files on your server, use the noindex meta tag or response header, or remove the page entirely.”

For years there was a folk workaround: an unofficial Noindex: directive typed straight into robots.txt. Google’s crawler quietly honored it — until it didn’t. In July 2019 Google announced it was “retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019,” noting that these undocumented rules were used by almost nobody and, worse, contradicted other rules in most of the files that did use them. The blessed alternatives from that announcement: noindex in a robots meta tag or X-Robots-Tag header, 404/410 status codes, password protection, or Search Console’s Remove URL tool for quick temporary removal. If a tutorial still shows Noindex: inside robots.txt, it has been wrong for seven years.

The footgun: Disallow a page that carries noindex

Here’s the trap that follows directly from the mechanism. You want a page gone from Google, so you do both things — belt and suspenders:

# robots.txt
User-agent: *
Disallow: /internal/

<!-- /internal/pricing-draft -->
<meta name="robots" content="noindex">

The suspenders just cut the belt. A noindex tag only works when the crawler fetches the page and reads it — and your Disallow rule forbids exactly that fetch. Google never sees the noindex, keeps the URL in its index on link evidence alone, and the page can sit in results indefinitely. The two directives don’t stack; they cancel.

The fix is sequencing: leave the page crawlable, let Googlebot fetch it and process the noindex, confirm it’s dropped out of the index, and only then add the Disallow if you also want to stop the crawling. For most pages you can skip the last step entirely — a noindexed page that gets crawled occasionally costs you nothing.

The decision table

Each mechanism answers a different question, so pick by the outcome you actually want:

You want	Use	Why
Page out of search results, still reachable by visitors	<meta name="robots" content="noindex">	Removes/prevents the index entry; requires the page to stay crawlable
A PDF, image, or API response out of results	X-Robots-Tag: noindex response header	Same effect as the meta tag for content that has no <head>
Crawlers to stop hammering infinite URL spaces (faceted filters, calendars, search results pages)	robots.txt Disallow	This is what crawl control is for — saving crawl budget on pages you never wanted fetched
Content actually private	Authentication	robots.txt is public and advisory; per RFC 9309 it “is not a form of access authorization”
A URL gone from Google fast	Search Console URL removal + noindex (or 404/410)	Removal tool hides it quickly; the status code or tag makes it permanent
Page out of results and never crawled	noindex first, Disallow later	The order matters — see the footgun above

The one combination with no valid use case is the one people reach for first: Disallow as a deindexing tool.

How matching actually works

The second half of robots.txt confusion is the matching algorithm, because it’s neither “first rule wins” nor regex. Take:

User-agent: *
Disallow: /shop/
Allow: /shop/sale/
Disallow: /*.pdf$
Allow: /shop/sale/catalog.pdf

For /shop/sale/summer, both Disallow: /shop/ (6 octets) and Allow: /shop/sale/ (11 octets) match. Longest wins: allowed. Order in the file is irrelevant — you can put the Allow above, below, or in a different part of the group.

For /shop/sale/catalog.pdf, the wildcard rule /*.pdf$ (7 octets) matches, but Allow: /shop/sale/catalog.pdf (23 octets) is longer: allowed. Mind the $: patterns are prefix matches by default — anything may follow — so /*.pdf alone also blocks /whitepaper.pdf?download=1, while /*.pdf$ matches only paths that end in .pdf and lets the query-string version through. $ doesn’t add matching power; it removes the implicit trailing wildcard, which cuts both ways.

And on an exact tie — Allow: /page vs Disallow: /page — the Allow wins. That’s in the RFC, and it’s the tiebreak Google implements.

The percent-encoding edge case nobody writes about

Both sides of a comparison get percent-encoding-normalized before matching (RFC 9309 §2.2.2): non-ASCII characters are encoded, and escapes of unreserved characters are decoded, so /caf%C3%A9 and /café are the same path, and %62%61%7A matches baz. Uncontroversial.

But §2.2.3 goes one step further. Since * and $ are magic inside patterns, the RFC suggests (Figure 6) that a pattern can use %2A to match a literal asterisk in a URL — Disallow: /path/file-with-a-%2A.html supposedly blocks /path/file-with-a-*.html — and %24 likewise for a literal dollar sign.

Google’s open-source parser doesn’t implement that. In robots.cc, patterns are only ever escaped into percent-encoding, never decoded out of it, and nothing maps %2A back to a literal *. So against the crawler that matters most:

• The URL path /file-with-a-*.html stays a literal * (ASCII, so normalization leaves it alone).
• The pattern %2A stays the three literal characters %2A.
• They never match. The RFC’s Figure 6 example silently does nothing.

What the pattern Disallow: /file-with-a-%2A.html does block is a request whose path arrives already percent-encoded — /file-with-a-%2A.html on the wire — because then both sides hold the same three characters. Two URLs that render identically in a browser bar behave differently depending on whether the * was sent raw or encoded. If you genuinely need to block a URL containing a literal asterisk under Google semantics, your realistic option is a wildcard — Disallow: /file-with-a-* — and accepting that it matches more than the one character.

Our robots.txt tester implements RFC 9309 with exactly this documented deviation, on purpose. The source comment spells out the reasoning: Google parity is the practical target for a tester, because a verdict that’s RFC-pure but disagrees with Googlebot would be worse than useless. Everything else follows the spec — longest-match in octets, allow-wins ties, $ only at pattern end, case-insensitive fields, CR/LF/CRLF line endings, BOM stripping, merged groups under equally specific user-agent tokens, and Google’s documented 500 KiB parse limit.

Test the file before the crawler does

The tester is paste-only — it never fetches your robots.txt, everything runs in your browser. Paste a file and a list of URLs and it tells you, per URL and per user-agent, which exact rule line decided the verdict and why — including tie explanations and wildcard-group fallbacks. It also lints as it parses: misspelled directives (Dissallow: is depressingly common), rules before any User-agent line, $ in the middle of a pattern, blank lines splitting what is actually one group — and yes, if your file contains a Noindex: line, it warns you directly that Google stopped supporting it on 1 September 2019 and points you at the meta tag / X-Robots-Tag alternatives.

If you’re writing a file from scratch rather than debugging one, the robots.txt generator builds a clean one — and remember that Sitemap: lines belong in it, pointing at output from the sitemap generator. For the discovery side of the pipeline (getting crawled sooner, which is a different problem from being allowed to crawl at all), see IndexNow vs sitemaps; for the on-page half of index control, the meta robots section of the on-page SEO checklist covers where noindex should and shouldn’t appear.

The short version

The folklore says	The mechanism says
”Disallow it and it’ll drop out of Google”	Disallow stops crawling; the URL can stay indexed from external links alone
”Add Noindex: to robots.txt”	Never standardized; Google retired it September 1, 2019
”Disallow and noindex, to be safe”	The Disallow hides the noindex from the crawler — they cancel, not stack
”First matching rule wins”	Longest matching rule wins, measured in octets; Allow wins exact ties (RFC 9309)
“Disallow: /*.pdf needs the $”	Patterns are prefix matches; $ removes the implicit trailing wildcard
”%2A matches a literal * (RFC 9309 Fig. 6)“	Google’s parser never decodes it — the pattern only matches a URL that arrives with a literal %2A
”robots.txt keeps content private”	It’s a public, advisory file; RFC 9309 says it is “not a form of access authorization”