robots.txt Is a Parsing Specification: The Rules Your Crawler Actually Follows

robots.txt is usually discussed as a list of rules — "disallow this path, allow that one" — and it is usually written as if the file were read the way a human reads it. RFC 9309, the Robots Exclusion Protocol standard published in September 2022, says otherwise. It specifies an algorithm: product-token group selection, path-prefix matching with octet counting, percent-encoding normalisation, and a set of rules for what happens when the file is missing, broken, or unreachable. Most surprises in production come from not knowing which rule applied.

Direct Answer: A crawler selects the group whose product token matches its user-agent name (case-insensitively), merges all matching groups, and then matches Allow/Disallow path patterns against the request URI. The longest matching pattern wins, and on an exact tie the Allow rule wins. If no group matches and no * group exists, no rules apply and everything is allowed. Critically, a robots.txt that is unreachable means the crawler MUST assume complete disallow — while a robots.txt that returns a 4xx means the crawler MAY access anything. The same broken file can therefore either open or close your site depending on the status code your server returns.

Group selection: which rules apply to you

The first decision is which group applies, and RFC 9309 is precise about it. Crawlers identify themselves with a product token, and the rules are: "Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group."

There are three cases, and the standard gives each its own rule:

Case 1 — a matching group exists. The crawler obeys that group.

Case 2 — several groups match the same token. "If there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group and parsed according to Section 2.2.2." The RFC gives the example: two user-agent: ExampleBot groups are merged into one, and their Disallow lines combine.

This is a common authoring mistake with a concrete consequence. A file that says:

User-agent: Googlebot
Disallow: /search

User-agent: Googlebot
Disallow: /admin

does not mean "Googlebot may crawl /search but not /admin." It means Googlebot is subject to both rules — /search and /admin are both disallowed. Splitting one agent across two groups tightens the restriction rather than loosening it, and it is easy to write by accident when a file is assembled by several people or generated by a template.

Case 3 — no matching group. "If no matching group exists, crawlers MUST obey the group with a user-agent line with the '*' value, if present." And if there is no * group either, "no rules apply" — everything is crawlable.

Note also that the product token is a distinct concept from the full User-Agent header. The RFC specifies that the token "MUST contain only uppercase and lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens ("-")" and that it "SHOULD be a substring of the identification string that the crawler sends," for example the User-Agent HTTP header. This is the formal basis for why Googlebot matches but Googlebot-Image does not — and why a file written against one of Google's many agents does not apply to the others.

Path matching: longest wins, ties go to Allow

This is the rule that most often produces a surprising result, and it is stated with unusual clarity:

The matching MUST start with the first octet of the path. The most specific match found MUST be used. The most specific match is the match that has the most octets. Duplicate rules in a group MAY be deduplicated. If an "allow" rule and a "disallow" rule are equivalent, then the "allow" rule SHOULD be used.

So specificity is measured in octets of the pattern, not in specificity of the idea, and not by rule order. Two consequences worth internalising:

Order does not matter. A Disallow: /admin/ placed above Allow: /admin/public/ still yields Allow, because /admin/public/ is the longer pattern. Position in the file is irrelevant.

Length is what counts. Disallow: /admin/ and Disallow: /admin are different patterns with different octet counts, so they can win against different rules.

The tie-break rule is the second sentence: when an Allow and a Disallow are equivalent (same pattern), "the "allow" rule SHOULD be used." That is the documented basis for the common pattern:

User-agent: *
Disallow: /admin/
Allow: /admin/public/

/admin/public/ is longer, so it wins on specificity. The Allow is a refinement of the broader Disallow, and the resolution is length-based rather than order-based.

If nothing matches at all, "the URI is allowed" — and the RFC adds one implicit exemption: "The /robots.txt URI is implicitly allowed." That is why a Disallow: / does not prevent crawlers from fetching your robots.txt itself.

Encoding: the part that breaks quietly

RFC 9309 spends more space on encoding than most people expect, and it is where real-world mismatches happen. Two rules do the work.

First, normalisation before comparison: "Octets in the URI and robots.txt paths outside the range of the ASCII coded character set, and those in the reserved range defined by [RFC3986], MUST be percent-encoded as defined by [RFC3986] prior to comparison."

Second, and counter-intuitively, un-encoding of percent-encoded ASCII: "If a percent-encoded ASCII octet is encountered in the URI, it MUST be unencoded prior to comparison, unless it is a reserved character in the URI as defined by [RFC3986] or the character is outside the unreserved character range."

The net effect, illustrated by the RFC's own table, is that /foo/bar/%62%61%7A and /foo/bar/baz are treated as the same path. So a rule written against the literal string /baz will match a request for /%62%61%7A. If your analytics shows a URL variant that your robots.txt rule "should" have caught, percent-encoding is the first thing to check.

The matching rule itself: "The match evaluates positively if and only if the end of the path from the rule is reached before a difference in octets is encountered." That is prefix matching with the rule's end as the boundary — and note that a rule without a trailing wildcard matches any path starting with that prefix. Disallow: /admin also matches /administrator and /admin-tools. There is no implicit path-segment boundary.

The special characters

RFC 9309 requires support for exactly three, and the third is the one that catches people:

Character	Meaning	Example
`#`	Designates a line comment	`allow: / # comment in line`
`$`	Designates the end of the match pattern	`allow: /this/path/exactly$`
`*`	Designates 0 or more instances of any character	`allow: /this/*/exactly`

The $ is the one that behaves least like intuition. Disallow: /page.html$ matches /page.html and not /page.html?x=1 — the $ anchors to the end of the path. Without it, Disallow: /page.html matches the whole family including query strings. This is precisely the tool for the "block the paginated variant but allow the clean one" case, and it is the difference between blocking one URL and blocking thousands.

* matches zero or more of any character, so Disallow: /*.pdf$ is the standard way to block a file extension.

What happens when robots.txt is missing, broken, or unreachable

This section is the one that most changes how you should think about the file, because the outcomes are asymmetric in a way that surprises people.

4xx response ("Unavailable"). "'Unavailable' means the crawler tries to fetch the robots.txt file and the server responds with status codes indicating that the resource in question is unavailable. For example, in the context of HTTP, such status codes are in the 400-499 range." And then: "If a server status code indicates that the robots.txt file is unavailable to the crawler, then the crawler MAY access any resources on the server."

5xx or network error ("Unreachable"). "If the robots.txt file is unreachable due to server or network errors, this means the robots.txt file is undefined and the crawler MUST assume complete disallow."

So a 500 on /robots.txt means total crawl lockout, and a 404 means no restrictions at all. Those are opposite outcomes from the same broken deployment, and the difference is one digit in a status code. If you are migrating hosts or your origin is flapping, a robots.txt served from the same failing tier can take your entire site out of the index.

Redirects. "The crawlers SHOULD follow at least five consecutive redirects, even across authorities." And: "If there are more than five consecutive redirects, crawlers MAY assume that the robots.txt file is unavailable." Note that when a robots.txt is reached within five redirects, "the robots.txt file MUST be fetched, parsed, and its rules followed in the context of the initial authority" — so a redirect from example.com to www.example.com applies the rules to the original host, which is what people expect but is worth confirming.

Parsing errors. "Crawlers MUST try to parse each line of the robots.txt file. Crawlers MUST use the parseable rules." A malformed line does not invalidate the file — the parseable rules still apply. This is why a stray character on one line costs you that one rule and not the whole file.

Unknown records. "Crawlers MAY interpret other records that are not part of the robots.txt protocol -- for example, 'Sitemaps'." And critically: "Parsing of other records MUST NOT interfere with the parsing of explicitly defined records. For example, a 'Sitemaps' record MUST NOT terminate a group." This is the formal basis for the standard Sitemap: placement at the end of the file — a Sitemap: line in the middle of a group does not end that group.

Limits, caching, and the file's real requirements

The access method is narrow and worth stating exactly, because "my robots.txt is there but the crawler can't use it" usually means one of these failed. The rules "MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service." The file "MUST be UTF-8 encoded" and have media type "text/plain". Note "all lowercase" and "top-level path" — /robots.txt is not /Robots.txt, not /robots.txt/, and not /subdir/robots.txt.

The parsing limit: "Crawlers SHOULD impose a parsing limit to protect their systems... The parsing limit MUST be at least 500 kibibytes." A file larger than 512 KiB may be truncated by the crawler, so rules beyond that point may never be read. A file with thousands of generated Disallow lines can silently lose its most important rules.

Caching: "Crawlers MAY cache the fetched robots.txt file's contents... Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable." The 24-hour figure is why vendor documentation repeats it — OpenAI's and Perplexity's both state their systems may take up to 24 hours to reflect a change, and this is where that number comes from.

Testing your file against the spec

The rules are deterministic enough to test mechanically. This is the check worth running after every meaningful change to the file:

# 1. Does it return 200 and the right content type?
curl -sS -D - -o /dev/null https://example.com/robots.txt | head -20

# 2. Is it under the parsing limit, and what does the crawler actually receive?
curl -sS https://example.com/robots.txt | wc -c
curl -sS https://example.com/robots.txt | head -30

# 3. Does the Googlebot group exist, and does it merge correctly?
curl -sS https://example.com/robots.txt | awk '
  /^user-agent:/ { ua=$2; next }
  /^(dis)?allow:/ { if (ua) print ua, $0 }
' | sort -u

Step 3 is the one that catches the merged-group bug: sorting the user-agent/rule pairs and looking for duplicates of the same token reveals when one agent has been split across multiple groups, because you will see the same token appear more than once.

And because group selection is case-insensitive while the file may be inconsistent, check your token spelling against what the crawler actually sends — Googlebot, Googlebot-Image, Googlebot-News, and Google-Extended are four different product tokens, and a rule written for one does not cover the others. The AI crawler equivalents are covered in AI Crawlers Are Not Googlebot.

For the surrounding logic — what robots.txt does and does not control relative to noindex and sitemaps — see Robots.txt vs Noindex vs Sitemap: What Each One Does, and for measuring what actually got crawled, Server Logs for SEO: See What Googlebot Requests.

References & Further Reading

IETF, RFC 9309 — Robots Exclusion Protocol, September 2022 (Standards Track). Supports the product-token definition and character restrictions, case-insensitive group matching, the requirement to merge multiple matching groups, the fallback to the * group, the longest-match rule and the Allow tie-break, the percent-encoding normalisation and un-encoding rules, the three special characters (#, $, *), the "other records" rule that a Sitemap record must not terminate a group, the access-method requirements (/robots.txt, lowercase, top-level, UTF-8, text/plain), the five-redirect limit, the 4xx-versus-5xx availability distinction including "MUST assume complete disallow" on unreachability, the parseable-rules requirement, the 24-hour caching recommendation, and the 500 KiB minimum parsing limit.
Google Search Central, Overview of Google crawlers and fetchers (user agents), accessed 2026-11-14. Supports the distinction between common, special-case, and user-triggered clients, and the statement that common crawlers always respect robots.txt rules for automatic crawls.
Google Search Central, How Search works, accessed 2026-11-14. Background on the crawling and indexing process in which robots.txt is retrieved and applied.
Google Search Central, Robots.txt vs Noindex vs Sitemap: What Each One Does — the publication's own companion article on the distinct controls.