Crawl Budget Best Practices
Crawl budget is shaped by the responses a server returns. Learn the source definitions, a worked response example, and why response hygiene cannot force indexing.
Crawl budget describes how much attention a search crawler gives a site. Google's crawler overview explains how crawlers discover and fetch pages, and RFC 9110 defines the HTTP semantics that govern those requests and responses. The real problem is that teams try to influence a crawler with tactics, while the lever they actually control is the response their server returns.
Direct answer: Keep the URL inventory clean, return accurate status codes, and use clear HTTP responses so a crawler spends its effort on pages that matter. The Google overview describes crawling and RFC 9110 defines response semantics. The rule that server responses steer crawling effort is a synthesis.
Delta: This article adds a claim-status table and a response-level decision test beyond the sources: it maps each server response to a crawling decision, with claim status explicit.
The mechanism
Google's crawler overview describes how crawlers discover URLs, fetch them over HTTP, and process what comes back. RFC 9110 defines the HTTP method and status code semantics a server uses to answer a request, including codes such as 200, 404, and 5xx, and headers that describe the response. Crawling is therefore a conversation governed by a shared specification. The crawler requests a resource; the server answers with a code and metadata; the crawler decides what to do next based on that answer.

The practical consequence is that every request a crawler makes consumes effort. If a site serves many near-duplicate URLs with fresh 200 responses, or returns 5xx errors that force repeated attempts, the crawler spends its attention on low-value or failed requests instead of useful pages. A clean inventory with deliberate status codes and correct headers reduces wasted fetches. Because the server is the other party in the conversation, the team can improve the exchange by making each response truthful and predictable.
The sources describe crawler behavior and HTTP semantics, but they do not set a specific request limit or promise a ranking outcome. Any statement about how much effort a site receives is an inference. What is grounded in the sources is that responses communicate state: a 404 signals a missing resource, a 5xx signals a server problem, and a redirect signals that a resource moved. The meaning of those codes comes from the RFC; the fact that crawlers act on them comes from the crawler overview. Together they support a practical rule: make the response match reality.
The consequence for practice is a server-side discipline. Return accurate status codes, avoid soft errors that render a 200 for a missing page, and keep parameters from multiplying the number of distinct URLs. The team controls its own responses, not the crawler's internal allocation of effort.
Good practice is therefore measured by the quality of the responses a site produces, not by a hoped-for change in crawler behavior.
A worked example
The following is hypothetical and illustrative. An online store exposes the same product under several parameter URLs, a filter page returns a 200 response while showing an empty result, and the server intermittently returns 5xx errors under load. Three checks decide the review. First, do duplicate URLs return a deliberate status or a consolidation signal rather than fresh 200 responses? Second, does the empty page return a truthful code instead of a soft 200? Third, do the 5xx responses line up with repeated crawler requests in the logs? The review finds all three issues. The team consolidates the parameters, corrects the empty-page response, and reduces server errors. The outcome is a response-cleanup decision, not a ranking guarantee. The numbers are hypothetical; the verification method is the contribution.
Evidence status
| Claim | Status | Source | Verified | Recheck when |
|---|---|---|---|---|
| Crawlers discover and fetch pages over HTTP | source-reported | Google, Crawler overview | 2026-11-18 | Guide updated |
| HTTP status codes and headers define response semantics | source-reported | IETF, RFC 9110 | 2026-11-18 | RFC revised |
| Accurate server responses reduce wasted crawler effort | inferred | Derived from the sources | 2026-11-18 | Crawler behavior changes |
The first two rows are factual claims taken from the sources. The third is an inference that connects the two: neither source states that response hygiene increases the attention a site receives. Recheck the factual rows when the guide or the RFC changes, and the inferred row when crawler behavior is described differently.
What it cannot do
Crawl budget best practices cannot force indexing or ranking. A clean response tells a crawler what a page is, but the decision to index or to rank that page is separate and is not specified by either source. Treating response hygiene as a direct ranking lever overstates what is documented. A site can return perfect status codes and still fail to rank, because ranking depends on factors the crawler overview does not enumerate and the HTTP specification does not address. The value of the practice is reducing waste, not buying position.
The hidden cost is that cleaning URL inventory and fixing status codes touches templates, routing, and server configuration. The work is cross-functional and slow, and regression is easy: a new template can reintroduce duplicate parameters the following month. The cost is front-loaded and then recurring, not a one-time fix. Teams often underestimate the coordination required, because the visible symptom is a crawl report while the actual change lives in code that several groups own.
Returning a 200 for a missing page, often called a soft error, or choosing the wrong redirect type sends a misleading signal. RFC 9110 defines what the codes mean, but applying them correctly is local work that depends on the site's logic. A wrong code can cause more wasted requests than the problem it was meant to solve. Because the crawler responds to what it is told, an incorrect response is worse than no response at all.
There is a boundary around the guidance. It covers crawler requests and HTTP responses and nothing beyond them. It does not address site authority, content quality, or links from other pages, and it cannot control how a particular crawler allocates its effort internally. The source describes that allocation only at a high level, so any specific claim about quantities should be treated as local inference. A team that expects the guidance to explain a sudden change in crawl volume will not find that explanation here.
Because sites change, response hygiene needs periodic review. Server logs and crawl reports should be checked against the status codes the site intends to return. Each decision should be recorded with a recheck condition, and any request-volume target should be marked as local rather than source-derived. Without that record, a future reviewer cannot tell whether a response was deliberate or an accident of configuration.
The counterargument that limiting URLs is fair for small sites is incomplete when a large catalog relies on generated parameters.
A decision test
- Inventory the URLs a crawler is able to request.
- Group URLs that serve the same or near-identical content.
- Check the status code each group actually returns.
- Replace soft errors with truthful codes.
- Consolidate duplicate URLs deliberately, using clear signals.
- Fix server errors that force repeated requests.
- Review logs against intended responses and record a recheck date.
Response hygiene is a discipline of alignment between what a page is and what the server says it is. The test makes that alignment checkable and keeps any target as a local decision rather than a documented promise. For further reading, see What AI Search Actually Measures.
References & Further Reading
- Google, 2026. Google Crawler (User Agent) Overview — How Google discovers, crawls, renders, and indexes pages.
- R. Fielding, M. Nottingham, and J. Reschke, 2022. HTTP Semantics (RFC 9110) — The normative definition of HTTP methods, status codes, and headers.