Skip to content
Novandri.
All articles
Technical SEOWeb InfrastructureCrawlingHTTP

What Happens When a Crawler Requests a URL?

A step-by-step breakdown of DNS resolution, TCP/TLS handshakes, HTTP transmission, and rendering queues, detailing where crawling fails.

Rachmadwipa Novandri10 min read

When a search engine crawler fetches a URL, it does not behave like a typical user browsing in Chrome or Safari. It operates as an automated, high-throughput batch pipeline designed to discover, parse, and store web documents while strictly budgeting computing and network resources.

Direct Answer: When a crawler requests a URL, it resolves the domain via DNS, negotiates TCP and TLS connections, dispatches an HTTP GET request, inspects response headers, streams the initial HTML payload, and queues discovered links. Unlike browsers, crawlers decouple initial HTML crawling from resource-heavy JavaScript execution, deferring rendering to a secondary queue or skipping it entirely if initial access fails.

Understanding each layer of this transaction reveals exactly why technical SEO issues occur long before a search engine ever considers page content or keywords.

[Crawler Frontier]
       │
       ▼ (1. DNS Resolution)
[Authoritative DNS] ──► IP Address
       │
       ▼ (2. Connection & Security)
[TCP 3-Way Handshake + TLS 1.3] ──► Encrypted Socket
       │
       ▼ (3. HTTP Request)
[Edge / CDN Cache] ──── HIT ───► Fast Response (200 OK)
       │ (MISS)
       ▼
[Origin Web Server / Reverse Proxy]
       │
       ▼
[App Runtime / Static File]
       │
       ▼ (4. Stream Response & Headers)
[Crawler Ingestion Engine]
  ├── Status & Header Check (X-Robots-Tag, Content-Type)
  ├── Fast Path: Parse Raw HTML & Extract Links
  └── Deferred Path: Web Rendering Service (Headless Chromium)

1. DNS Resolution: Mapping the Hostname

The lifecycle begins in the crawler's scheduling frontier. Once a URL is picked for crawling, the crawler must translate the hostname (e.g., example.com) into an IP address.

  1. Resolver Cache Check: The crawler checks its local caching DNS resolvers. If the record is cached within its Time-to-Live (TTL), the lookup is instantaneous.
  2. Recursive Query: If the record has expired or is uncached, the resolver queries the root nameservers, the Top-Level Domain (TLD) nameservers, and finally the domain’s authoritative nameservers for A (IPv4) or AAAA (IPv6) records.

Failure Modes and SEO Impact

If authoritative nameservers fail to respond within timeout thresholds (typically 2 to 5 seconds for automated bots), or return SERVFAIL, the request terminates immediately. In Google Search Console, this manifests as a DNS error under Crawl Stats. Repeated DNS failures signal host instability, prompting crawlers to aggressively scale down their crawl rate to prevent hammering an ostensibly failing infrastructure.

2. Network Handshake: TCP and TLS Negotiation

Once an IP address is resolved, the crawler opens a network socket with the destination server or CDN edge node.

TCP 3-Way Handshake

To establish reliable transmission, client and server perform the standard SYN, SYN-ACK, ACK exchange defined in RFC 793. Over transatlantic connections, this single round trip can introduce 60–120ms of baseline network latency before any data transfers.

TLS Handshake (RFC 8446)

On HTTPS connections, the crawler immediately negotiates TLS. In TLS 1.3, this exchange is streamlined to a single round-trip:

  • Client Hello: The crawler sends supported cryptographic cipher suites, a key share, and the Server Name Indication (SNI) extension indicating the target hostname.
  • Server Hello & Certificate: The server returns its selected cipher, its key share, and its cryptographic certificate chain.
  • Verification: The crawler validates the certificate against its trusted certificate authorities (checking expiration, host mismatch, and revocation status).

Failure Modes and SEO Impact

If the server's certificate is expired, missing intermediate certificates, or presents an SNI mismatch, modern search crawlers will abort the connection immediately. Googlebot will not bypass self-signed or invalid TLS certificates. The fetch is recorded as a Failed: HTTPS error, blocking crawling and subsequent indexing entirely.

3. The HTTP Request: Headers, Compression, and Edge Routing

With an encrypted channel established, the crawler dispatches an HTTP request (typically HTTP/2 or HTTP/1.1):

GET /path/to/resource HTTP/2
Host: example.com
User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate, br

Key characteristics of a crawler request:

  • User-Agent: Googlebot uses specific mobile and desktop user-agent tokens.
  • Accept-Encoding: Modern crawlers support Brotli (br) and Gzip (gzip), drastically cutting payload byte transfer.
  • No Cookies / LocalStorage: Crawlers generally send stateless requests. They do not persist sessions, shopping carts, or authentication cookies.

Edge Caching vs. Origin Fetch

In production architectures, this request strikes an Edge server (CDN or Reverse Proxy like Cloudflare, Fastly, or Nginx):

  • Cache HIT: The edge returns the static HTML immediately without touching the application server.
  • Cache MISS: The edge opens an upstream connection to the application server (e.g., Node.js, PHP-FPM, Python), which runs business logic, queries databases, and streams rendered HTML back to the edge.

Failure Modes and SEO Impact

  • WAF & Rate Limiter Blocks (403 Forbidden): Web Application Firewalls (Cloudflare Under Attack mode, AWS WAF rules) often mistake aggressive crawler concurrency for a Layer 7 DDoS attack. A 403 status drops pages from the index if persistent.
  • Origin Overload (502 Bad Gateway / 504 Gateway Timeout): When a crawler hits hundreds of cache-miss URLs concurrently, unoptimized database queries can exhaust application worker pools, generating 502/504 errors that force crawlers into an exponential backoff state.

4. Response Evaluation: Status Codes and Response Headers

Before reading the HTML body, the crawler processes the HTTP response line and headers:

HTTP/2 200 OK
content-type: text/html; charset=UTF-8
content-encoding: br
x-robots-tag: noindex, nofollow
cache-control: public, max-age=3600
last-modified: Wed, 11 Nov 2026 08:30:00 GMT
etag: W/"4f5a-605d..."

The crawler makes immediate architectural decisions based on these headers:

  1. Status Code:
    • 200 OK: Valid document; proceeds to parsing.
    • 301/308 or 302/307: Redirects; crawler queues the Location target URL for subsequent crawling (up to a fixed redirect limit, typically 5 to 10 hops).
    • 404 / 410: Discards document; scheduled for de-indexing.
    • 429 / 5xx: Triggers crawler back-off to protect server health.
  2. Content-Type: Must match text/html (or application/pdf, image/png, etc.). If a server returns text/plain for an HTML page, crawlers may not parse links or structure correctly.
  3. X-Robots-Tag: Directives like noindex or noarchive take effect immediately at the header level before body processing.

5. Raw HTML Processing vs. Deferred Rendering

Once headers pass validation, the crawler streams the raw HTML payload. Here lies the fundamental difference between a browser and a search engine crawler.

Crawlers do not wait for assets to download or scripts to execute before extracting URLs. An internal streaming parser inspects the raw HTML markup immediately:

  • Discovers internal and external <a href="..."> links and adds them to the crawl frontier queue.
  • Reads <meta name="robots"> tags.
  • Identifies <link rel="canonical"> signals.

If your internal links or crucial navigation elements are injected exclusively through JavaScript event listeners (<div onClick="...">), the raw HTML parser misses them completely during this phase.

The Deferred Path: Web Rendering Service (WRS)

Executing JavaScript at the scale of billions of web pages requires massive server infrastructure. Googlebot utilizes a two-stage indexing model:

  1. Wave 1 (Crawl & Parse): Raw HTML is fetched, status codes recorded, static links discovered, and static text indexed.
  2. Wave 2 (Render): If the page relies on client-side rendering (CSR), the document is queued for the Web Rendering Service, which loads the HTML into a headless Chromium instance to execute JavaScript, populate the DOM, and evaluate rendered CSS.

Depending on global processing queues and site reputation, the delay between Wave 1 and Wave 2 can range from seconds to several days. If critical metadata, content, or canonical tags exist only in the client-rendered DOM, indexing will experience measurable latency.

6. Diagnostic Workflow: Tracing the Request Path with curl

To diagnose bottlenecks along the request lifecycle, run curl with a custom timing format string. This isolates exactly where latency or connection failures occur.

Create a formatting file or execute inline:

curl -s -w "\
Lookup Time (DNS):     %{time_namelookup}s\n\
Connect Time (TCP):    %{time_connect}s\n\
TLS Handshake:         %{time_appconnect}s\n\
Pre-Transfer Time:     %{time_pretransfer}s\n\
Time to First Byte:    %{time_starttransfer}s\n\
Total Request Time:    %{time_total}s\n\
HTTP Response Code:    %{http_code}\n" \
-o /dev/null -H "User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
https://example.com/demo-path

Sample Output Interpretation

Lookup Time (DNS):     0.021s
Connect Time (TCP):    0.054s
TLS Handshake:         0.089s
Pre-Transfer Time:     0.089s
Time to First Byte:    0.284s
Total Request Time:    0.312s
HTTP Response Code:    200

How to evaluate these numbers:

  • time_namelookup (> 0.100s): Slow authoritative DNS resolution. Consider switching to a distributed, anycast DNS provider.
  • time_connect - time_namelookup (> 0.150s): Network routing friction between the testing location (or crawler edge) and your host.
  • time_appconnect - time_connect (> 0.150s): Inefficient TLS termination, outdated cipher negotiation, or lack of TLS 1.3 support.
  • time_starttransfer - time_pretransfer (Backend TTFB, > 0.500s): Application origin bottleneck. Indicates uncached database queries, slow template compilation, or application worker saturation.

7. Lifecycle Failure Matrix

The following decision matrix maps each stage of the request path to its failure symptoms and diagnostic resolution:

Lifecycle StageSymptom / ErrorCrawler ConsequenceRoot Cause & Diagnostic Step
DNS ResolutionSERVFAIL, NXDOMAIN, TimeoutImmediate crawl abort; reduced crawl budget; Crawl Stats DNS alert.Misconfigured NS records or nameserver DDoS. Verify via dig +trace example.com.
TCP / TLS HandshakeSSL_ERROR_BAD_CERT_DOMAIN, TCP ResetCrawl failure recorded as HTTPS error; page cannot be fetched or indexed.Expired certificate or missing SNI configuration. Test using openssl s_client -connect host:443 -servername host.
Edge / WAF403 Forbidden, CAPTCHA challengeContent blocked; existing indexed pages risk removal if status persists.Security rules blocking crawler User-Agents or IP ranges. Inspect WAF block logs and whitelist verified Googlebot CIDRs.
Server Response500, 502 Bad Gateway, 504 TimeoutCrawler triggers exponential back-off; crawl rate drops across entire host.Unhandled app exceptions or reverse proxy upstream timeouts. Inspect web server error logs (/var/log/nginx/error.log).
HTML TransferTruncated payload, incorrect MIME typeParsing failure; links not extracted; page treated as unsupported file.Faulty compression (gzip/br chunk encoding error) or missing Content-Type: text/html.
Rendering QueueMissing content in rendered DOM, script timeoutClient-side content not indexed; links missed during Wave 2.Heavy JavaScript bundle, unhandled Promise rejection, or blocking API calls. Verify in Search Console URL Inspection Live Test.

Where to Go From Here

Understanding the request lifecycle clarifies that technical SEO is, at its foundation, web infrastructure engineering. Every millisecond lost in DNS resolution, every TLS handshake failure, and every unhandled 5xx error directly impacts how effectively search engines discover and understand your site.

In upcoming chapters of this knowledge hub, we will examine specific layers of this stack in granular detail:

References & Further Reading