Skip to content
Novandri.
← All articles

Category

HTTP

5 articles in this category.

Written by me.

HTTPCachingWeb Infrastructure

Cache-Control Is a Contract, Not a Suggestion: Reading HTTP Freshness Correctly

How RFC 9111 freshness, revalidation, and Age actually work — and why s-maxage, stale-while-revalidate, and Vary decide what your users see.

8 min read
Read article
CrawlingHTTPWeb Infrastructure

How Crawlers Actually Fetch Pages: Protocol, Limits, and Caching on the Wire

What Googlebot and AI crawlers do at the HTTP level — protocols, encodings, size limits, conditional requests, and the crawl rate signal you can read in your logs.

11 min read
Read article
Crawlingrobots.txtTechnical SEO

robots.txt Is a Parsing Specification: The Rules Your Crawler Actually Follows

RFC 9309 defines exactly how robots.txt groups, matching, precedence, and encoding work — and how to test your file against the spec.

11 min read
Read article
Technical SEOHTTPWeb Infrastructure

HTTP Status Codes for SEO: Diagnose 200, Redirects, 404, and 5xx

A practical guide to HTTP status codes for SEO: how 200, 301, 302, 404, 410, 429, and 5xx impact crawling, indexing, and canonicalization with curl triage recipes.

12 min read
Read article
Technical SEOWeb InfrastructureCrawling

What Happens When a Crawler Requests a URL?

A step-by-step breakdown of DNS resolution, TCP/TLS handshakes, HTTP transmission, and rendering queues, detailing where crawling fails.

10 min read
Read article