Skip to content
Novandri.
← All articles

Category

Crawling

16 articles in this category.

Written by me.

Crawlingrobots.txtAI Search

AI Crawlers Are Not Googlebot: How to Govern Crawlers You Don't Control

A practical, evidence-based guide to AI crawler user agents, robots.txt governance, and the emerging IETF preference standard for AI usage.

12 min read
Read article
CrawlingHTTPWeb Infrastructure

How Crawlers Actually Fetch Pages: Protocol, Limits, and Caching on the Wire

What Googlebot and AI crawlers do at the HTTP level — protocols, encodings, size limits, conditional requests, and the crawl rate signal you can read in your logs.

11 min read
Read article
Crawlingrobots.txtTechnical SEO

robots.txt Is a Parsing Specification: The Rules Your Crawler Actually Follows

RFC 9309 defines exactly how robots.txt groups, matching, precedence, and encoding work — and how to test your file against the spec.

11 min read
Read article
Technical SEOCrawl BudgetWeb Infrastructure

Crawl Budget: When It Matters and How to Diagnose It

A realistic guide to crawl budget: understanding crawl capacity vs demand, identifying when crawl budget matters, and eliminating wasteful crawler traps.

7 min read
Read article
Technical SEOIndexingCrawling

Discovered, Crawled, Rendered, Indexed: Why Stages Differ

Diagnose why URLs get stuck in Google Search Console: the difference between Discovered, Crawled, Rendered, and Indexed, and how to fix indexing backlogs.

7 min read
Read article
Technical SEODNSWeb Infrastructure

DNS for SEO: How Domain Resolution Works and How to Debug It

A guide to DNS for technical SEO: how recursive resolution works, managing TTLs for zero-downtime cutovers, and debugging crawler DNS failures with dig.

7 min read
Read article
Technical SEOE-commerceCrawling

E-commerce Technical SEO: Product URLs and Faceted Navigation

Control e-commerce URL growth by classifying filters, sorting, and pagination, then define crawl, canonical, and indexation rules for each URL type.

6 min read
Read article
Technical SEOCrawlingWeb Infrastructure

How Googlebot Discovers and Crawls Pages

An engineering breakdown of how Googlebot discovers and crawls URLs: crawl frontiers, internal link graphs, sitemap processing, and crawl demand dynamics.

7 min read
Read article
Technical SEOHTTPWeb Infrastructure

HTTP Status Codes for SEO: Diagnose 200, Redirects, 404, and 5xx

A practical guide to HTTP status codes for SEO: how 200, 301, 302, 404, 410, 429, and 5xx impact crawling, indexing, and canonicalization with curl triage recipes.

12 min read
Read article
Technical SEOMobile IndexingCrawling

Mobile-First Indexing: Check Content and Rendering Parity

Audit mobile-first indexing parity: verify text, structured data, metadata, and media parity between desktop and smartphone Googlebot renders.

8 min read
Read article
Technical SEOCrawlingIndexing

Robots.txt vs Noindex vs Sitemap: What Each One Does

Learn the precise differences between robots.txt, noindex, and XML sitemaps: how crawl control differs from indexation control, and avoiding fatal directive conflicts.

7 min read
Read article
Technical SEOURL ArchitectureWeb Infrastructure

SEO URL Structure: Normalize Hostnames, Paths, and Parameters

Design deterministic, crawlable URL architectures: RFC 3986 anatomy, URL normalization rules, trailing slashes, parameter hygiene, and consistency auditing.

6 min read
Read article
Technical SEOServer LogsCrawling

Server Logs for SEO: See What Googlebot Requests

Analyze server access logs for technical SEO: verify genuine Googlebot IP addresses, trace crawl behavior, detect 5xx errors, and isolate crawl traps.

8 min read
Read article
Technical SEOTechnical AuditCrawling

A Technical SEO Audit That Follows the Request Path

Run a risk-based technical SEO audit from DNS and HTTP delivery through rendering, crawling, and indexing, tying each finding to evidence and an owner.

6 min read
Read article
Technical SEODebuggingCrawling

Three Technical SEO Bugs Diagnosed from Symptom to Root Cause

Work through three clearly labeled technical SEO lab scenarios: a 200 page with missing rendered content, a WAF-blocked crawler, and conflicting URL signals.

7 min read
Read article
Technical SEOWeb InfrastructureCrawling

What Happens When a Crawler Requests a URL?

A step-by-step breakdown of DNS resolution, TCP/TLS handshakes, HTTP transmission, and rendering queues, detailing where crawling fails.

10 min read
Read article