Skip to content
Novandri.
← All articles

Category

robots.txt

3 articles in this category.

Written by me.

Crawlingrobots.txtAI Search

AI Crawlers Are Not Googlebot: How to Govern Crawlers You Don't Control

A practical, evidence-based guide to AI crawler user agents, robots.txt governance, and the emerging IETF preference standard for AI usage.

12 min read
Read article
AI Searchrobots.txtTechnical SEO

llms.txt, robots.txt, and the IETF Drafts: Sorting AI File Formats by Evidence

What llms.txt is, what the IETF aipref drafts standardise, and why 'just add a file' is not yet an SEO technique — with the sources checked.

11 min read
Read article
Crawlingrobots.txtTechnical SEO

robots.txt Is a Parsing Specification: The Rules Your Crawler Actually Follows

RFC 9309 defines exactly how robots.txt groups, matching, precedence, and encoding work — and how to test your file against the spec.

11 min read
Read article