Sitemaps in AI search: what still matters
Which AI search systems document using XML sitemaps, why lastmod has to be honest, how sitemap index files work, and where llms.txt fits beside them.
crawlers
15 posts
Crawl budget when AI bots show up at once
Handling load from several AI crawlers at once: which honor Crawl-delay, how to cache and rate-limit, and how to slow bots without blocking search crawlers.
Reading server logs for AI bot traffic
Find AI crawlers in access logs, verify them against published IP ranges, read CDN and origin logs differently, and build the analysis in a spreadsheet.
nosnippet, max-snippet and data-nosnippet on AI surfaces
How Google's snippet directives apply to AI Overviews and AI Mode, how they combine, where an accidental max-snippet:0 comes from, and how to audit them.
AI Overviews eligibility: every technical requirement
What Google documents as the technical bar for AI Overviews and AI Mode: crawlable, indexed, snippet-eligible, and included. Plus what doesn't count.
When your CDN or firewall blocks bots robots.txt allows
How CDN bot-management settings and firewall rules override robots.txt for AI crawlers, how to detect it, and how to write an allow rule that holds.
JavaScript content and what never makes the trip
Which AI crawlers render JavaScript, which page patterns lose content on the way, and how to compare view-source against the rendered DOM on one page or a site.
llms.txt: what it is, how it works, and whether to bother
Is llms.txt worth it? The proposal, the file format with a worked example, what is documented about who reads it, and when the afternoon pays off.
Status codes, redirects, and canonical drift on AI surfaces
How redirect chains, soft 404s, 4xx and 5xx responses, and conflicting canonicals decide which URL gets fetched, indexed, and attributed — and how to check.
Training vs. retrieval crawlers: the block decision
Should you block GPTBot? Which AI agents train, which search, which fetch for a user, what each block costs, and four robots.txt policies to copy.
robots.txt patterns for AI: allow, block, and the middle
Four positions cover almost every site. The copy-paste file for each, the matching rules that quietly break them, and the maintenance the file needs.
What do AI crawlers extract from your page?
An element-by-element inventory of what AI crawlers keep, discard, and only conditionally read — plus the parts you can exclude on purpose.