GPTBot vs OAI-SearchBot: a field guide to the AI crawlers
Every AI bot worth naming in your robots.txt, what each one actually controls, and which ones decide whether an assistant can find you at all.
fundamentals
16 posts
AI crawler access: the technical layer of AI search
Which AI bots exist, what each one actually controls, why robots.txt is only half the answer, and how to check whether assistants can reach your pages.
The 40–60 word answer block, and how to write one
The highest-weighted answer-readiness signal is a passage that answers its own heading in one breath. What it is, how to write one, how to check it.
What an answer engine reads when it reads your page
A walk through what actually gets extracted from a page when an AI system reads it, and which parts of your HTML never make the trip.