How to check whether AI crawlers can actually reach you
Reading robots.txt tells you what you intended. Six checks that tell you what actually happens when an AI crawler requests one of your pages.
crawlers
15 posts
GPTBot vs OAI-SearchBot: a field guide to the AI crawlers
Every AI bot worth naming in your robots.txt, what each one actually controls, and which ones decide whether an assistant can find you at all.
AI crawler access: the technical layer of AI search
Which AI bots exist, what each one actually controls, why robots.txt is only half the answer, and how to check whether assistants can reach your pages.