Crawl budget when AI bots show up at once

Several AI crawlers fetching the same site in the same hour is a load problem, not a policy problem, and it needs a load answer. The usual reaction is a Disallow line, which solves the load by removing the site from whichever assistant that crawler feeds. That trade is sometimes right. It is rarely what the person writing the rule meant to decide.

The tools that actually address load are caching, rate limiting by verified agent, and the HTTP status codes crawlers already understand. Crawl-delay helps with some crawlers and does nothing for most of the ones that matter. This post covers which crawlers document what, how to absorb the traffic before throttling it, how to throttle without touching search crawlers, and the one robots.txt mistake that turns an overloaded server into a site-wide block.

It sits under AI crawler access: the technical layer of AI search. If you have not yet confirmed who is actually hitting you, start with reading server logs for AI bot traffic, because the right response depends on which agents are responsible.

Is this actually a crawl budget problem?

Usually not in Google’s sense of the term, and it helps to separate the two meanings. Google’s crawl budget guide is written for large sites, which it roughly estimates as 1 million or more unique pages changing weekly, or 10,000 or more changing daily. For everyone else, Google’s own crawl is not limited by budget in a way worth managing.

What site owners mean when AI crawlers arrive is different: server capacity. Several independent crawlers, each with its own schedule, can overlap with each other and with real visitors. Google describes its own side of this as a crawl capacity limit, which it raises when responses are fast and stable and lowers when it sees slower responses, 5xx errors, or 429 rate-limit signals. That is the model to hold for every well-behaved crawler: they back off when the server says it is struggling. The question is how to say it.

Which crawlers honor Crawl-delay?

Few, and not the one that matters most. Crawl-delay is not part of the robots.txt standard. RFC 9309 says crawlers may interpret records outside the protocol, which means they are free to ignore them.

Google’s specification is blunt: other fields such as crawl-delay are not supported. The Bing entry rests on a 2012 post, which is the most recent primary statement we found; treat it as historical until Microsoft restates it. And “undocumented” means exactly that. It does not mean ignored, and it does not mean honored, so do not build a load plan on it.

If Anthropic’s crawlers are a real share of your traffic, a Crawl-delay line in their group is a legitimate tool:

User-agent: ClaudeBot
Crawl-delay: 2

Add it to that agent’s group only. A Crawl-delay in the * group does nothing for Google, and Bing reads only its own group when one exists, so a default delay meant for Bing has to be repeated there.

What should you do before throttling anything?

Make each crawler request cheap. A fetch that a CDN or reverse proxy answers from cache costs the origin almost nothing, and crawlers fetch the same public HTML that a cache is built for. Most load complaints about crawlers are complaints about uncached pages being rendered on every request.

Four checks, in order of return:

1. Is the HTML cached at the edge? Many setups cache images and scripts but pass every HTML request through to the application. For public, non-personalized pages, a short HTML cache lifetime absorbs repeated fetches.
2. Do crawler requests bypass the cache? Query strings, cookies, and Vary headers can make every crawler request a cache miss. Tracking parameters on internal links are a common cause.
3. Are crawlers wandering into infinite URL spaces? Faceted navigation, calendars, and internal search results generate URLs without end. A Disallow on those paths for all agents is a load fix that costs no visibility, because those URLs were never worth indexing.
4. Does robots.txt itself load fast? It is the one file every crawler fetches first.

The fourth is more important than it looks. RFC 9309 says that if robots.txt returns a server error, the crawler must assume complete disallow. An overloaded origin that times out on robots.txt is, for the duration, telling every compliant crawler to fetch nothing. Serve robots.txt as a static file from the edge so it stays up when the application does not. The same RFC says crawlers should not use a cached copy for more than 24 hours, so a short outage does not linger forever, but it is avoidable entirely.

How do you slow AI crawlers without blocking them?

Rate-limit by verified agent and answer the excess with 429 and a Retry-After header. A 429 says “too many requests, come back later,” which is the message you mean. A Disallow says “never come here,” which is a different decision.

Google documents this pattern for its own crawler. Its guide to reducing crawl rate says to return 500, 503, or 429 instead of 200 for a short period, optionally with Retry-After. It also warns against doing this for longer than 1 to 2 days: if Googlebot sees these codes on the same URL for multiple days, the URL may be dropped from the index. For a sustained rate problem with Google, the page points to a special request through Search Console instead.

For the AI crawlers, set the limits in the CDN or WAF, scoped per agent:

• Match the agent and its published IP range together. A user-agent string is self-declared. OpenAI and Perplexity publish IP lists as JSON, and Google supports reverse DNS verification. When your CDN or firewall blocks bots robots.txt allows covers writing these rules without catching the wrong traffic.
• Throttle training crawlers before search crawlers. A training crawler that is slowed costs you nothing on any answer surface today. A search crawler that is slowed may index you less completely. Training vs. retrieval crawlers sorts which agent is which.
• Never put Googlebot in an AI-crawler rule. Google’s AI features run on the same crawl as Search. A rule aimed at “AI bots” that catches Googlebot is a rule against Google Search.
• Use a per-agent budget, not a site-wide one. A single bot limit shared by every crawler lets one aggressive agent use up the allowance for the others.

The order to escalate in: cache, then Crawl-delay for the agents that honor it, then per-agent 429s, then a Disallow on specific heavy paths, and only last a Disallow: / for that agent. Each step down that list costs more visibility than the one before it.

How do you see which crawlers cause the load?

Start in your CDN or hosting dashboard; most of them break traffic down by user agent or bot category without any log access. Sort requests by agent over the last week, and compare the top crawlers against the overall request volume and the origin response times for the same window. If spikes in origin latency line up with one agent’s request rate, that agent is the one to address.

If your provider exposes raw logs in a spreadsheet-friendly export, a pivot table of requests by user agent and hour gives the same picture. Where neither exists, ask whoever runs the server for a daily count of requests per user agent with the response status, and a list of the twenty most-requested paths per agent. That is enough to tell an infinite-URL problem from a genuinely heavy crawler.

For those with shell access to an access log in the common combined format, this counts requests per AI agent:

grep -oE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Amazonbot|Bytespider|CCBot' access.log | sort | uniq -c | sort -rn

Remember that the user-agent string can be spoofed, so a large count is a lead, not a verdict, until it is checked against the published IP ranges. Reading server logs for AI bot traffic covers verification and the fuller analysis.

What should you check after changing anything?

Check that every search crawler can still reach the pages you want found. Load fixes have a way of drifting into access changes, especially WAF rules. After any change:

1. Re-read robots.txt and confirm no Disallow was added to a search agent by accident. The robots.txt tester evaluates a file against specific agents and paths.
2. Fetch a few important URLs with each search crawler’s user-agent string and confirm a 200, not a 403 or a challenge page.
3. Watch response codes by agent in the CDN dashboard for a few days. A rising share of 429s to a search crawler means the limit is too tight.

The AI crawlers reference lists the agents and what each one controls.

Where GEOBee fits

GEOBee reads robots.txt for every crawl and flags pages where AI crawlers are blocked, which is the check to run after a load change to confirm nothing was closed by accident. AI bot access is one of seven GEO signals; the three lenses lists the agents it checks. It reads robots.txt rules, not WAF or rate-limit behavior, so the fetch test above is still needed for that layer.

It is also a crawler, so it is subject to the same courtesy. The Threads setting defaults to 5 concurrent workers, and Crawl settings recommends dialing it back for fragile or rate-limited servers.

To confirm a load fix left your AI bot access intact across every page, download GEOBee and run a quick scan during the 7-day trial.