When an AI Agent Comes Knocking, What Does Your Server Need?

That list of AI crawler IPs sitting in your WAF — who pasted it in, and when?

We ask because on 2026-08-05 we pulled four official bot IP lists at once and put them side by side, and the gap was frankly absurd. OpenAI's GPTBot, which handles training crawls, publishes 21 prefixes, with a creationTime frozen at 2025-10-30 — nine months without a change. The user-triggered ChatGPT-User publishes 289 prefixes, with a creationTime of 01:03 that same morning. Same company, nominally the same kind of thing, roughly 14× apart in scale.

There is a harder problem. That day we called the endpoints twice, minutes apart, and ChatGPT-User's prefix count went from 290 to 289. A list that can change in the time it takes you to make coffee — how long does a hand-pasted copy of it survive inside a WAF?

That is what this article is about: user-triggered AI Agent traffic and training crawlers are not the same thing in engineering terms, yet most sites currently govern both with one set of bot rules. We will pull the two apart, explain the three mechanisms that break a hand-maintained whitelist, list the four things your server has to get right, and close with a sync script you can run as-is — including the trap where Anthropic's own list rejects Python's default User-Agent.

Caption: On the left, scheduled bulk crawling — it can wait in line. On the right, someone is mid-conversation and the system is fetching right now — there is no second attempt.

User-Triggered Agents Publish 14× More Prefixes Than Training Crawlers

Start with the numbers. The four lists below were pulled first-hand on 2026-08-05 — not quoted from someone else's roundup:

List	Purpose	Published at	Prefixes	creationTime
OpenAI GPTBot	Training crawl	openai.com/gptbot.json	21	2025-10-30
OpenAI OAI-SearchBot	Search indexing	openai.com/searchbot.json	35	2026-01-02
OpenAI ChatGPT-User	User-triggered real-time fetch	openai.com/chatgpt-user.json	289	2026-08-05 01:03
Anthropic (shared by three bots)	Training / user-triggered / search	claude.com/crawling/bots.json	20 (19 of them /32)	2026-05-01

(All measured 2026-08-05. All four numbers move; re-run this yourself before you rely on it.)

Want to verify it? One line is enough:

curl -s https://openai.com/chatgpt-user.json \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d['prefixes']), d['creationTime'])"

The most telling thing in that table is not the absolute counts, it is the gap in update cadence. The training crawler's egress IPs are extremely stable — nine months without a single change, which says those machines sit permanently on those prefixes and crawl slowly, on a schedule. User-triggered fetching is the exact opposite: the list had been rewritten in full that same morning. What follows is our reading, not an official statement: the two are triggered in fundamentally different ways. One is "we scheduled a pass over the web." The other is "someone is talking to us right now and the system has to go get that page this second." To fetch it this second, you go out from whichever machines are closest to the user and have headroom at that moment — so the egress footprint is naturally wide and scattered.

If the term AI Agent is still fuzzy, fill in the background with our complete guide to what an AI Agent is, how it works, and which tools matter first, then come back to the rule design below.

Traffic That Can Queue vs. Traffic With Someone Waiting: Five Engineering Differences

Apply one set of bot rules to both kinds of traffic and one side is guaranteed to be handled wrong — and the side that usually gets it wrong is user-triggered agent traffic being throttled as if it were a crawler.

Where exactly do they differ? Laid out:

Dimension	Training / bulk crawl (e.g. GPTBot)	User-triggered real-time fetch (e.g. ChatGPT-User, Claude-User)
Trigger	Scheduled, batched	A single question a user just asked
Shape of the load	Continuous, predictable	Sporadic, spiky, unpredictable
Latency tolerance	High	Extremely low
Cost of a timeout	It crawls again next time	That user does not see your content, and there is no retry
Appropriate handling	Rate-limit, relax the cache	Protect response speed above all

Row four is the whole point of the table. When a training crawler times out, the loss is close to zero — it comes back. When a user-triggered fetch times out, the loss is one real impression: someone asked a question, the AI went to fetch your page, did not get it, and the answer ends up citing somebody else's content. There is no make-up exam.

The pattern we run into most often while auditing clients' cloud environments is not a misconfigured rule — it is a rule that was never split in the first place. One "AI bot: N requests per minute" rule sits there, GPTBot and ChatGPT-User side by side in the UA match table with identical settings, and whoever wrote it never considered them two different things. That configuration causes no visible trouble on an ordinary day, and when it does cause trouble, nothing alerts: a blocked request is just one more 403 line in the log.

As for how to size that load row, and how to work the bandwidth bill back out of your own CDN data, there is another angle in how much bandwidth AI crawlers actually consume, calculated from your own CDN logs. If your organization runs agents internally as well, the traffic characteristics follow the same logic in both directions — the deployment scenarios in the AI Agent enterprise application guide are worth reading alongside this.

Why a Hand-Maintained IP Whitelist Is Guaranteed to Break

Not "might" — will. The only open question is when. Three mechanisms, independent of each other, and they compound.

First, the cadences are so far apart that no single maintenance interval is correct. One list has not moved in nine months; the other was updated today. Which one do you schedule maintenance around? Schedule around GPTBot and ChatGPT-User is several revisions stale. Schedule around ChatGPT-User and somebody has to do this every single day. Manual scheduling cannot absorb two inputs whose cadences differ this much.

Second, the orders of magnitude differ, and "paste it in by hand" does not survive 289 rules. 21 prefixes: one person, ten minutes, and the work is still checkable. 289? You will skip a line, transpose a digit, drop a character — and nobody reviews 289 entries one by one. More to the point: at the next update you need to know which entries changed and which were removed, and eyeballing a 289-line diff is not a thing that happens.

Third, it moves while you are not looking. Measured 2026-08-05: two calls minutes apart took ChatGPT-User's list from 290 to 289. That is not monthly, not weekly — it can change at any moment.

Caption: A stale hand-maintained whitelist does not break and does not alert — it just quietly leaves requests from new prefixes standing outside the door.

Stack the three mechanisms and the conclusion is simple: the whitelist has to be automated. A hand-maintained one failing is only a matter of time.

And the shape of that failure is the genuinely nasty part — it is not a crash, it is quietly dropping real traffic. The server does not slow down, the service does not go down, the dashboard stays green, and the only thing you will ever see is a few more 403s or 429s in the log — each of which was a person waiting for an answer. No default alerting rule anywhere will tell you this is happening.

---

Who Owns This Sync Chain at Your Company?

289 prefixes, and still moving that same day — leave this sync chain un-automated and it will fail sooner or later, with nobody noticing. The CloudInsight technical team can help you inventory the rules in your cloud environment that need periodic syncing, and put automation and alerting behind them.

👉 Contact our technical team

---

OpenAI Publishes Three Lists, Anthropic Publishes One: What That Means for Your Rules

Both vendors publish IP lists, but they do it differently, and that difference directly caps how granular your rules can get.

OpenAI publishes three separately: gptbot.json (21 prefixes), searchbot.json (35), and chatgpt-user.json (289) — all measured 2026-08-05. Three separate lists means you can distinguish training, search, and user-triggered traffic at the IP layer, without depending on the User-Agent. Throttle the training crawl, let user-triggered fetches through: that rule is writable.

Anthropic publishes one. claude.com/crawling/bots.json measured 20 prefixes on 2026-08-05, shared by ClaudeBot (training), Claude-User (user-triggered), and Claude-SearchBot (search). What does that mean in practice? The IP layer cannot tell you whether a request is training or serving someone mid-conversation, so any differentiated handling falls back to User-Agent matching.

And a User-Agent can be forged. Cloudflare's verified bots documentation (page marked Last updated 2026-07-01) lists honest self-identification as one of the verification bars, met through Web Bot Auth cryptographic signatures, a published IP list paired with a stable UA, or reverse DNS; its separate fake bot managed rules explain that a request matching a known bot's UA but coming from an unverifiable source gets flagged as a fake bot. The practical conclusion is therefore unambiguous: when you handle Anthropic traffic differently, UA matching needs the IP list as a second gate. Both gates, not one. To build that verification layer out properly, see how to confirm an AI crawler's real identity.

There is also an easily missed difference in prefix style. 19 of Anthropic's 20 prefixes are /32 — single IPs; OpenAI's are mostly /24 and /28. "One rule" therefore means structurally very different things in terms of the WAF rules or IP list objects it consumes — if your plan caps IP list entries, do that arithmetic before you design anything.

One last point, straight from the vendor. Per Anthropic's official documentation, blocking by IP may not achieve opt-out correctly or durably, because it also prevents the crawler from reading your robots.txt; the same page describes the division of labour across the three bots and notes support for Crawl-delay. In other words, even when your goal is to block, the vendor advises against making IP blocking the primary mechanism. The correct use of an IP list is identity verification and differentiated service, not a gate.

The robots.txt standard spells the mechanics out more precisely. IETF RFC 9309, "Robots Exclusion Protocol" (Standards Track, September 2022), Section 2.3.1.3 covers the case where a crawler fetches robots.txt and receives a 4xx status code — 403 being one of them: "If a server status code indicates that the robots.txt file is unavailable to the crawler, then the crawler MAY access any resources on the server." It may treat your entire site as unrestricted. Section 2.3.1.4 says the opposite: when the answer is a 5xx, or the network layer is simply unreachable, "this means the robots.txt file is undefined and the crawler MUST assume complete disallow." Same plain-English description — "could not read robots.txt" — but under the standard, a 403 and a 500 are opposite outcomes, and what an IP rule blocks is usually the former.

Worth noting: Crawl-delay is not in RFC 9309 either. The whole standard defines only three record types — user-agent, allow, and disallow — and about everything else it says "Crawlers MAY interpret other records that are not part of the robots.txt protocol" (§2.2.4). A vendor supporting it is an extra courtesy, not an obligation you can invoke.

Four Things to Get Right on the Server Side: Timeouts, Cache, Status Codes, First Response

Once agent traffic is split out of your crawler rules, the next thing to change is the server itself. Four items, in the order you should tackle them.

① Give the real-time path its own timeout budget. User-triggered fetches travel the "someone is waiting" path. Setting its timeout to the same value as the crawler path means serving a human to a robot's standard. In practice this path should have a tighter budget: better to return a slightly leaner version quickly than to leave the caller hanging until the connection drops. For how to plan resources and timeouts together, see the resource allocation chapter of the complete server guide: types, selection, and setup.

② Split the cache strategy. Relax TTL on content pages for agents, so origin fetches drop to a minimum while response speed holds — those two are really one thing seen from two sides: a cache hit is fast, an origin fetch is slow. The key to splitting is not letting agent requests bypass the cache layer (some rules bypass cache for bot UAs outright, which sends exactly the traffic that most needs speed down the slowest path). For tuning hit rate and compression settings, the CDN optimization playbook covers the operational side in more depth.

③ Get the error semantics right. If you are rate limiting, return 429 with a Retry-After header — do not substitute 403 or 503. This is not pedantry: the status code is a signal you send, and the other side decides what to do next based on the code you return. A 403 says "you have no permission, stop coming." A 503 says "I am down." Only 429 says "too fast, come back shortly." The price of the wrong code is a temporary throttle being read as a permanent refusal.

④ Ship the body text in the first response. A fetcher that takes the HTML once and never executes JavaScript sees exactly what your first response contained. Body text that only materializes after front-end rendering does not exist as far as it is concerned. There is no middle ground here — it is not "slightly worse," it is zero. To check this layer systematically across the whole site, run the AI crawler accessibility audit process.

Caption: The four server-side items — ① a separate timeout budget (stopwatch) ② a split cache strategy (stacked discs) ③ correct status code semantics (traffic signal) ④ complete body text in the first response (jigsaw)

---

Timeouts, Cache and WAF Rules Scattered Across Different Platforms?

Timeout budgets, cache splitting and WAF rules each live somewhere different on a different platform, and aligning them gets harder still across clouds. CloudInsight resells AWS, GCP, Azure, Alibaba Cloud and Tencent Cloud, with unified billing management and Chinese-language technical support in Taiwan's time zone.

👉 Consult on enterprise plans now｜Join us on LINE for real-time consultation

---

Automating the Official IP List Sync: Three Real Problems One Script Has to Solve

At this point "write a cron job that fetches the official JSON" sounds like thirty minutes of work. In practice you hit three things, and the first has nothing to do with your program logic.

Getting a response: Anthropic's list blocks Python's default UA

Measured 2026-08-05, same URL https://claude.com/crawling/bots.json, only the User-Agent changed, and the result changed with it:

User-Agent	HTTP status code
Python-urllib/3.13	403
curl/8.7.1	200
Mozilla/5.0 ... Chrome/128.0 Safari/537.36	200

Two lines are enough to reproduce it yourself:

curl -s -o /dev/null -w '%{http_code}\n' -A 'Python-urllib/3.13' https://claude.com/crawling/bots.json
curl -s -o /dev/null -w '%{http_code}\n' -A 'curl/8.7.1'         https://claude.com/crawling/bots.json

Use Python's standard-library urllib without setting a User-Agent explicitly and what goes out on the wire is Python-urllib/3.x — hence the 403. What makes this trap nasty is how it fails: urlopen() raises HTTPError: 403, and if the script runs under cron, nobody reads stderr, or something wraps it in a try/except that simply passes, you get no notification whatsoever — only a list frozen forever at an old revision. The rules that manage bot identity verification end up blocked by bot protection themselves. The loop is a little funny, but it is real.

Passing validation: never let an empty list overwrite live rules

The second problem is putting the guard in front of the write. At minimum, check three things: was the HTTP status 200, is the parsed entry count non-zero, and is the delta against the previous version within threshold. If any one fails, abort and alert — do not write. What happens without the guard? An empty list quietly overwrites a good one.

The reason goes back to the "silent failure" character of hand-maintained whitelists: an emptied whitelist does not take the service down, it just starts blocking every agent request while your dashboard stays entirely green.

Pushing it live: the two vendors' prefix styles consume different rule quotas

The third problem lives in the conversion step. When you turn the lists into WAF rules or IP list objects, remember that Anthropic's is mostly single-point /32 (measured 2026-08-05, 19 of the 20 prefixes are /32) while OpenAI's is mostly /24 and /28. "Syncing one list" consumes a structurally different quota in each case, so capacity planning has to be done separately for each.

The script below covers all three problems and runs as-is:

#!/usr/bin/env python3
"""Sync the official AI bot IP lists into CIDR files ready to feed a WAF."""
import json
import sys
import urllib.error
import urllib.request
from pathlib import Path

# Key detail: with no UA set, urllib sends Python-urllib/3.x and claude.com returns 403.
UA = "Mozilla/5.0 (compatible; bot-ip-sync/1.0; +https://example.com/contact)"
TIMEOUT = 20
SOURCES = {
    "openai-gptbot": "https://openai.com/gptbot.json",
    "openai-searchbot": "https://openai.com/searchbot.json",
    "openai-chatgpt-user": "https://openai.com/chatgpt-user.json",
    "anthropic-bots": "https://claude.com/crawling/bots.json",
}
MAX_DELTA_RATIO = 0.30          # Abort rather than overwrite live rules if the count moves >30% from the previous version
OUT_DIR = Path("ip-lists")

def fetch(url: str) -> dict:
    req = urllib.request.Request(
        url, headers={"User-Agent": UA, "Accept": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
        if resp.status != 200:
            raise RuntimeError(f"HTTP {resp.status}")
        return json.loads(resp.read().decode("utf-8"))

def extract_cidrs(payload: dict) -> list:
    out = []
    for item in payload.get("prefixes", []):
        cidr = item.get("ipv4Prefix") or item.get("ipv6Prefix")
        if cidr:
            out.append(cidr)
    return sorted(set(out))

def sync_one(name: str, url: str):
    data = fetch(url)
    cidrs = extract_cidrs(data)
    if not cidrs:
        raise RuntimeError("empty list, refusing to overwrite live rules")

    target = OUT_DIR / f"{name}.txt"
    if target.exists():
        previous = [ln for ln in target.read_text().splitlines() if ln.strip()]
        if previous:
            delta = abs(len(cidrs) - len(previous)) / len(previous)
            if delta > MAX_DELTA_RATIO:
                raise RuntimeError(
                    f"count went from {len(previous)} to {len(cidrs)}, over the "
                    f"{MAX_DELTA_RATIO:.0%} threshold, aborted"
                )

    OUT_DIR.mkdir(parents=True, exist_ok=True)
    target.write_text("\n".join(cidrs) + "\n", encoding="utf-8")
    return len(cidrs), data.get("creationTime")

def main() -> int:
    failed = 0
    for name, url in SOURCES.items():
        try:
            count, created = sync_one(name, url)
            print(f"[OK]   {name}: {count} prefixes, creationTime={created}")
        except (urllib.error.HTTPError, urllib.error.URLError,
                RuntimeError, ValueError) as exc:
            failed += 1
            print(f"[FAIL] {name}: {exc}", file=sys.stderr)
    if failed:
        print(f"{failed} list(s) failed to sync, live rules left untouched.", file=sys.stderr)
    return 1 if failed else 0

if __name__ == "__main__":
    sys.exit(main())

We actually ran this script on 2026-08-05: all four lists came back, and the output counts matched that day's measured values (21 / 35 / 289 / 20). Then we truncated one of the output files down to a single line and ran it again — the count guard blocked the overwrite exactly as intended and exited with a non-zero status. Do not skip the non-zero exit code — it is the only signal that lets your scheduler raise an alert on your behalf.

How to tell whether you have prepared enough

Three signals, all of them already in your own logs, no extra tooling required:

1. Whether agent UAs appear at all, and in what proportion — do ChatGPT-User, Claude-User and OAI-SearchBot show up? What share does each hold? Not appearing at all usually means you are blocking them at some earlier layer.
2. The share of 429s you return to agents — is rate limiting catching real-time fetches by mistake? This share should sit far below the equivalent figure for training crawlers.
3. The timeout rate on agent requests — if this path's timeout rate looks roughly the same as ordinary user traffic, your timeout budget probably is not genuinely split.

For the record, our own site's robots.txt sets Allow for every mainstream AI crawler — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Claude-SearchBot and Google-Extended (measured 2026-08-05). Open the door first; everything downstream only means something after that.

These signals can only tell you that your server catches agent requests. Catching them is the entry bar, not the outcome — a successful request is not the same as content being used. To confirm the other half, you have to look from the AI answer's end back toward your site. AI SEO Hacker, run by the same team as CloudInsight, has written up how to check whether AI answers are citing your content — the question you only get to ask once the server side is done.

Frequently Asked Questions

Q: What is the difference between ChatGPT-User and GPTBot, and can I block just one of them?

A: GPTBot handles training crawls; ChatGPT-User fetches a page in real time only when a user asks for it mid-conversation. Their IP lists are published separately (measured 2026-08-05: 21 and 289 prefixes respectively), so yes, you can block just one side. Blocking GPTBot affects training data. Blocking ChatGPT-User affects the person asking a question right now.

Q: Why do Claude's three bots share one IP list, and what does that mean for my rules?

A: ClaudeBot, Claude-User and Claude-SearchBot all share claude.com/crawling/bots.json (measured 2026-08-05: 20 prefixes, 19 of them single-IP /32s). That means the IP layer cannot separate training from user-triggered traffic, so differentiated handling falls back to User-Agent matching — and a UA can be forged, so you need the IP list as a second gate.

Q: How often should the official IP lists be synced?

A: It depends which one you are matching against. Measured 2026-08-05, GPTBot's list had not moved in nine months, ChatGPT-User's had been updated in the early hours of that same day, and two calls minutes apart took it from 290 to 289. With cadences that far apart, the practical approach is a single scheduled automatic sync for all of them plus an alert on abnormal count changes — not a human deciding when an update is due.

Q: What happens when an AI Agent visit times out?

A: There is a person waiting for an answer behind a user-triggered fetch, so the consequence of a timeout is nothing like it is for a training crawler — it is not "it will crawl again next time," it is that this user does not get your content, and this particular chance does not come round again. So this path's timeout budget should be tighter than the crawler path's, and it should not share one timeout configuration with bulk crawling.

Q: My site needs JavaScript to display content — can AI Agents read it?

A: Not necessarily, and the risk is high. A fetcher that takes the HTML once without executing JavaScript sees only what was in the first response; body text that only appears after front-end rendering does not exist as far as it is concerned. The safest approach is to confirm your main text ships inside the first response's HTML, then actually fetch the page with different User-Agents and compare the differences.

Conclusion: Plan for Agent Traffic as User Traffic, Not as Crawler Traffic

This entire article really comes down to one sentence: user-triggered AI Agent traffic is user traffic underneath — it just arrives in a different shell.

The evidence behind that sentence is the table. Measured 2026-08-05, ChatGPT-User publishes 289 prefixes and had been updated in the early hours of that morning; GPTBot has only 21 and has not moved in nine months — one is a live service network in constant motion, the other a stable batch-crawling cluster. They look different because they are doing different jobs.

Three steps get you started, and the order matters:

Step one, split agent UAs out of your crawler rules — this needs no new tooling at all, only opening the rules you already have and confirming that GPTBot and ChatGPT-User are not sharing one rate-limit configuration.

Step two, automate the official IP list sync and add alerting — use the sync script in this article or an equivalent; what matters is the count guard and the non-zero exit code, so that failure becomes visible.

Step three, check your timeout budget and the completeness of your first response — split the timeout, then fetch your own page with JavaScript disabled and see whether the body text is still there.

Do these three and your site will not suddenly be cited by more AI systems. But at least, when they come to the door, the door will be open.

---

🎯 Take Action Now

When an AI Agent shows up, someone is waiting for the answer — and that one timeout does not get a second chance. The CloudInsight technical team helps Taiwanese enterprises make sense of multi-platform cloud environments, pulling scattered settings such as timeout budgets, cache strategy and rule syncing into one architecture you can actually explain.

👉 Consult now to get the plan that fits you best 👉 Join our official LINE account for real-time technical support

---

References

• IETF RFC 9309, "Robots Exclusion Protocol" (Standards Track, September 2022; the formal specification for robots.txt, retrieved 2026-08-05)
• Anthropic official documentation: does Anthropic crawl data from the web, and how can site owners block the crawler (measured 200 on 2026-08-05)
• Cloudflare verified bots documentation (page marked Last updated 2026-07-01)
• Cloudflare fake bot managed rules
• Cloudflare's announcement of new AI bot categories
• OpenAI official bot IP lists: https://openai.com/gptbot.json, https://openai.com/searchbot.json, https://openai.com/chatgpt-user.json (measured 2026-08-05)
• Anthropic official bot IP list: https://claude.com/crawling/bots.json (measured 2026-08-05)